Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study.

Arya Rao, Michael Pang, John Kim, Meghana Kamineni, Winston Lie, Anoop K Prasad, Adam Landman, Keith Dreyer, Marc D Succi

August 2023 J Med Internet Res

Synopsis of Social media discussions

Discussions extensively analyze the study’s findings, with mentions of accuracy percentages like 72% and 77%, and debates about methodology and reproducibility, highlighting a collective focus on the technological implications and future possibilities of AI in medicine. The tone is both evaluative and optimistic, reflecting excitement about AI’s role but also calls for cautious interpretation.

A
Agreement
 Strong agreement

Most discussions support the study’s positive findings, emphasizing ChatGPT’s high accuracy rates of around 71-77% in clinical decision-making, showing strong consensus about its potential benefits.

I
Interest
 High level of interest

The discussions consistently highlight curiosity and relevance, citing the study’s methodology, performance metrics, and implications for healthcare and medical training, indicating high interest.

E
Engagement
 High engagement

Participants reference specific performance figures, question the methodology, and consider future applications, demonstrating deep engagement with the content.

I
Impact
 High level of impact

The talks underline the transformative potential of AI in healthcare workflows, its role as an adjunct, and implications for training and clinical decision support, indicating significant perceived impact.

Social Mentions

YouTube

2 Videos

Bluesky

2 Posts

Facebook

2 Posts

Twitter

41 Posts

Blogs

5 Articles

News

144 Articles

Metrics

Video Views

17

Total Likes

57

Extended Reach

377,444

Social Features

196

Timeline: Posts about article

Top Social Media Posts

Posts referencing the article

Evaluating ChatGPT's Role in Enhancing Clinical Workflow Efficiency

Evaluating ChatGPT's Role in Enhancing Clinical Workflow Efficiency

This study assessed ChatGPT's ability to support clinical decision-making, showing high overall accuracy and potential for assisting in various clinical tasks, especially as it accesses more clinical information.

Medicine RTCL TV

December 4, 2023

11 views

Evaluating ChatGPT's Role in Enhancing Clinical Workflow Accuracy

Evaluating ChatGPT's Role in Enhancing Clinical Workflow Accuracy

This study evaluates ChatGPT's ability to support clinical decision-making, testing it on 36 medical case vignettes and comparing responses to human assessments. It highlights ChatGPT's strengths and limitations in diagnosis and clinical management support.

Medicine RTCL TV

November 20, 2023

6 views

  • SANDRAJOHNSTON
    @sandra04850184 (Twitter)

    RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…
    view full post

    January 14, 2024

    6

  • SANDRAJOHNSTON
    @sandra04850184 (Twitter)

    RT @meshincubator: Study link: https://t.co/CVIMlzGhNh
    view full post

    January 14, 2024

    2

  • Akihiro Nomura
    @akihiro_nomuryn (Twitter)

    RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…
    view full post

    January 14, 2024

    6

  • JMIR Publications
    @jmirpub (Twitter)

    RT @meshincubator: Study link: https://t.co/CVIMlzGhNh
    view full post

    January 14, 2024

    2

  • JMIR Publications
    @jmirpub (Twitter)

    RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…
    view full post

    January 14, 2024

    6

  • Scott McGrath
    @smcgrath.phd (Bluesky)


    view full post

    November 13, 2023

    1

  • Dexter
    @Dexter_90_ (Twitter)

    @Bloncka_hlasi @kajki_ Zkoušela jste chatGPT? Dost často se plete, ale někdy se trefí, tak to může stát za pokus. https://t.co/X1fbmkBFTJ
    view full post

    October 18, 2023

    1

  • Suede Awesome
    @ajbass (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/vUruVrHjnu #AI
    view full post

    September 16, 2023

  • MedPro Group
    @MedProProtector (Twitter)

    Study examines #ChatGPT’s capacity for ongoing clinical decision support: https://t.co/EHeq0WF3Je #AI https://t.co/fwJJ8fJ0EI
    view full post

    September 15, 2023

  • Daily Dose of Data
    @data_debunks (Twitter)

    According to a comprehensive study assessing ChatGPT's clinical decision-making aka. Doctor playing capabilities, ChatGPT managed to achieve an overall accuracy of 71.7%. A link to the study: https://t.co/y9S78OPKt4
    view full post

    September 8, 2023

  • 中島 啓 kei nakashima M.D., Ph.D.
    @keinakashima1 (Twitter)

    Chat GPTは医学的な臨床判断において，71.7%の正確性を示した． Journal of Medical Internet Research https://t.co/pqcqSATjGg
    view full post

    September 3, 2023

    9

  • AI Aorta
    @ai_aorta19268 (Twitter)

    Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, Landman A, Dreyer K, Succi MD Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study JMIR 2023 doi: 10.2196/48659 PMID: 37606976 https://t.co/b3p6atLdd6
    view full post

    September 1, 2023

  • Brian Ahier
    @ahier (Twitter)

    RT @ahier: ChatGPT plays doctor with 72% success https://t.co/3bXBC8ejLz https://t.co/UGtNm7ICLW
    view full post

    September 1, 2023

    3

  • nisten
    @nisten (Twitter)

    @Scobleizer Where the are their actual prompts..methodology.. or any code.. or testing setup... how the fuck is anyone supposed to reproduce this...unless they cherrypick allow you to do it?! https://t.co/grKIrbBnR4
    view full post

    August 30, 2023

  • Andrew Hundt
    @athundt (Twitter)

    @jmirpub @RetractionWatch The design of this study looks seriously problematic: https://t.co/4W5jjM521h
    view full post

    August 30, 2023

  • Scott McGrath
    @smcgrath.phd (Bluesky)


    view full post

    August 29, 2023

    1

  • Hillary Johnson Author, Journalist
    @oslersweb (Twitter)

    In future, might AI dx ME faster + more accurately because MD bigotry and ignorance is removed from process? That's assuming dx continues to be symptom-based (no lab-based markers). More reliable dx would also provide more accurate prevalence numbers. https://t.co/jSh4VmYcAs
    view full post

    August 29, 2023

    10

  • Luka.eth
    @LukaWeb3 (Twitter)

    RT @ahier: ChatGPT plays doctor with 72% success https://t.co/3bXBC8ejLz https://t.co/UGtNm7ICLW
    view full post

    August 29, 2023

    3

  • Brian Ahier
    @ahier (Twitter)

    ChatGPT plays doctor with 72% success https://t.co/3bXBC8ejLz https://t.co/UGtNm7ICLW
    view full post

    August 29, 2023

    3

    3

  • Spektrumlab
    @spektrumlab (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/JQsJZkSjxM
    view full post

    August 28, 2023


  • @Azulvictoria (Twitter)

    RT @JackieUdeC: Un estudio demostró que #ChatGPT tiene una precisión del 72% en la toma de decisiones clínicas y un 77% en el diagnóstico f…
    view full post

    August 28, 2023

    2

  • Florencia Tevy
    @florenciatevy (Twitter)

    RT @JackieUdeC: Un estudio demostró que #ChatGPT tiene una precisión del 72% en la toma de decisiones clínicas y un 77% en el diagnóstico f…
    view full post

    August 28, 2023

    2

  • O6lvl4
    @o6lvl4 (Twitter)

    RT @CeoImed: ｢臨床ワークフロー全体のGPTの有用性｣ ･ 患者との関わりから、鑑別診断、検査、診断、管理を通じて、GPTの意思決定サポートを評価 ･ 最初の鑑別診断で最低の精度60%で、最終診断で最高精度77% 現時点ではある程度情報が集まってからGPT使…
    view full post

    August 26, 2023

    1

  • 河野健一 Kenichi Kono ｜ 脳外科医 CEO｜AI 医療 MBA｜脳血管内手術支援AI
    @CeoImed (Twitter)

    ｢臨床ワークフロー全体のGPTの有用性｣ ･ 患者との関わりから、鑑別診断、検査、診断、管理を通じて、GPTの意思決定サポートを評価 ･ 最初の鑑別診断で最低の精度60%で、最終診断で最高精度77% 現時点ではある程度情報が集まってからGPT使用するのが臨床上は良さそう https://t.co/WXX9BlNiGT https://t.co/cxDa4OqJzC
    view full post

    August 25, 2023

    9

    1

  • Ikemen
    @HandsomeKinoko (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study｜ @HandsomeKinoko #note https://t.co/3HzUI3PNSC
    view full post

    August 25, 2023

  • JMIR Publications
    @jmirpub (Twitter)

    RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…
    view full post

    August 23, 2023

    6

  • Jacqueline Sepúlveda
    @JackieUdeC (Twitter)

    Un estudio demostró que #ChatGPT tiene una precisión del 72% en la toma de decisiones clínicas y un 77% en el diagnóstico final. Por ahora hay que analizar con cautela estos hallazgos, ya que podrían impactar los procesos formativos. #SaludDigital https://t.co/13UTrsKQQ8
    view full post

    August 23, 2023

    1

    2

  • HAS-veille
    @HAS_veille (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study https://t.co/V6wXS4jjog
    view full post

    August 23, 2023

  • Olivier Miossec
    @omiossec_med (Twitter)

    #ChatGPT Displays 72% Accuracy in Clinical Decision Making https://t.co/3ItjOC8G0j
    view full post

    August 23, 2023

  • Mass General Brigham Innovation
    @MGBInnovation (Twitter)

    RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…
    view full post

    August 22, 2023

    6

  • Tom Bellis
    @tommybellis (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study https://t.co/tDc3lOdrfh https://t.co/Gn4rQ873UY
    view full post

    August 22, 2023

  • Journal of Medical Internet Research
    @JMIR_ (Twitter)

    RT @jmirpub: New in JMIR: Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and #Usability #UX #Study h…
    view full post

    August 22, 2023

    1

  • JMIR Publications
    @jmirpub (Twitter)

    New in JMIR: Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and #Usability #UX #Study https://t.co/AHPDOvHKCs https://t.co/T7nSSf3QWN
    view full post

    August 22, 2023

    1

  • cagsei
    @cagsei (Twitter)

    ChatGPT klinik karar vermede 'etkileyici' bir doğruluk göstermiş. ChatGPT'nin tüm tıbbi uzmanlık alanlarında ve klinik bakım aşamalarında yaklaşık yüzde 72 doğruluğa sahip olduğu ve nihai tanı koymada ise yüzde 77 doğruluğa sahip olduğu tespit edilmiş https://t.co/nYfGfHXIdr
    view full post

    August 22, 2023

  • 2 Minute Medicine®
    @2minmed (Twitter)

    RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…
    view full post

    August 22, 2023

    6

  • MESH Incubator™ at Mass General Brigham
    @meshincubator (Twitter)

    Study link: https://t.co/CVIMlzGhNh
    view full post

    August 22, 2023

    2

    2

  • MESH Incubator™ at Mass General Brigham
    @meshincubator (Twitter)

    Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/B6OwhPI7Fi @clindatsci @MassGenBrigham @MGBInnovation @MGH_RI @MarcSucciMD @AryaRao_ Michael Pang https://t.co/Hzgubag7qJ
    view full post

    August 22, 2023

    17

    6

  • PubmedGPT
    @pubmedgpt (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow DOI: https://t.co/XuDrb4NnND #ChatGPT #PubmedGPT #AIinClinicalDecisionMaking✏ https://t.co/D2qRfDsnFh
    view full post

    April 26, 2023

    1

  • PubmedGPT
    @pubmedgpt (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow View ORCID ProfileArya Rao, Michael Pang, John Kim, Meghana Kamineni, Winston Lie, Anoop K. Prasad, Adam Landman, Keith J Dreyer, doi: https://t.co/XuDrb4NnND https://t.co/GTtXbWJzx4
    view full post

    April 23, 2023

    1

  • Marc Succi, MD
    @MarcSucciMD (Twitter)

    Preprints from @meshincubator: Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making https://t.co/TjdhqWFxat Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/kiVOMbN27O
    view full post

    April 5, 2023

  • Robert West, PhD ✝️
    @westr (Twitter)

    Assessing the Utility of #ChatGPT Throughout the Entire Clinical Workflow https://t.co/Ii7ROeioEI #EHR #EMR #AI #LLM #MedTwitter “ChatGPT achieves an average performance of 71.8% across all vignettes and question types.” https://t.co/CdvGSngaKN
    view full post

    February 26, 2023

    1

  • Arya Rao
    @AryaRao_ (Twitter)

    RT @medrxivpreprint: Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/YOn8XhKQOX #medRxiv
    view full post

    February 26, 2023

    1

  • medRxiv
    @medrxivpreprint (Twitter)

    Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/YOn8XhKQOX #medRxiv
    view full post

    February 26, 2023

    1

Abstract Synopsis

  • This study evaluated ChatGPT's ability to support clinical decision-making by testing it on 36 standardized medical case vignettes, comparing its responses to human assessments, and finding that it achieved a high overall accuracy of about 71%.
  • ChatGPT performed best in providing final diagnoses (around 77%) but was less accurate in generating initial differential diagnoses (about 60%), and its performance was weaker on questions related to clinical management and differential diagnosis compared to general medical knowledge.
  • The results suggest that ChatGPT has significant potential for assisting in clinical workflows, especially as it accesses more clinical information, but it still shows limitations in early diagnostic reasoning and managing complex clinical tasks.