J Med Internet Res August 2023 PMID 37606976

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study.

Arya Rao, Michael Pang, John Kim, Meghana Kamineni, Winston Lie, Anoop K Prasad, Adam Landman, Keith Dreyer, Marc D Succi

Mentions

196
Peak: 24 in Aug 2023
across 6 sources since Feb 2023

Extended reach

377,444
potential audience

Total likes

57

Video views

17

Mentions per month

Feb 2023 – Jan 2024

Where it's discussed

  • News144 articles
  • X / Twitter41 posts
  • Blogs5 articles
  • Bluesky2 posts
  • YouTube2 videos
  • Facebook2 posts

AI synopsis

What people are saying

Discussions extensively analyze the study’s findings, with mentions of accuracy percentages like 72% and 77%, and debates about methodology and reproducibility, highlighting a collective focus on the technological implications and future possibilities of AI in medicine. The tone is both evaluative and optimistic, reflecting excitement about AI’s role but also calls for cautious interpretation.

AgreementStrong agreement

Most discussions support the study’s positive findings, emphasizing ChatGPT’s high accuracy rates of around 71-77% in clinical decision-making, showing strong consensus about its potential benefits.

InterestHigh interest

The discussions consistently highlight curiosity and relevance, citing the study’s methodology, performance metrics, and implications for healthcare and medical training, indicating high interest.

EngagementHigh engagement

Participants reference specific performance figures, question the methodology, and consider future applications, demonstrating deep engagement with the content.

ImpactHigh impact

The talks underline the transformative potential of AI in healthcare workflows, its role as an adjunct, and implications for training and clinical decision support, indicating significant perceived impact.

Abstract in brief

  • This study evaluated ChatGPT's ability to support clinical decision-making by testing it on 36 standardized medical case vignettes, comparing its responses to human assessments, and finding that it achieved a high overall accuracy of about 71%.
  • ChatGPT performed best in providing final diagnoses (around 77%) but was less accurate in generating initial differential diagnoses (about 60%), and its performance was weaker on questions related to clinical management and differential diagnosis compared to general medical knowledge.
  • The results suggest that ChatGPT has significant potential for assisting in clinical workflows, especially as it accesses more clinical information, but it still shows limitations in early diagnostic reasoning and managing complex clinical tasks.

Top posts

Most liked on X and Bluesky
MESH Incubator™ at Mass General Brigham
@meshincubator · X

Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/B6OwhPI7Fi @clindatsci @MassGenBrigham @MGBInnovation @MGH_RI @MarcSucciMD @AryaRao_ Michael Pang https://t.co/Hzgubag7qJ

Aug 22, 2023 · 17 likes · 6 shares · 756 followers View post →
Hillary Johnson Author, Journalist
@oslersweb · X

In future, might AI dx ME faster + more accurately because MD bigotry and ignorance is removed from process? That's assuming dx continues to be symptom-based (no lab-based markers). More reliable dx would also provide more accurate prevalence numbers. https://t.co/jSh4VmYcAs

Aug 29, 2023 · 10 likes · 4,400 followers View post →
河野健一 Kenichi Kono | 脳外科医 CEO|AI 医療 MBA|脳血管内手術支援AI
@CeoImed · X

「臨床ワークフロー全体のGPTの有用性」 ・ 患者との関わりから、鑑別診断、検査、診断、管理を通じて、GPTの意思決定サポートを評価 ・ 最初の鑑別診断で最低の精度60%で、最終診断で最高精度77% 現時点ではある程度情報が集まってからGPT使用するのが臨床上は良さそう https://t.co/WXX9BlNiGT https://t.co/cxDa4OqJzC

Aug 25, 2023 · 9 likes · 1 share · 15,855 followers View post →
中島 啓 kei nakashima M.D., Ph.D.
@keinakashima1 · X

Chat GPTは医学的な臨床判断において,71.7%の正確性を示した. Journal of Medical Internet Research https://t.co/pqcqSATjGg

Sep 3, 2023 · 9 likes · 13,230 followers View post →
Brian Ahier
@ahier · X

ChatGPT plays doctor with 72% success https://t.co/3bXBC8ejLz https://t.co/UGtNm7ICLW

Aug 29, 2023 · 3 likes · 3 shares · 53,118 followers View post →
MESH Incubator™ at Mass General Brigham
@meshincubator · X

Study link: https://t.co/CVIMlzGhNh

Aug 22, 2023 · 2 likes · 2 shares · 756 followers View post →

All mentions

SANDRAJOHNSTON
@sandra04850184 · X

RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…

Jan 14, 2024 · 6 shares · 952 followers View post →
SANDRAJOHNSTON
@sandra04850184 · X

RT @meshincubator: Study link: https://t.co/CVIMlzGhNh

Jan 14, 2024 · 2 shares · 952 followers View post →
Akihiro Nomura
@akihiro_nomuryn · X

RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…

Jan 14, 2024 · 6 shares · 1,899 followers View post →
JMIR Publications
@jmirpub · X

RT @meshincubator: Study link: https://t.co/CVIMlzGhNh

Jan 14, 2024 · 2 shares · 35,899 followers View post →
JMIR Publications
@jmirpub · X

RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…

Jan 14, 2024 · 6 shares · 35,899 followers View post →
Scott McGrath
@smcgrath.phd · Bluesky

Nov 13, 2023 · 1 like View post →
Dexter
@Dexter_90_ · X

@Bloncka_hlasi @kajki_ Zkoušela jste chatGPT? Dost často se plete, ale někdy se trefí, tak to může stát za pokus. https://t.co/X1fbmkBFTJ

Oct 18, 2023 · 1 like · 61 followers View post →
Suede Awesome
@ajbass · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/vUruVrHjnu #AI

Sep 16, 2023 · 498 followers View post →
MedPro Group
@MedProProtector · X

Study examines #ChatGPT’s capacity for ongoing clinical decision support: https://t.co/EHeq0WF3Je #AI https://t.co/fwJJ8fJ0EI

Sep 15, 2023 · 3,436 followers View post →
Daily Dose of Data
@data_debunks · X

According to a comprehensive study assessing ChatGPT's clinical decision-making aka. Doctor playing capabilities, ChatGPT managed to achieve an overall accuracy of 71.7%. A link to the study: https://t.co/y9S78OPKt4

Sep 8, 2023 · 8 followers View post →
中島 啓 kei nakashima M.D., Ph.D.
@keinakashima1 · X

Chat GPTは医学的な臨床判断において,71.7%の正確性を示した. Journal of Medical Internet Research https://t.co/pqcqSATjGg

Sep 3, 2023 · 9 likes · 13,230 followers View post →
AI Aorta
@ai_aorta19268 · X

Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, Landman A, Dreyer K, Succi MD Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study JMIR 2023 doi: 10.2196/48659 PMID: 37606976 https://t.co/b3p6atLdd6

Sep 1, 2023 · 20 followers View post →
Brian Ahier
@ahier · X

RT @ahier: ChatGPT plays doctor with 72% success https://t.co/3bXBC8ejLz https://t.co/UGtNm7ICLW

Sep 1, 2023 · 3 shares · 53,118 followers View post →
nisten
@nisten · X

@Scobleizer Where the are their actual prompts..methodology.. or any code.. or testing setup... how the fuck is anyone supposed to reproduce this...unless they cherrypick allow you to do it?! https://t.co/grKIrbBnR4

Aug 30, 2023 · 17,806 followers View post →
Andrew Hundt
@athundt · X

@jmirpub @RetractionWatch The design of this study looks seriously problematic: https://t.co/4W5jjM521h

Aug 30, 2023 · 1,248 followers View post →
Scott McGrath
@smcgrath.phd · Bluesky

Aug 29, 2023 · 1 like View post →
Hillary Johnson Author, Journalist
@oslersweb · X

In future, might AI dx ME faster + more accurately because MD bigotry and ignorance is removed from process? That's assuming dx continues to be symptom-based (no lab-based markers). More reliable dx would also provide more accurate prevalence numbers. https://t.co/jSh4VmYcAs

Aug 29, 2023 · 10 likes · 4,400 followers View post →
Luka.eth
@LukaWeb3 · X

RT @ahier: ChatGPT plays doctor with 72% success https://t.co/3bXBC8ejLz https://t.co/UGtNm7ICLW

Aug 29, 2023 · 3 shares · 309 followers View post →
Brian Ahier
@ahier · X

ChatGPT plays doctor with 72% success https://t.co/3bXBC8ejLz https://t.co/UGtNm7ICLW

Aug 29, 2023 · 3 likes · 3 shares · 53,118 followers View post →
Spektrumlab
@spektrumlab · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/JQsJZkSjxM

Aug 28, 2023 · 531 followers View post →
Azulvictoria
@Azulvictoria · X

RT @JackieUdeC: Un estudio demostró que #ChatGPT tiene una precisión del 72% en la toma de decisiones clínicas y un 77% en el diagnóstico f…

Aug 28, 2023 · 2 shares · 4,105 followers View post →
Florencia Tevy
@florenciatevy · X

RT @JackieUdeC: Un estudio demostró que #ChatGPT tiene una precisión del 72% en la toma de decisiones clínicas y un 77% en el diagnóstico f…

Aug 28, 2023 · 2 shares · 3,740 followers View post →
O6lvl4
@o6lvl4 · X

RT @CeoImed: 「臨床ワークフロー全体のGPTの有用性」 ・ 患者との関わりから、鑑別診断、検査、診断、管理を通じて、GPTの意思決定サポートを評価 ・ 最初の鑑別診断で最低の精度60%で、最終診断で最高精度77% 現時点ではある程度情報が集まってからGPT使…

Aug 26, 2023 · 1 share · 30 followers View post →
河野健一 Kenichi Kono | 脳外科医 CEO|AI 医療 MBA|脳血管内手術支援AI
@CeoImed · X

「臨床ワークフロー全体のGPTの有用性」 ・ 患者との関わりから、鑑別診断、検査、診断、管理を通じて、GPTの意思決定サポートを評価 ・ 最初の鑑別診断で最低の精度60%で、最終診断で最高精度77% 現時点ではある程度情報が集まってからGPT使用するのが臨床上は良さそう https://t.co/WXX9BlNiGT https://t.co/cxDa4OqJzC

Aug 25, 2023 · 9 likes · 1 share · 15,855 followers View post →
Ikemen
@HandsomeKinoko · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study| @HandsomeKinoko #note https://t.co/3HzUI3PNSC

Aug 25, 2023 · 1,274 followers View post →
JMIR Publications
@jmirpub · X

RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…

Aug 23, 2023 · 6 shares · 35,899 followers View post →
Jacqueline Sepúlveda
@JackieUdeC · X

Un estudio demostró que #ChatGPT tiene una precisión del 72% en la toma de decisiones clínicas y un 77% en el diagnóstico final. Por ahora hay que analizar con cautela estos hallazgos, ya que podrían impactar los procesos formativos. #SaludDigital https://t.co/13UTrsKQQ8

Aug 23, 2023 · 1 like · 2 shares · 3,359 followers View post →
HAS-veille
@HAS_veille · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study https://t.co/V6wXS4jjog

Aug 23, 2023 · 4,138 followers View post →
Olivier Miossec
@omiossec_med · X

#ChatGPT Displays 72% Accuracy in Clinical Decision Making https://t.co/3ItjOC8G0j

Aug 23, 2023 · 1,088 followers View post →
Mass General Brigham Innovation
@MGBInnovation · X

RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…

Aug 22, 2023 · 6 shares · 2,698 followers View post →
Tom Bellis
@tommybellis · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study https://t.co/tDc3lOdrfh https://t.co/Gn4rQ873UY

Aug 22, 2023 · 80 followers View post →
Journal of Medical Internet Research
@JMIR_ · X

RT @jmirpub: New in JMIR: Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and #Usability #UX #Study h…

Aug 22, 2023 · 1 share · 284 followers View post →
JMIR Publications
@jmirpub · X

New in JMIR: Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and #Usability #UX #Study https://t.co/AHPDOvHKCs https://t.co/T7nSSf3QWN

Aug 22, 2023 · 1 share · 35,899 followers View post →
cagsei
@cagsei · X

ChatGPT klinik karar vermede 'etkileyici' bir doğruluk göstermiş. ChatGPT'nin tüm tıbbi uzmanlık alanlarında ve klinik bakım aşamalarında yaklaşık yüzde 72 doğruluğa sahip olduğu ve nihai tanı koymada ise yüzde 77 doğruluğa sahip olduğu tespit edilmiş https://t.co/nYfGfHXIdr

Aug 22, 2023 · 57 followers View post →
2 Minute Medicine®
@2minmed · X

RT @meshincubator: Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Ent…

Aug 22, 2023 · 6 shares · 5,409 followers View post →
MESH Incubator™ at Mass General Brigham
@meshincubator · X

Study link: https://t.co/CVIMlzGhNh

Aug 22, 2023 · 2 likes · 2 shares · 756 followers View post →
MESH Incubator™ at Mass General Brigham
@meshincubator · X

Great to see coverage of our new @jmirpub study in the @BostonGlobe. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/B6OwhPI7Fi @clindatsci @MassGenBrigham @MGBInnovation @MGH_RI @MarcSucciMD @AryaRao_ Michael Pang https://t.co/Hzgubag7qJ

Aug 22, 2023 · 17 likes · 6 shares · 756 followers View post →
PubmedGPT
@pubmedgpt · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow DOI: https://t.co/XuDrb4NnND #ChatGPT #PubmedGPT #AIinClinicalDecisionMaking✏ https://t.co/D2qRfDsnFh

Apr 26, 2023 · 1 like · 14 followers View post →
PubmedGPT
@pubmedgpt · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow View ORCID ProfileArya Rao, Michael Pang, John Kim, Meghana Kamineni, Winston Lie, Anoop K. Prasad, Adam Landman, Keith J Dreyer, doi: https://t.co/XuDrb4NnND https://t.co/GTtXbWJzx4

Apr 23, 2023 · 1 like · 14 followers View post →
Marc Succi, MD
@MarcSucciMD · X

Preprints from @meshincubator: Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making https://t.co/TjdhqWFxat Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/kiVOMbN27O

Apr 5, 2023 · 918 followers View post →
Robert West, PhD ✝️
@westr · X

Assessing the Utility of #ChatGPT Throughout the Entire Clinical Workflow https://t.co/Ii7ROeioEI #EHR #EMR #AI #LLM #MedTwitter “ChatGPT achieves an average performance of 71.8% across all vignettes and question types.” https://t.co/CdvGSngaKN

Feb 26, 2023 · 1 like · 19,318 followers View post →
Arya Rao
@AryaRao_ · X

RT @medrxivpreprint: Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/YOn8XhKQOX #medRxiv

Feb 26, 2023 · 1 share · 754 followers View post →
medRxiv
@medrxivpreprint · X

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow https://t.co/YOn8XhKQOX #medRxiv

Feb 26, 2023 · 1 share · 17,461 followers View post →