Mentions
Extended reach
Total likes
Video views
AI synopsis
Many participants noted that only 5% of the studies reviewed used real patient data, highlighting the disconnect between research and clinical application. For instance, one post remarked on the need for evaluations that consider fairness and bias, while another mentioned the inadequacies in current methodologies. The tone of urgency and phrases like 'need broader evaluations' suggest a strong community drive towards improving AI applications in healthcare.
Most posts express a general agreement with the article's findings regarding the need for improved evaluation in healthcare LLMs.
The discussion shows strong interest in the implications of the research, highlighting its relevance to ongoing debates in healthcare technology.
Many participants engage deeply, referencing specific data points from the study and suggesting areas for improvement.
Contributors view the study as having a significant impact on the future of LLM applications in healthcare, emphasizing potential changes in evaluation standards.
Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam https://t.co/EXl5HYzd0w
But we know how to test efficacy in medicine. Clinical trials are messy, more expensive than in silico studies, and require multidisciplinary expertise. But they're still the right thing to do. Only 5% of LLM in medicine studies even use real data (https://t.co/gc2sl3VFYn)!!
｢大規模言語モデルの医療応用の評価｣ JAMA ･ 2022年〜2024年2月までに発表された 519 件の研究を対象としたsystematic review ･ 評価に実際の患者データを使用したのはわずか 5% ･ 評価の主に正確性であり、公平性、バイアス、毒性の評価はあまり研究されていない https://t.co/5wuW7A7dEP https://t.co/fRd9TgVDMR
This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of standardized frameworks and limited use of real patient data. https://t.co/3CRQ4Cb5jd
This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of standardized frameworks and limited use of real patient data. https://t.co/O7NbeErlL3 https://t.co/zqkUcvCfkg
This panel discussion focuses on evaluating large language models (LLMs) with frameworks and tools in healthcare. Key topics include a systematic review highlighting evaluation shortcomings and recommendations from…
This video discusses the influence of large language models (LLMs) in health care productivity and their potential applications. We analyze a systematic review highlighting key components such as data type and…
RT @AMAEdHub: New from JN Learning: Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/N8L2Dw4bVA
New from JN Learning: Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/N8L2Dw4bVA
RT @AMAEdHub: New today: Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/T3DSTvzU4x #CME
New today: Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/T3DSTvzU4x #CME
RT @fedelosco:
RT @fedelosco:
RT @fedelosco:
RT @fedelosco:
Only 5% of studies evaluated large language models in healthcare using real patient care data, mostly focusing on medical knowledge assessments. by Bedi S, Liu Y (...) Shah NH et 16 al. in JAMA https://t.co/qGnsCScNuI #MedX #MedResearch
RT @JAMA_current: This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of…
RT @JAMA_current: This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of…
RT @JAMA_current: This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of…
This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of standardized frameworks and limited use of real patient data. https://t.co/3CRQ4Cb5jd
This systematic review characterizes the current performance of LLM in evaluating clinical health care settings, including uniformity, thoroughness, and robustness and proposes a framework for their testing and evaluation across health care applications. https://t.co/OMQn79N0Fi]
Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/Aj9wsQ4tyN https://t.co/4ptRvoxsPp
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review | Digital Health | JAMA | JAMA Network https://t.co/cRqAU7aReh
Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/5vqXJSF6dD
RT @AdamRodmanMD: But we know how to test efficacy in medicine. Clinical trials are messy, more expensive than in silico studies, and requi…
RT @AdamRodmanMD: But we know how to test efficacy in medicine. Clinical trials are messy, more expensive than in silico studies, and requi…
RT @AdamRodmanMD: But we know how to test efficacy in medicine. Clinical trials are messy, more expensive than in silico studies, and requi…
RT @AdamRodmanMD: But we know how to test efficacy in medicine. Clinical trials are messy, more expensive than in silico studies, and requi…
But we know how to test efficacy in medicine. Clinical trials are messy, more expensive than in silico studies, and require multidisciplinary expertise. But they're still the right thing to do. Only 5% of LLM in medicine studies even use real data (https://t.co/gc2sl3VFYn)!!
Interesting: "Existing evaluations of LLMs mostly focus on accuracy of ques answering for medical exams, without consideration of real patient care data. Dimensions such as fairness, bias, toxicity & deployment considerations received limited attention" https://t.co/b6TebwSlKE
Testing and Evaluation of Health Care Applications of #LLM : A Systematic Review @JAMANetwork https://t.co/lcEb5XYaWj
From diagnostics to patient communication, large language models are transforming healthcare. This @JAMA_current review by #StanDOM's @drnigam, @niravrshah, Arnold Milstein & Michael Pfeffer, sheds light on their diverse applications & effectiveness. https://t.co/HD2KEwkvvp
The pace of AI in medical education is rapidly advancing. I appreciate summaries like this that show the gaps and where we can focus. https://t.co/H8dwGRp4om
Testing and Evaluation of Health Care Applications of Large Language Models
RT @CeoImed: ｢大規模言語モデルの医療応用の評価｣ JAMA ･ 2022年〜2024年2月までに発表された 519 件の研究を対象としたsystematic review ･ 評価に実際の患者データを使用したのはわずか 5% ･ 評価の主に正確性であり、公平性…
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review | Artificial Intelligence | JAMA | JAMA Network https://t.co/pxth5SMzu2
RT @juanelosag: cuantificar los sesgos, cubrir una gama más amplia de tareas y especialidades y reportar métricas de desempeño estandarizad…
cuantificar los sesgos, cubrir una gama más amplia de tareas y especialidades y reportar métricas de desempeño estandarizadas para permitir una implementación a gran escala.” https://t.co/1bRFPXxnGN
Testing and Evaluation of #HealthCare Applications of Large Language Models @JAMAplusAI @JAMANetworkOpen #AI #HealthAI #LLM #TechHartford https://t.co/PHImDKf3Fx
Testing and Evaluation of Health Care Applications of Large Language Models
Testing and Evaluation of Health Care Applications of Large Language Models
(2/2) Critical administrative tasks (prescribing, billing) were neglected (<1%), and bias assessment protocols were inadequate (15.8%). Bedi et al. Stanford, UCSF, et al. report in JAMA Oct 15, 2024 on doi:10.1001/jama.2024.21700 https://t.co/F4uAmqtbJo
(1/2)A systematic review (n=519) revealed substantial methodological flaws in LLM healthcare evaluations. Authentic clinical data were scarce (5%), with question-answering dominating (84.2%). https://t.co/JJoXDwo0O7
RT @CeoImed: ｢大規模言語モデルの医療応用の評価｣ JAMA ･ 2022年〜2024年2月までに発表された 519 件の研究を対象としたsystematic review ･ 評価に実際の患者データを使用したのはわずか 5% ･ 評価の主に正確性であり、公平性…
RT @CeoImed: ｢大規模言語モデルの医療応用の評価｣ JAMA ･ 2022年〜2024年2月までに発表された 519 件の研究を対象としたsystematic review ･ 評価に実際の患者データを使用したのはわずか 5% ･ 評価の主に正確性であり、公平性…
RT @CeoImed: ｢大規模言語モデルの医療応用の評価｣ JAMA ･ 2022年〜2024年2月までに発表された 519 件の研究を対象としたsystematic review ･ 評価に実際の患者データを使用したのはわずか 5% ･ 評価の主に正確性であり、公平性…
RT @CeoImed: ｢大規模言語モデルの医療応用の評価｣ JAMA ･ 2022年〜2024年2月までに発表された 519 件の研究を対象としたsystematic review ･ 評価に実際の患者データを使用したのはわずか 5% ･ 評価の主に正確性であり、公平性…
｢大規模言語モデルの医療応用の評価｣ JAMA ･ 2022年〜2024年2月までに発表された 519 件の研究を対象としたsystematic review ･ 評価に実際の患者データを使用したのはわずか 5% ･ 評価の主に正確性であり、公平性、バイアス、毒性の評価はあまり研究されていない https://t.co/5wuW7A7dEP https://t.co/fRd9TgVDMR
https://t.co/jwSwM6swCg
https://t.co/S7GP9HZ269
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
Testing and Evaluation of Health Care Applications of Large Language Models. A Systematic Review. https://t.co/c0wYR1dN5G
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
https://t.co/xnv4q7QTO2
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @daforerog: Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review https://t.co/438Kp6WosP
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review https://t.co/438Kp6WosP
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
Testing and Evaluation of Health Care Applications of Large Language Models #science #publication #research #publications https://t.co/AsAvv0eBpa
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
Systematic review finds only 5% of #LLM studies in healthcare use real patient data. We need broader evaluations that address bias, fairness, and more diverse tasks. #AI #HealthCare #MedTwitter https://t.co/mFRB0BKnOD
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
#Evidence that more scrutiny, care, and caution is needed regarding the use of #AI in #healthcare. I assume, this is also true for most other areas... https://t.co/jNngY6nAkM
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/KfwEUCsqsK
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
RT @EricTopol: Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam ht…
Of over 500 LLM #AI reports in healthcare, only 5% used real patient data. https://t.co/SAhfJxcdhz @JAMA_current @drnigam https://t.co/EXl5HYzd0w
A review of 519 studies found only 5% used real patient data to evaluate Large Language Models (LLMs) in healthcare. Most focused on medical exams, few on admin tasks like writing prescriptions. Real data and broader evaluations needed. https://t.co/Ui46zI9dLD
RT @emiliomonteb: Testing and Evaluation of Health Care Applications of Large Language ModelsA Systematic Review https://t.co/b7NBd05m1c…
Testing and Evaluation of Health Care Applications of Large Language ModelsA Systematic Review https://t.co/b7NBd05m1c #AI #IA #LLM https://t.co/IevhiBcVvV
https://t.co/qSFwuEJS5G #RADONC #LLMs #AI
Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/Bi0xVukb7m?
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. https://t.co/E4reSupeee @JAMA_current @JAMANetwork #ArtificialIntelligence #MedEd #MEDTECH #DigitalHealth @StanfordMed https://t.co/6XN7M8Qzhf
Testing and Evaluation of Health Care Applications of Large Language Models https://t.co/APxR4xVXct
RT @JAMAplusAI: This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of s…
This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of standardized frameworks and limited use of real patient data. https://t.co/XlRwt5Uweq https://t.co/xZAsuEnxIW
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review https://t.co/ULthWLsDZW #LLMs
RT @JAMA_current: This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of…
This study identifies inconsistent evaluation practices of large language models (LLMs) in health care, finding a lack of standardized frameworks and limited use of real patient data. https://t.co/O7NbeErlL3 https://t.co/zqkUcvCfkg