Since its introduction in November of 2022, ChatGPT, a user-friendly general-purpose AI model, has been lauded as a monumental advance in interactive AI, and raising interest in its potential applications.

A recent STAT blog post, by Professor of Health Care Policy, Ateev Mehrotra, MD, PhD, Ruth Hailu, and colleagues, explores one of the potential applications of ChatGPT, diagnosing illnesses.

They compared the performance of ChatGPT to symptom checkers. Symptom checkers are websites and apps that help people diagnose what might be wrong with them. In a 2015 study comparing online to physician diagnoses, the Health Care Policy team’s findings revealed that online symptom checkers had a success rate of 44 % compared to the 84%  success rate of physicians.

In this new work, using the 45 vignettes employed in the original study, the team tested ChatGPT’s accuracy in producing correct diagnoses. Findings revealed that Chat GPT, even in its infancy, approaches the accuracy of a physician diagnoses at 87%.

Such results inspire hope that Chat GPT’s accuracy, interactive nature, and wide reach will establish AI as a standard in clinical care; using AI to assist and aid in the reduction of misdiagnoses due to human error.

While promising, ChatGPT and anticipated interactive AI are not without concerns. To start, ChatGPT’s results are sensitive to how information is presented. Context and language will be decisive in attaining accurate responses and more rigorous research is required in this area.

It is also undetermined how AI will be implemented in a clinical setting; how will patient information be fed into an algorithm as part of clinic workflow?

There is also concern regarding trust in AI output: what will happen if the physician disagrees; how will this affect patient interactions, and how will such disagreements be resolved?

Finally, it must be duly noted that AI models are prone to harmful biases; as mentioned: diversity in the response is not ensured, but risks amplifying biases and stereotypes that are embedded in the source material.

More rigorous testing and analysis will be needed to address these concerns; in the meantime, with the rise in accuracy and availability of interactive AI, the health care system will need to find new and informed ways to respond to the future of computer-assisted diagnosis.