A study led by Dr. Adam Rodman, an esteemed American expert in internal medicine at Beth Israel Deaconess Medical Center, set out to investigate the potential benefits of chatbots for doctors in identifying illnesses. The study's findings, as reported in The New York Times, were nothing short of surprising-the chatbot, operating independently, demonstrated a higher level of diagnostic accuracy than the doctors.
Initially, Dr. Adam Rodman was confident that chatbots would help, but the results challenged this notion. This finding challenges the prevailing notion that AI is not yet advanced enough to replace doctors in medical diagnosis. The study provided doctors with chatbots but found that many doctors were sceptical of the chatbots' suggestions and prompts, leading to vague diagnoses.
The cases given to the doctors under the study were based on real patients and part of a set of 105 cases that have been used by researchers since the 1990s. These doctors were randomly picked and were a mix of residents and physicians recruited by large American hospital systems. All the doctors were given six cases to diagnose and were expected to come up with a diagnosis assisted by AI, but this did not happen. The doctors were adamant about their beliefs and did not take the chatbot's suggestions. It was also observed that the doctors who used AI for this study did only a little better than those who did not use AI at all.
Now comes the shocking part. The diagnosis of the chatbot alone outdid the doctors. The average of the chatbot in scrutinizing the medical condition of the patient was around 90%, whereas the average of the doctors was 76%. This study also shed light on how doctors, even when provided with AI assistance, weren't able to make the most out of it.
"It was only a fraction of the doctors who realised they could literally copy-paste the entire case history into the chatbot and just ask it to give a comprehensive answer to the entire question," said Dr. Jonathan H. Chen, a physician and computer scientist at Stanford, who authored the study.
"Only a fraction of doctors actually saw the surprisingly smart and comprehensive answers the chatbot was capable of producing."