Atropos Health, the pioneer in translating real-world clinical data into high-quality personalized real-world evidence for care, today announced the publication of “Answering Real-World Clinical Questions Using Large Language Model Based Systems” which clearly details ChatRWD’s superiority over other large language models (LLMs) when it comes to trust, quality, and accuracy (limited hallucinations). The paper comes on the heels of the successful beta launch in Q42023.
ChatRWD™ reduces the time to produce high-quality publication-grade real-world evidence from months to minutes through a chat-based AI co-pilot. The study tested what evidence can be produced using LLMs in two different ways, and then evaluated each answer (or non-answer) on five quality measures.
Key findings:
- ChatRWD beta outperformed other LLMs with more completed answers (94%)
- Independent physician reviewers found ChatRWD beta produced the best answer on 60% of the questions and reported 44% of the answers to be trustworthy enough to inform practice. The closest alternative was OpenEvidence (which leverages existing literature) with 46% and 30% respectively.
- ChatRWD beta provided answers with evidence 58% of the time. OpenEvidence was the runner up at 24%. Other LLMs were <= 10%.
- Hallucinations, especially citing studies or reports that do not exist, occurred for literature-based LLMs. Depending on the LLM, up to 50% of the provided citations were unable to be found. Top sources of error for ChatRWD beta were question definition and that occurred less than 20% of the time when an answer was generated.
“As generative AI technologies move from hype to utilization in healthcare, it is critical that their relevance, reliability, and actionability are rigorously measured and independently verified,” said Atropos Health Chief Medical Officer and co-founder Dr. Saurabh Gombar. “High-quality clinical evidence must remain the cornerstone of value in healthcare. We put our technology to the test, and the results were clear: ChatRWD has no alternative. Independent physician evaluators trusted ChatRWD’s results, and while other LLMs were complimentary when answering well studied questions, only ChatRWD consistently was able to create evidence for novel questions.”
ChatRWD is the first application incorporating chat-to-database capability to help healthcare leaders advance and accelerate evidence generation rapidly, saving organizations valuable time, money, and staffing resources. ChatRWD users are guided through a simple workflow to validate study design choices to reduce the risk of hallucination. This unique vertical user experience is available to healthcare institutions, data platforms, research institutions, life sciences companies and more who install Atropos Health’s Generative AI platform GENEVA OS™ (Generative Evidence Acceleration Operating System). Questions are answered with data from the Atropos Evidence™ Network, the largest federated data network.with over 300 million patient records and every answer comes with a Real World Fitness Score™ (RWFS) – evaluating the “fit-for-purpose” of the data set used to answer the question. The average RWFS for datasets used in the evaluation was 80 (out of 100), indicating that the data was highly appropriate to answer the questions at hand.
“Built on the core technology GENEVA OS, ChatRWD is not tied to any single LLM ,” said Neil Sanghavi, President of Atropos Health. “GENEVA OS is deployed locally at a customer’s or data holder’s site and it does not expose any patient-level clinical or claims data to LLMs; the LLMs in ChatRWD are used to create the query and design the study, but GENEVA OS is used to execute the query – without any connection to the LLMs.”
The publication outlines the approach and the benefits to accuracy in the generation of evidence. Based on the outcome of this study, the full launch of ChatRWD will follow with the potential to democratize high-quality, accurate, transparent, and personalized real-world evidence for everyone. To become a ChatRWD beta user, sign-up at https://www.atroposhealth.com/chatrwd
About Atropos Health
Atropos Health is the developer of GENEVA OS™, the operating system for rapid healthcare evidence across a robust network of real-world data. Healthcare and life science organizations work with Atropos Health to close evidence gaps from bench to bedside, improving individual patient outcomes with data-driven care, expediting research that advances the field of medicine, and more. We aim to transform healthcare with timely, relevant real-world evidence.
To learn more about Atropos Health, visit www.atroposhealth.com or connect through LinkedIn or follow on X (Twitter) @AtroposHealth.
View source version on businesswire.com: https://www.businesswire.com/news/home/20240702832027/en/