To evaluate the accuracy and reliability of BooksMed, a novel Large Language Model (LLM) that provides evidence-based responses to complex neurological queries.

Clinical decision-making in neurology requires reliable and up-to-date evidence-based information that takes into account the complexity and rapid evolution of the field. Due to the lack of specialized performance benchmarks and inability of the existing LLMs to navigate the vast and specialized neurological literature, they fail to provide evidence-based responses. 

To ensure that responses are evidence-based and reliable, BooksMed employs the GRADE framework (Grading of Recommendations, Assessment, Development, and Evaluations). We evaluated BooksMed using ExpertMedQA, a questionnaire designed to integrate neurology-focused, expert-level queries that require a thorough understanding of contemporary literature and its critical appraisal. A global group of physicians (n=21) across seven countries assessed responses by BooksMed and validated ExpertMedQA, ensuring a robust testing and evaluation system. 

The responses by BooksMed were factually accurate, comprehensive, formatted correctly, and incorporated relevant references. The physicians’ ratings of BooksMed, supported by a false detection rate corrected p-value < 0.0001 (binomial test of proportions for ratings 4 or 5 on Likert scale), demonstrate its consistency in providing responses that are aligned with clinicians' knowledge and training.

With its ability to navigate and respond to complex neurological inquiries within seconds, BooksMed will make a significant contribution to clinical decision-making in neurology. In addition, ExpertMedQA's validation ensures that the information provided by BooksMed is accurate, clinically relevant, and evidence-based, making it an invaluable tool for neurological research and practice, ultimately improving patient care.