Accuracy of artificial intelligence responses on antibiotic use in endodontics

Abstract

Aim: The increasing application of generative artificial intelligence (AI) large language models (LLMs) in various fields, including dentistry, gives rise to questions regarding the reliability of the information they produce. The objective of this study was to analyze the accuracy and consistency of responses provided by three artificial intelligence chatbots when answering questions about antibiotic use in endodontics.

Methodology: A total of 24 questions (12 dichotomous, 12 open-ended) were developed based on the American Dental Association (ADA) clinical practice guideline on antibiotic use for the urgent management of pulpal- and periapical-related dental pain and intraoral swelling. The questions were presented to three AI chatbots: ChatGPT 3.5, Google Bard (Gemini), and Microsoft Copilot. The questions were asked three times a day over the course of 10 days. The responses obtained were categorized as correct, incorrect, or insufficient according to the guidelines. The results were statistically analyzed using the chi-square test.

Results: A total of 2,160 responses were collected from AI chatbot platforms. The overall accuracy rate for responses regarding antibiotic use for dental pain was 72.5%. A statistically significant difference was observed among the accuracy rates of the platforms (p < 0.05), with ChatGPT achieving 82.4%, Google Gemini 71.3%, and Microsoft Copilot 63.7%. No statistically significant differences were found in the accuracy of responses across the morning, noon, and evening sessions for any of the models (p > 0.05).

Conclusion: Although AI chatbots hold promise as supplementary knowledge sources for patient guidance in dentistry, their current responses, particularly in response to open-ended questions, lack the depth and reliability required for antibiotic use. Continued efforts are necessary to optimize these tools to ensure clinical relevance and patient safety in endodontic antibiotic therapy.