Comparative analysis of the performance of ChatGPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro large language models on theoretical orthodontic questions
DOI:
https://doi.org/10.5577/intdentres.777Keywords:
Artificial intelligence, ChatGPT, dental education, knowledge assessment, large language models, orthodonticsAbstract
Aim: This study aimed to comparatively evaluate the performance of ChatGPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro large language models (LLMs) in answering theoretical orthodontic questions under web search-enabled conditions.
Methodology: A total of 30 open-ended questions covering the topics of growth and development, functional analysis, mixed dentition, extraoral appliances, malocclusion, and orthodontic treatment planning were prepared from Contemporary Orthodontics (7th edition, 2025). The questions were presented to all three models under standardized conditions in separate sessions in March 2026 with web search enabled. The responses were anonymized and independently evaluated in a blinded manner by two orthodontists using a 5-point Likert scale incorporating accuracy, scientific adequacy, completeness, and clarity into a single overall score. Data were analyzed using IBM SPSS Statistics (version 31.0); the Friedman test was used for model comparisons, and statistical significance was set at p < 0.05.
Results: Mean performance scores were 4.40 ± 0.72 for ChatGPT-5.4, 4.43 ± 0.61 for Claude Sonnet 4.6, and 4.32 ± 0.88 for Gemini 3.1 Pro. Median scores were 4.50, 4.50, and 4.75, respectively. No statistically significant difference in overall performance was found among the models (χ² = 0.400; df = 2; p = 0.819; Kendall's W = 0.007). Subgroup analyses by topic category also revealed no significant differences in any category (all p > 0.05). Inter-rater reliability was limited (α = 0.706).
Conclusion: All three models achieved high-performance scores, with no statistically significant difference in overall performance among them; this does not establish equivalence. With independent verification, these LLMs may serve as supportive tools in orthodontic education.
Downloads
References
Agrawal P, Nikhade P. Artificial intelligence in dentistry: past, present, and future. Cureus. 2022;14(8):e27405. https://doi.org/10.7759/cureus.27405
Khanagar SB, Al-Ehaideb A, Maganur PC, et al. Developments, application, and performance of artificial intelligence in dentistry — a systematic review. J Dent Sci. 2021;16(1):508-522. https://doi.org/10.1016/j.jds.2020.06.019
Chakraborty C, Pal S, Bhattacharya M, Dash S, Lee SS. Overview of chatbots with special emphasis on artificial intelligence-enabled ChatGPT in medical science. Front Artif Intell. 2023;6:1237704. https://doi.org/10.3389/frai.2023.1237704
Fields HW, Larson BE, Sarver DM, Proffit WR. Contemporary Orthodontics. 7th ed. St Louis: Elsevier; 2025.
Kazimierczak N, Kazimierczak W, Serafin Z, Nowicki P, Nożewski J, Janiszewska-Olszowska J. AI in orthodontics: revolutionizing diagnostics and treatment planning — a comprehensive review. J Clin Med. 2024;13(2):344. https://doi.org/10.3390/jcm13020344
Cesur E, Tuncer K, Sevgi D, Balaban BC, Arslan C. Evaluation of the quality and reliability of YouTube™ videos created by orthodontists as an information source for clear aligners. Turk J Orthod. 2024;37(1):44-49. https://doi.org/10.4274/TurkJOrthod.2023.2022.127
Makrygiannakis MA, Giannakopoulos K, Kaklamanos EG. Evidence-based potential of generative artificial intelligence large language models in orthodontics: a comparative study of ChatGPT, Google Bard, and Microsoft Bing. Eur J Orthod. 2026;48(1):cjae017. https://doi.org/10.1093/ejo/cjae017
Vassis S, Powell H, Petersen E, et al. Large-language models in orthodontics: assessing reliability and validity of ChatGPT in pretreatment patient education. Cureus. 2024;16(9):e68085. https://doi.org/10.7759/cureus.68085
Erdem B, Özcan M, Şar Ç. Comparative analysis of artificial intelligence chatbots in orthodontic emergency scenarios: ChatGPT-3.5, ChatGPT-4.0, Copilot, and Gemini. Angle Orthod. 2026;96(1):100-105. https://doi.org/10.2319/021825-146.1
Sokmen T, Kar İ, Tuncer C, Baloş Tuncer B. Comparison of the information of generative artificial intelligence large language models and professional guidelines regarding nutritional advice for orthodontic patients. BMC Oral Health. 2025;25(1):1845. https://doi.org/10.1186/s12903-025-07153-1
Patil S, Eisenhuth G, El-Bialy T, Licari FW. Reliability and performance of four large language models in orthodontic knowledge assessment. J Dent Educ. 2026;90(4):462-465. https://doi.org/10.1002/jdd.14002
CHART Collaborative. Reporting guideline for chatbot health advice studies: the Chatbot Assessment Reporting Tool (CHART) statement. BMJ Med. 2025;4(1):e001632. https://doi.org/10.1136/bmjmed-2025-001632
Krippendorff K. Content Analysis: An Introduction to Its Methodology. 2nd ed. Thousand Oaks, CA: Sage; 2004.
Taşyürek M, Adıgüzel Ö, Cangül S, Ortaç H. Comparative evaluation of the accuracy and completeness of responses provided by ChatGPT-5, Gemini 2.5 Flash, Grok 4, and DeepSeek-R1 to open-ended questions on cracked and fractured teeth. J Med Dent Invest. 2025;6:e250161. https://doi.org/10.5577/jomdi.e250161
Kandemir M, Sarıbaş EE. Comparative performance of large language models in answering periodontology questions from the Turkish Dental Specialty Examination: a cross-sectional study on accuracy and coverage. BMC Oral Health. 2025;25(1):1804. https://doi.org/10.1186/s12903-025-07241-2
Turan Gökduman C, Arılı Öztürk E, Aktaş Ş, Çanakçi BC. Comparison of chatbots' accuracy in endodontics questions in dentistry specialization exam in Türkiye: ChatGPT-4o, Gemini Advanced, Copilot, and Claude. BMC Oral Health. 2026;26(1):28. https://doi.org/10.1186/s12903-025-07346-8
Karatekin AO. Comparison and review of different versions of OpenAI ChatGPT, Anthropic Claude and Google Gemini large language models' performance on endodontics questions in the Turkish Dentistry Specialization Exam. Eur J Dent Educ. 2026. https://doi.org/10.1111/eje.70121
Taşyürek M, Adıgüzel Ö, Ortaç H. Comparative evaluation of responses from ChatGPT-5, Gemini 2.5 Flash, Grok 4, and Claude Sonnet-4 chatbots to questions about endodontic iatrogenic events. Healthcare (Basel). 2025;13(20):2615. https://doi.org/10.3390/healthcare13202615
Danesh A, Pazouki H, Danesh K, Danesh F, Danesh A. The performance of artificial intelligence language models in board-style dental knowledge assessment: a preliminary study on ChatGPT. J Am Dent Assoc. 2023;154(11):970-974. https://doi.org/10.1016/j.adaj.2023.07.016
Downloads
Published
Issue
Section
Categories
License
Copyright (c) 2026 The Author(s)
This work is licensed under a Creative Commons Attribution 4.0 International License.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Open Access articles are distributed under the terms of the