ABSTRACT

Objective

To evaluate the reliability and quality of Grok artificial intelligence’s (AI) responses to commonly asked questions about pediatric dentofacial health.

Methods

Four queries related to detrimental oral habits were submitted to Grok AI in separate sessions. The generated responses were independently evaluated by orthodontists, pediatricians, and parents without a medical or dental background. Reliability was assessed using the eight-item modified DISCERN instrument, and overall information quality was evaluated using the Global Quality Scale (GQS). Group comparisons were performed using the Kruskal-Wallis test, and correlations among scores were analyzed using Spearman’s rank correlation.

Results

Across evaluator groups, mean modified DISCERN scores ranged from 23.85 to 27.66, and mean GQS scores ranged from 3.61 to 4.12, corresponding, overall, to moderate-to-good reliability and quality. No statistically significant differences were observed among orthodontists, pediatricians, and parents for any query. Correlation analyses showed significant positive associations among several modified DISCERN and GQS parameters, particularly across different queries within the same evaluator groups. These findings suggested consistent scoring patterns, especially among parents and pediatricians.

Conclusion

Grok AI provided moderately reliable and generally useful responses regarding pediatric dentofacial health. However, its responses should be considered as supportive educational information rather than a substitute for a professional pediatric or orthodontic assessment. Further validation using larger question sets, multiple AI models, and more diverse evaluator groups is needed.

INTRODUCTION

Artificial intelligence (AI) applications, particularly chatbots powered by large language models, are being increasingly adopted in healthcare. These systems offer patients round-the-clock accessibility, user-friendliness, and rapid responses, enabling individuals to obtain information at any time of day (1, 2). Owing to these features, they are regarded as a practical source of information by both the public and healthcare professionals.

Nevertheless, the accuracy and reliability of AI-based chatbots remain a matter of debate. Several studies have demonstrated that such tools may occasionally generate incorrect or irrelevant content and may even present misinformation as factual (3, 4). More recent investigations have further highlighted that outputs generated by AI models can contain biases related to gender, race, or socioeconomic status (5). In addition, chatbot responses are often produced at a level exceeding recommended health literacy standards, which may hinder patients’ understanding and reduce their active participation in care (6-8).

Recent studies have begun to evaluate the reliability of AI chatbots across different medical disciplines. Kurt Demirsoy et al. (9) examined ChatGPT’s responses in orthodontics and found that, while the model could contribute to patient education, it remained limited in terms of factual accuracy. Similarly, Terzi et al. (10) reported that ChatGPT’s responses regarding dental implants and peri-implant phenotypes were accessible but lacked sufficient depth for clinical decision-making. These findings underscore the potential utility of AI-based chatbots and emphasize the need for systematic evaluation across diverse clinical contexts. Although Grok has recently been included in comparative evaluations of large language models in dentistry, its reliability in pediatric dentofacial health counseling remains insufficiently characterized (11). This gap provided the rationale for selecting Grok for the present study.

Dentofacial health is closely linked to growth and development during childhood and adolescence. Harmful oral habits can negatively influence jaw development, disrupt muscle balance, and impair orofacial functions (12). In this context, both parents and healthcare professionals—particularly pediatricians and orthodontists—play a critical role in guiding healthy development. Preventive strategies and effective counseling, however, depend on access to trustworthy information (13). The reliability of information provided by AI-based digital advisors is therefore of great importance, not only for empowering parents but also for supporting clinicians in their educational and counseling roles.

This study aimed to assess the reliability of Grok AI, an AI-based chatbot, in providing guidance on dentofacial health. The chatbot’s responses were evaluated by participants from different professional backgrounds using the modified DISCERN (mDISCERN) tool and the Global Quality Scale (GQS). By exploring the reliability of AI-based conversational agents in this field, the study seeks to contribute to both clinical practice and patient and parent education.

MATERIALS AND METHODS

In this study, frequently asked questions about detrimental oral habits were submitted to Grok AI, an artificial- intelligence-based conversational agent. To minimize the risk that the system’s adaptive learning would influence subsequent responses, each question was entered separately into new sessions. The generated answers were then evaluated by three distinct participant groups through an online survey. Each group reviewed the same set of responses, thereby allowing a comprehensive assessment of Grok AI’s accuracy, clarity, and relevance to the topic.

The following questions were directed to the chatbot:

1. What are the effects of mouth breathing on the dentofacial region?

2. What are the impacts of prolonged pacifier and bottle use on the head and facial region?

3. How do habits such as nail-biting, thumb-sucking, and biting foreign objects (e.g., pencils) affect the dentofacial region?

4. What are the effects of frenulum and tongue-tie on the dentofacial region?

The responses generated by Grok AI were assessed by three groups, including two expert panels, to capture perspectives across different demographics and educational backgrounds. The participant groups were defined as follows:

1. Pediatricians (medical specialists in child health),

2. Orthodontists (dental specialists in orthodontics),

3. Parents: individuals with at least an undergraduate degree,  who have children but no professional background in medicine or dentistry.

Inclusion Criteria

• Having completed specialty training in pediatrics or orthodontics and voluntarily consenting to participate in the survey.

• For inclusion in the parent group: having at least an undergraduate  degree, being a parent, and not belonging to a health-related  profession.

Exclusion Criteria

• Being a resident in pediatrics or orthodontics.

• Declining to provide informed consent for participation.

• Parents who, despite meeting the above criteria, were affiliated with medical or dental professions.

The sample size was calculated using the G*Power program (version 3.1.9.2; Axel Buchner, University of Düsseldorf, Düsseldorf, Germany) to detect a medium effect size with 90% power. The required sample size was determined to be 66 participants (effect size: 0.4382637).

Evaluation of Responses

The chatbot’s answers were evaluated using two validated instruments: the mDISCERN questionnaire and the GQS. Assessments were performed independently by three groups: orthodontists, pediatricians, and parents with no professional background in healthcare.

Evaluation Scales

The DISCERN instrument is a widely used three-part tool designed to evaluate the reliability and quality of online health information (3, 14). In this study, only the first section consisting of eight items was applied to examine the reliability of responses generated by the AI-based chatbot (14). The eight criteria assessed in the mDISCERN scale were as follows: (i) clarity of aims, (ii) achievement of stated aims, (iii) relevance of content to the topic, (iv) citation of information sources, (v) currency of information, (vi) impartiality and balance, (vii) provision of additional sources of support, and (viii) acknowledgment of areas of uncertainty. Each item was rated on a scale from 1 (no) to 5 (yes). The total score was categorized as poor (8-15), moderate (16-31), and good (32-40) (15).

The GQS is a five-point instrument used to assess the quality of medical information sources (2). The scale is defined as follows:

• 1 point: poor quality, lacking information, not useful for patients,

• 2 points: poor quality, many important details missing, limited usefulness,

• 3 points: moderate quality, some important information adequately addressed, others insufficiently covered,

• 4 points: good quality, most key information adequate and useful for patients,

• 5 points: excellent quality, fluent, comprehensive, and highly useful.

Accordingly, GQS scores of 1-2 indicate low quality, scores of 3 indicate moderate quality, and scores of 4-5 indicate high quality.

This study was approved by the Ordu University Non-Interventional Scientific Research Ethics Committee (approval no: 2025/246, date: 18.07.2025). Electronic informed consent was obtained from all participants before participation in the online survey.

Statistical Analysis

Statistical analyses were performed using IBM SPSS Statistics for Windows, version 20.0 (IBM Corp., Armonk, NY, USA). The normality of mDISCERN and GQS score distributions was assessed using the Shapiro-Wilk test. Descriptive statistics were presented as mean ± standard deviation and minimum-maximum values. Because the score distributions did not meet normality assumptions and the GQS represents an ordinal rating scale, nonparametric methods were used. The Kruskal-Wallis test was applied to compare mDISCERN and GQS scores among the three evaluator groups. Spearman’s rank correlation analysis was used to evaluate associations between mDISCERN and GQS scores within each evaluator group. All statistical tests were two-tailed, and a p-value of < 0.05 was considered statistically significant.

RESULTS

The quality and reliability of Grok AI’s responses were evaluated using the mDISCERN and GQS instruments across three participant groups: orthodontists, pediatricians, and parents. As shown in Table 1, the mean mDISCERN scores for the four queries did not differ significantly among the groups. The mean mDISCERN scores ranged from 23.85 to 24.90 in the orthodontist group, from 25.95 to 27.66 in the pediatrician group, and from 25.52 to 27.09 in the parent group. Similarly, GQS scores showed no statistically significant differences among the three groups, with mean values ranging from 3.61 to 4.12. These findings indicate that Grok AI’s responses were rated at comparable levels by orthodontists, pediatricians, and parents, corresponding overall to moderate-to-good levels of reliability and quality.

In the orthodontist group (Table 2), significant positive correlations were observed primarily among scores from the same assessment instrument across different items. Among GQS scores, the strongest correlations were found between Query 2 and Query 3 (r = 0.855, p < 0.001), Query 2 and Query 4 (r = 0.821, p < 0.001), and Query 3 and Query 4 (r = 0.792, p < 0.001). Similarly, mDISCERN scores showed significant positive correlations across different queries, including Query 2 and Query 3 (r = 0.756, p < 0.001), Query 2 and Query 4 (r = 0.747, p < 0.001), and Query 3 and Query 4 (r = 0.719, p < 0.001). These findings suggest that orthodontists exhibit a consistent scoring pattern when evaluating Grok AI’s responses across different dentofacial health topics.

In the pediatrician group (Table 3), significant positive correlations were also observed among GQS scores across different queries. These correlations ranged from r = 0.616 to r = 0.865, with the strongest association observed between GQS scores for Query 3 and Query 4 (r = 0.865, p < 0.001). Strong positive correlations were also detected among mDISCERN scores across different queries, ranging from r = 0.792 to r = 0.867. In addition, significant correlations were found between GQS and mDISCERN scores for Query 2 (r = 0.588, p = 0.003) and for Query 3 (r = 0.529, p = 0.008). These findings indicate that pediatricians demonstrate a consistent evaluation pattern, particularly in terms of response quality and reliability across the four queries.

Among parents (Table 4), GQS scores across different queries showed very strong positive correlations. The strongest correlations were detected between Query 2 and Query 4 (r = 0.928, p < 0.001), Query 2 and Query 3 (r = 0.884, p < 0.001), and Query 3 and Query 4 (r = 0.871, p < 0.001). Similarly, mDISCERN scores showed strong positive correlations across different queries, including Query 2 and Query 3 (r = 0.892, p < 0.001); Query 3 and Query 4 (r = 0.868, p < 0.001); Query 1 and Query 4 (r = 0.852, p < 0.001); Query 1 and Query 3 (r = 0.831, p < 0.001); Query 2 and Query 4 (r = 0.780, p < 0.001); and Query 1 and Query 2 (r = 0.677, p = 0.001). A significant positive correlation was also found between Query 3 GQS and Query 3 mDISCERN scores (r = 0.447, p = 0.042). These results suggest that parents tended to evaluate the quality and reliability of Grok AI’s responses consistently across different questions.

Grok AI’s responses were rated similarly by all three evaluator groups, with no significant between-group differences in mDISCERN or GQS scores. Correlation analyses revealed significant positive associations among several GQS and mDISCERN parameters, particularly across different queries evaluated by the same evaluator groups. These findings suggest consistency in evaluators’ scoring patterns, particularly among parents and pediatricians.

DISCUSSION

In this study, the reliability and quality of responses generated by Grok AI, an AI-based conversational agent, were evaluated in the context of pediatric dentofacial health, based on assessments by orthodontists, pediatricians, and parents. The findings demonstrated that both mDISCERN and GQS scores exhibited overall moderate-to-good reliability and quality. Mean mDISCERN scores ranged between 23 and 27, while GQS scores ranged from 3.6 to 4.1, with no statistically significant differences observed among the three groups. These results suggest that Grok AI’s responses were perceived as broadly comparable by both professional and non-professional evaluators.

Correlation analyses provided additional insight into evaluator scoring patterns. Significant positive correlations were observed among several GQS and mDISCERN parameters, particularly across different queries evaluated by the same evaluator groups. In the orthodontist group, correlations were primarily observed within GQS scores and within mDISCERN scores across different queries. In the pediatrician group, within-scale correlations and some GQS–mDISCERN correlations were significant. Among parents, very strong positive correlations were observed across GQS scores and mDISCERN scores for different queries. These findings suggest that participants consistently evaluated Grok AI’s responses across dentofacial health topics. However, these results should be interpreted as evidence of consistent scoring patterns rather than complete agreement between the two instruments.

Similar findings have been reported in previous studies. Kurt Demirsoy et al. (9) examined ChatGPT’s responses in the field of orthodontics and reported that, although the content was accessible, limitations remained in terms of factual accuracy. Similarly, Terzi et al. (10) evaluated AI-generated responses regarding dental implants and peri-implant phenotypes and concluded that, while the information may be useful for patient education, it lacked sufficient depth to support clinical decision-making. In line with these reports, our findings suggest that Grok AI may serve as a supportive educational resource to provide general information for parents. However, it should not be considered a substitute for professional pediatric or orthodontic assessment.

The general medical literature also supports similar conclusions. Liu et al. (16) demonstrated that, although ChatGPT offers practical benefits in generating medical content, issues of accuracy and currency remain. Giannakopoulos et al. (17) reported that large language models provide only limited reliability in the context of evidence-based dentistry. While Hwang et al. (18) highlighted the contribution of deep learning applications in dentistry, Yu et al. (19) emphasized that, despite the growing role of AI in clinical decision support systems, concerns regarding reliability and bias persist. Likewise, Topol (20) highlighted the transformative potential of AI in healthcare delivery but underscored the need to address ethical and transparency-related challenges. Taken together with our findings in pediatric dentofacial health, these observations support a cautious interpretation of AI-based conversational agents as adjunctive educational tools rather than as independent clinical decision-making resources.

One of the factors to consider when interpreting information generated by large language models is “hallucination.” These models can occasionally generate incorrect or unsubstantiated information in a convincing manner; this can pose a problem, particularly when answering questions related to medicine or dentistry (21, 22). Users may not always recognize such errors, especially when the provided response appears clear and authoritative. Although our study did not specifically evaluate hallucinations, the variability in reliability scores indicates that health information generated by AI should be treated with caution and, when necessary, verified by healthcare professionals.

Pediatrics is an important field for the evaluation of digital advisors, particularly because parental education and child health literacy may influence healthcare-seeking behavior and decision-making (23). Kubb and Foran (23), as well as Walsh et al. (24), demonstrated that parents frequently use online health information, which can significantly influence their decision-making processes. Baumann et al. (25) reported that parents seek child health information through multiple sources, including digital media, print media, and personal contacts, and that this behavior may vary according to whether the concern involves general child health and development or acute illness. Similarly, Min et al. (26), in a pediatric medical chatbot project developed in Singapore, emphasized that such tools may provide accurate, empathetic, and accessible information in urgent care contexts, although improvements are needed to better accommodate health literacy. DeWitt et al. (27) reviewed mobile health applications and noted that, although content was generally user-friendly, enhancements in readability and structure were required. Likewise, Klapow et al. (28) found that chatbot initiatives targeting parents were valuable for educational purposes but suggested that improvements in literacy adaptation and accuracy could further enhance user experience. Collectively, these findings underscore the growing role of digital tools in pediatric counseling. By directly including parents as evaluators, this study adds a parental perspective to assessing AI-generated pediatric dentofacial health information, although further research is needed to confirm and expand these findings.

Study Limitations

This study has several limitations. First, because the analysis was based on a limited number of queries, the findings may not reflect Grok AI’s performance across the full spectrum of pediatric dentofacial health topics. Second, only one AI-based conversational agent was evaluated, and the lack of transparency regarding Grok AI’s training data and update frequency may have influenced the accuracy and reproducibility of its responses. Third, the evaluations were subjective, and participants’ varying levels of knowledge and experience may have affected the scoring. Moreover, inter-rater reliability was not assessed, which may limit interpretation of consistency across evaluators. The single-center design of the study may also limit the generalizability of the findings. Although the mDISCERN and GQS instruments are commonly used to assess the reliability and quality of online health information, both remain partially dependent on evaluators’ interpretation. As the study was cross-sectional and based on hypothetical queries, it may not fully capture real-life interactions between parents, clinicians, and AI-based tools. In addition, each query was evaluated using a single generated response, and variations in Grok AI’s responses across repeated sessions were not assessed. The evolving nature of large language models also means that responses may change over time, which could further limit reproducibility.

Future research should include comparative analyses of different AI models, larger and more diverse international samples, and direct assessments of patient experiences. In particular, the role of AI-based tools in parental education for pediatric populations warrants further investigation. Such efforts may clarify the potential contribution of AI-supported counseling to preventive care in addressing dentofacial health issues during childhood.

CONCLUSION

This study demonstrated that Grok AI provided responses with moderate-to-good reliability in the context of pediatric dentofacial health, as evaluated by orthodontists, pediatricians, and parents. These findings suggest that AI-based conversational agents may serve as supportive tools in parental education and preventive pediatrics. However, they remain insufficient for clinical decision-making, and further validation in larger and more diverse settings is warranted.