JBRA Assist. Reprod. 2025;29(Suppl 1):25-25
POSTER PRESENTATION
doi: 10.5935/1518-0557.20250079
1Cegyr, Eugin Group
Objective:
• Assess the accuracy of ChatGPT's responses to frequently asked fertility-related questions.
• Evaluate the clarity and relevance of ChatGPT's answers regarding common fertility topics.
• Compare the performance of ChatGPT across different domains, including diagnosis, treatment, medication, prognosis, lifestyle, time-related questions, emotional support, and reproductive technologies, in terms of accuracy, clarity, relevance, and readability.
Methods: Infertility is a challenging condition that leads many patients to seek online information about treatments and outcomes. Chatbots like ChatGPT can provide accessible guidance, but their reliability in specialized fields like fertility remains uncertain. Although studies on chatbot quality exist, key aspects-accuracy, clarity, and relevance-remain underexplored. Furthermore, no research has evaluated ChatGPT’s performance in Latin America, where cultural and linguistic nuances may impact its effectiveness. Assessing ChatGPT’s potential in fertility counseling could enhance patient communication and healthcare delivery. This crosssectional study analyzed ChatGPT 4.0’s responses to 50 common fertility-related questions. Responses were evaluated by 10 fertility specialists (5 senior, 5 junior) using the Global Quality Scale (GQS) for a general idea, and Likert scales for accuracy, clarity, and relevance. To measure readability, we analyzed the answers with the Spanish Flesch-Kincaid score. Domain-specific differences were statistically analyzed, providing insights into ChatGPT’s ability to support patient counseling. Fifty frequently asked fertilityrelated questions were selected from patient forums and blogs. ChatGPT 4.0´s responses, generated using prompts requesting answers 'as if ChatGPT were an infertility specialist, using the best available evidence' were evaluated by 10 fertility specialists. Ratings for accuracy, clarity, relevance, and readability were collected using structured tools. Data were analyzed to determine performance differences across domains.
Results: All specialists rated ChatGPT as at least "good" in answering fertility-related questions, with 62% (7/10) classifying its general performance as "very good" or "excellent." In agreement, in the GQS scale (1-5), 94% (47/50) of responses scored ≥3, with an overall mean of 3.6±0.6, reflecting answers generally rated as very good or excellent. All specialists (10/10) agreed ChatGPT could complement specialist counseling. Ratings across domains were consistently above 3, except for infertility diagnosis precision. Domain averages (1-5) showed no significant differences based on specialists' experience (p=NS). Except for infertility diagnosis, which was rated as good, other domains were consistently rated as very good or excellent (mean scores were 4.2±0.4):
• Infertility diagnosis: Precision 3.7, clarity 3.9, relevance 3.9.
• Treatment: Precision 4.0, clarity 4.1, relevance 4.2.
• Medication: Precision 4.0, clarity 4.0, relevance 4.0
• Prognosis: Precision 4.2, clarity 4.2, relevance 4.2
• Lifestyle: Precision 4.2, clarity 4.2, relevance 4.3.
• Time-related questions: Precision 3.9, clarity 4.1, relevance 4.2.
• Emotional support: Precision 4.4, clarity 4.4, relevance 4.5.
• ART technologies: Precision 4.0, clarity 4.2, relevance 4.3.
Readability was scored at 19.6±4.2, corresponding to a 10th-grade reading level, slightly difficult but accessible.
Conclusion: The findings suggest that ChatGPT 4.0 could serve as a valuable tool for initial infertility counseling, providing consistently high ratings for precision, clarity, and relevance. It may complement consultations by offering background information while the doctor validates the content and provides the essential value of human interaction in patient care.The main limitations include the lack of generalizability to other AI-based chatbots and the dependency on prompt design, which significantly impacts the quality of responses. Additionally, findings may not apply to more controversial topics, highlighting the need for further research in diverse contexts.