Received August 03, 2026
Accepted August 03, 2026

Objective: To assess whether the TRUST-MA v1.3-CAL framework produces reproducible methodological and clinical reliability judgments when independently applied by two large language models. Methods: In this pilot calibration study, 51 reproductive-medicine meta-analyses published in 2025–2026 were randomly selected from PubMed. ChatGPT (AI-X) and Claude (AI-Y) independently received the same full articles and available supplements and applied TRUST-MA v1.3-CAL in July 2026, without access to each other’s outputs. Practice Change Confidence (PCC) v1.0 was introduced during the sequence and was available for a consecutive nested subgroup of 24 cases (cases 27–50). Agreement was evaluated using paired comparisons, intraclass correlation coefficients (ICC), Lin’s concordance correlation coefficient, Bland–Altman analysis, and weighted Cohen’s kappa. Results: Mean TRUST-MA scores were 56.21±19.90 for ChatGPT and 64.28±18.22 for Claude. Claude scored 8.07 points higher on average (95% confidence interval [CI], 5.33–10.82; p<0.001). Absolute-agreement ICC was 0.800 (bootstrap 95% CI, 0.698–0.868). Exact A/B/C agreement was 68.6%, with linear weighted ?=0.451. PCC means were identical (3.54 for both models; Wilcoxon p=0.794), with ICC=0.844 (bootstrap 95% CI, 0.541–0.955), exact five-tier agreement of 66.7%, and linear weighted ?=0.653. TRUST-MA and PCC were correlated within both evaluators (Spearman ?=0.677 and 0.631; both p<0.001). Conclusion: TRUST-MA and PCC showed good score-level reproducibility across independent AI evaluators. However, systematic calibration differences in TRUST-MA scores and only moderate categorical agreement preclude treating the models as interchangeable. Standardized version reporting, threshold rules, and human adjudication remain necessary.