JBRA Assist. Reprod. 2026 2026;00(0):00-00
ORIGINAL ARTICLE
doi: 10.5935/1518-0557.20260148
Reproducibility of the TRUST-MA Framework for Assessing the Methodological and Clinical Reliability of Meta-Analyses: An Independent Dual-AI Evaluation
1Center for Human Reproduction Prof. Franco Jr, Ribeirao Preto, Brazil
2Paulista Center for Diagnosis - Research and Training, Ribeirao Preto, Brazil
3Department of Gynecology and Obstetrics, Sao Jose do Rio Preto School of Medicine (FAMERP), Sao Jose do Rio Preto, Brazil
4University of Beira Interior - Faculty of Health Sciences, Covilha, Portugal
5Reproductive Medicine Unit - University Hospital Centre Cova da Beira, Covilha, Portugal
CONFLICT OF INTEREST
The authors declare no conflict of interest related to this methodological study.
ABSTRACT
Objective: To assess whether the TRUST-MA framework produces reproducible methodological and clinical reliability judgments when independently applied by two large language models.
Methods: In this pilot calibration study, 51 reproductive-medicine meta-analyses published in 2025- 2026 were randomly selected from PubMed. ChatGPT (AI-X) and Claude (AI-Y) independently received the same full articles and available supplements and applied TRUSTMA in July 2026, without access to each other’s outputs. Practice Change Confidence (PCC) was introduced during the sequence and was available for a consecutive nested subgroup of 24 cases (cases 27-50). Agreement was evaluated using paired comparisons, intraclass correlation coefficients (ICC), Lin’s concordance correlation coefficient, Bland-Altman analysis, and weighted Cohen’s kappa.
Results: Mean TRUST-MA scores were 56.21±19.90 for ChatGPT and 64.28±18.22 for Claude. Claude scored 8.07 points higher on average (95% confidence interval [CI], 5.33-10.82; p<0.001). Absolute-agreement ICC was 0.800 (bootstrap 95% CI, 0.698-0.868). Exact A/B/C agreement was 68.6%, with linear weighted κ=0.451. PCC means were identical (3.54 for both models; Wilcoxon p=0.794), with ICC=0.844 (bootstrap 95% CI, 0.541-0.955), exact fivetier agreement of 66.7%, and linear weighted κ=0.653. TRUST-MA and PCC scores were correlated for each evaluator (Spearman ρ=0.677 and 0.631; both p<0.001).
Conclusion: TRUST-MA and PCC showed good scorelevel reproducibility across independent AI evaluators. However, systematic calibration differences in TRUST-MA scores and only moderate categorical agreement preclude treating the models as interchangeable. Standardized version reporting, threshold rules, and human adjudication remain necessary.
Keywords: artificial intelligence, evidence-based medicine, meta-analysis, reproducibility, risk assessment, systematic review
INTRODUCTION
Systematic reviews and meta-analyses occupy a central position in evidence-based medicine, yet their conclusions are only as dependable as their design, conduct, statistical synthesis, reporting, and clinical interpretation. Reporting guidelines such as PRISMA 2020 improve transparency, whereas appraisal frameworks such as AMSTAR 2 and ROBIS address methodological quality and risk of bias, and GRADE addresses certainty in a body of evidence (Guyatt et al., 2008; Page et al., 2021; Shea et al., 2017; Whiting et al., 2016). These instruments answer related but nonidentical questions. None was designed specifically to integrate documentary auditability, statistical coherence, clinically appropriate denominators, critical-flaw ceilings, and a decision-oriented interpretation for reproductive medicine.
Large language models (LLMs) may accelerate structured evidence appraisal, but their judgments remain sensitive to prompt architecture, source-document accessibility, model calibration, and interpretive ambiguity. Studies using structured prompts have reported promising performance for risk-of-bias assessment, while others have identified only modest agreement with expert appraisals or systematic tendencies to overestimate confidence (Forero et al., 2025; Hasan et al., 2024; Lai et al., 2024, 2025). Consequently, a reproducible instrument must constrain the evaluation process and make disagreements traceable rather than relying on the model’s overall impression.
The Trustworthiness Assessment of Meta-Analyses (TRUST-MA) was developed as an auditable framework for evaluating the methodological trustworthiness of metaanalyses. Its current calibration branch, TRUST-MA, uses ten weighted domains, a 100-point architecture, explicit documentary anchors, an auditability index, eight criticalflaw ceilings, and decision-oriented categories. The Practice
Change Confidence (PCC) complements TRUST-MA by asking a different question: whether the available evidence is sufficient to justify changing clinical practice.
The present study did not aim to determine whether ChatGPT or Claude is the superior model. Instead, the models were used as independent methodological auditors to test whether the same predefined framework generates reproducible results across different LLMs. We evaluated score-level agreement, categorical agreement, systematic calibration bias, and the relationship between methodological trustworthiness and confidence for practice change.
MATERIALSANDMETHODS
Study design and sample
This was a methodological pilot calibration and interevaluator reproducibility study. Fifty-one meta-analyses in reproductive medicine, published during 2025-2026, were randomly selected from PubMed. All evaluations were conducted in July 2026. The unit of analysis was the metaanalysis. The full published article and its available supplementary materials constituted the document package supplied to each evaluator.The PubMed sampling frame, search syntax, and computational procedure used to randomize records were not prospectively archived. The sample should therefore be interpreted as a randomly chosen pragmatic sample rather than a reproducible systematic sample of all reproductivemedicine meta-analyses published during the period.
Independent artificial-intelligence evaluation
Two general-purpose LLM interfaces were used: ChatGPT (OpenAI; AI-X) and Claude (Anthropic; AI-Y). Each model received the same meta-analysis, the same available supplements, and the same TRUST-MA instructions. Assessments were performed independently; neither evaluator had access to the other model’s output. Scores reported before any adjudication were used for the reproducibility analyses. Exact underlying model build identifiers were not recorded and no model was treated as a reference standard.
TRUST-MA
TRUST-MA is a pilot calibration instrument and has not yet undergone external validation. It awards points only for explicit, locatable documentary anchors; overall impressions are prohibited. The instrument contains ten weighted domains totaling 100 points: question-protocoleligibility (10), search-selection-completeness (10), data extraction/integrity/overlap (10), risk of bias of primary studies (12), clinical coherence-outcomes-denominators (12), statistical methods-synthesis (16), heterogeneitytransportability (10), robustness missing evidence (8), certainty applicability clinical relevance (6), and conclusion spin transparency (6).Item responses are coded as complete, partial, no, not applicable, not verifiable because of access limitations, or not verifiable because conduct was not reported. The standardized score equals 100 multiplied by points obtained divided by applicable points. The auditability index measures the proportion of applicable points supported by a locatable citation or correctly identified report limitation; auditability below 80% renders an assessment provisional.Eight prespecified critical flaws impose numeric score ceilings: mismatch between question or denominator and conclusion; clinically incompatible pooling; material statistical error; uncorrected participant duplication; probable omission or outcome-directed selection; failure to assess or incorporate risk of bias; contradiction between conclusion and estimate; and materially incorrect extracted data. Final classifications are A/high (85-100), B/good (70-<85), C1/low (50-<70), C2/very low (25-<50), and C3/minimal (0-<25). For the primary categorical analysis, the three C sublevels were combined because the original dataset recorded A/B/C. A prudence rule provisionally assigns the lower category when the score lies within two points of a threshold or auditability is below 80%. QR codes linking to the Claude and ChatGPT resources associated with TRUSTMA are provided in Figure 1.
Figure 1. QR codes providing access to the paired score dataset and the complete TRUST-MA and PCC instruments used in this study via the Claude and ChatGPT resources.
PCC
PCC is a complementary 0-10 scale comprising five domains scored 0-2: clinical relevance of the outcome, statistical robustness, clinical consistency, robustness of the primary evidence, and independent replication. Scores of 9-10 indicate a strong recommendation for practice change; 7-8, evidence sufficient for adoption in most patients; 5-6, selective use with shared decision-making; 3-4, insufficient evidence for routine adoption; and 0-2, no change to current practice. PCC was introduced during the sequential project and was available for a consecutive nested subgroup of 24 meta-analyses (cases 27-50). PCC reproducibility was therefore evaluated in an exploratory secondary analysis. Case 51 had been assessed before PCC was introduced.
Outcomes and statistical analysis
The primary outcome was absolute agreement in total TRUST-MA scores, quantified by a two-way randomeffects, single-measure ICC for absolute agreement (Koo & Li, 2016). Secondary outcomes included mean paired difference, Pearson and Spearman correlations, Lin’s concordance correlation coefficient (Lin, 1989), Bland-Altman bias and 95% limits of agreement (Bland & Altman, 1986), exact category agreement, and linearly and quadratically weighted Cohen’s kappa (McHugh, 2012). Normality of paired differences was examined using the Shapiro-Wilk test. A paired t test was used for normally distributed TRUST-MA differences, with Wilcoxon signed-rank analysis as confirmation. PCC differences were non-normal and were primarily assessed using the Wilcoxon signed-rank test. Nonparametric 95% CIs for ICC, concordance coefficient, weighted kappa, and exact agreement were obtained using 10,000 case-resampling bootstrap replicates with a fixed seed. Associations between TRUST-MA and PCC were exploratory and evaluated with Spearman correlation. A two-sided p value <0.05 was considered statistically significant.Because no participants, patient-level data, or private health information were involved, research-ethics approval and informed consent were not applicable.
RESULTS
TRUST-MA score reproducibility
All 51 meta-analyses had paired TRUST-MA scores. ChatGPT produced a mean score of 56.21±19.90 and Claude 64.28±18.22. The mean paired difference (Claude minus ChatGPT) was 8.07 points (95% CI, 5.33-10.82). Differences were compatible with normality (Shapiro-Wilk p=0.648), and the paired t test showed a significant systematic difference (t=5.91, df=50, p<0.001). The Wilcoxon analysis confirmed the result (p<0.001). The paired standardized effect was large (Cohen dz=0.83) (Table 1).
Table 1. Reproducibility results for TRUST-MA and PCC
CI, confidence interval; CCC, concordance correlation coefficient; ICC, intraclass correlation coefficient; PCC, Practice Change Confidence; SD, standard deviation. Confidence intervals in parentheses for ICC, CCC, exact agreement, and κ were obtained by 10,000 bootstrap replicates.
Figure 2. Paired TRUST-MA scores from ChatGPT and Claude
Figure 3. Bland-Altman agreement for TRUST-MA and PCCSolid lines indicate mean paired bias and dashed lines the 95% limits of agreement. TRUST-MA shows a positive Claude bias; PCC shows no mean bias.
Categorical agreement
Using the original A/B/C assignments, exact agreement occurred in 35 of 51 cases (68.6%; bootstrap 95% CI, 54.9%-80.4%). Simple Cohen’s κ was 0.364, linear weighted κ was 0.451 (95% CI, 0.265-0.618), and quadratic weighted κ was 0.568. No disagreement crossed directly between A and C. ChatGPT classified 35 reviews as C, 14 as B, and two as A; Claude classified 31 as C, 16 as B, and four as A (Table 2).
Table 2. Cross-classification of original TRUST-MA A/B/C categories
Values represent numbers of meta-analyses. Rows indicate ChatGPT classifications; columns indicate Claude classifications. A, high trustworthiness; B, good trustworthiness; C, combined low, very low, and minimal trustworthiness categories (C1-C3). Original pre-adjudication classifications are presented.
PCC exploratory subgroup
PCC was available for 24 consecutive cases. Both models had an identical mean of 3.54, with no systematic difference (median paired difference, 0; Wilcoxon p=0.794). Pearson correlation was 0.838 and Spearman correlation was 0.763 (both p<0.001). Absolute agreement was good (ICC=0.844; bootstrap 95% CI, 0.541-0.955), although the interval was wide because of the smaller sample. Lin’s coefficient was 0.838, Bland-Altman bias was 0.00, and limits of agreement were -2.38 to 2.38 points (Figure 3).Exact agreement across the five PCC decision tiers was 66.7%. Simple κ was 0.508, linear weighted κ was 0.653 (bootstrap 95% CI, 0.353-0.840), and quadratic weighted κ was 0.796. TRUST-MA and PCC were positively but incompletely associated: Spearman ρ=0.677 for ChatGPT and 0.631 for Claude (both p<0.001).
DISCUSSION
In this pilot calibration study, the same structured TRUST-MA framework applied independently by ChatGPT and Claude produced good score-level reproducibility across 51 recent reproductive-medicine meta-analyses. The result supports the central premise that explicit anchors, sourcelocation requirements, critical-flaw ceilings, and a standardized report structure can constrain otherwise open-ended LLM judgment. Nevertheless, Claude assigned scores approximately eight points higher than ChatGPT, and A/B/C agreement was only fair to moderate. The models therefore cannot be considered interchangeable.
The distinction between correlation and agreement is central. A Pearson correlation of 0.873 shows that both models tended to rank reviews similarly; it does not show that they assigned the same absolute scores. ICC, Lin’s coefficient, and Bland-Altman analysis jointly revealed good overall reproducibility combined with a clinically meaningful calibration shift. An eight-point difference can move a review across the 70-point boundary between C1 and B or across the 85-point boundary between B and A. Consequently, any implementation that uses categories for clinical decisions must preserve the exact scoring prompt, model identity, version, document package, and rule application.
The categorical findings also expose a specific operational issue. TRUST-MA’s prudence rule is intentionally conservative near decision thresholds, but the pre-adjudication records suggest that the rule was not applied uniformly. This is not merely a data-cleaning problem. It demonstrates why threshold logic should be made deterministic and why the framework’s mandatory adjudication procedure requires comparison at item level rather than averaging total scores. Future software implementations should calculate the score, critical-flaw cap, auditability modifier, prudence rule, and final category algorithmically after the evaluator has completed only the evidence-linked item judgments.
PCC behaved differently. In the 24-case nested subgroup, mean scores were identical and the ICC was good, with no systematic Bland-Altman bias. Weighted categorical agreement was higher than for TRUST-MA. PCC contains only five tightly anchored domains and a 0-10 range, which may reduce opportunities for model-specific accumulation of small scoring differences. The result is encouraging but exploratory: PCC was introduced during the evaluation sequence, the subgroup was smaller, and its confidence intervals were consequently wide.
The moderate-to-strong correlation between TRUST-MA and PCC supports convergent validity at a preliminary level while also confirming that the instruments do not measure the same construct. TRUST-MA asks whether the review’s methodological conclusion is trustworthy and auditable. PCC asks whether the evidence is sufficiently relevant, robust, consistent, and replicated to justify practice change. A methodologically rigorous review may correctly conclude that evidence is uncertain; it should receive methodological credit without automatically generating a high PCC score.
The present work differs from studies designed to compare LLMs against human experts. Prior investigations have shown that structured prompts can permit substantial LLM performance in risk-of-bias assessment, but results vary by model, domain, source quality, and reference standard (Forero et al., 2025; Hasan et al., 2024; Lai et al., 2024, 2025). TRUST-MA is not proposed as a replacement for AMSTAR 2, ROBIS, PRISMA, or GRADE. PRISMA evaluates reporting; AMSTAR 2 appraises review methodology without recommending a numerical total score; ROBIS focuses on risk of bias in the review; and GRADE evaluates certainty for outcomes. TRUST-MA combines documentary scoring, clinical coherence, statistical reproducibility, domainspecific relevance to reproductive medicine, explicit criticalflaw ceilings, and an auditable path from evidence location to decision category. This integration is the proposed innovation, but it requires external validation before clinical adoption.
Strengths and limitations
Strengths include the use of 51 paired evaluations, identical source packages, independent blinded model outputs, complementary continuous and categorical agreement statistics, bootstrap confidence intervals, and retention of pre-adjudication results. The study directly evaluates transportability of the scoring protocol across different LLM families rather than asking which commercial platform is superior.Several limitations require emphasis. First, there was no human expert consensus standard; the study demonstrates reproducibility, not accuracy or validity. Second, exact model build identifiers and model parameters were not recorded, limiting computational replication and exposing the results to model drift. Third, the PubMed search syntax, complete sampling frame, and randomization procedure were not archived. Fourth, only reproductive-medicine meta-analyses published in 2025-2026 were examined. Fifth, item-level scores, auditability indices, and disagreement types were not available in the present dataset, preventing domain-level reliability analysis. Sixth, PCC was introduced after case 26 and remains an exploratory nested analysis. Seventh, TRUST-MA is itself a calibration draft and its numerical weighting, thresholds, critical-flaw ceilings, and clinical consequences require human expert review, prospective testing, and comparison with established frameworks.
Implications and next steps
The next validation phase should prospectively freeze the instrument, prompt, model identifiers, temperature or equivalent settings, document-access rules, and sampling procedure. Each case should retain complete item-level outputs from two AI evaluators and at least two independent human methodologists, followed by prespecified adjudication. Primary endpoints should include domain-level weighted kappa, total-score ICC, agreement with expert consensus, calibration around thresholds, frequency and causes of critical-flaw disagreement, and test-retest stability after model updates. External samples should include other clinical specialties and different meta-analysis designs.
CONCLUSION
TRUST-MA generated good score-level reproducibility when independently applied by ChatGPT and Claude to 51 reproductive-medicine meta-analyses. PCC also showed good reproducibility in a consecutive exploratory subgroup and no systematic inter-model bias. However, the significant upward shift in Claude TRUST-MA scores and only moderate categorical agreement show that a standardized framework does not completely eliminate model-related calibration effects. TRUST-MA and PCC should currently be regarded as auditable research instruments requiring explicit model-version reporting, deterministic threshold implementation, human adjudication, and prospective external validation before routine clinical use.
REFERENCES
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1:307-10. PMID: 2868172. DOI: 10.1016/S0140-6736(86)90837-8.
Medline
Forero DA, Abreu SE, Tovar BE, Oermann MH. Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models. J Am Med Inform Assoc. 2025;32:1471-6. PMID: 40680299. DOI: 10.1093/jamia/ocaf117.
Medline
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ, GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924-6. PMID: 18436948. DOI: 10.1136/bmj.39489.470347.AD.
Medline
Hasan B, Saadi S, Rajjoub NS, Hegazi M, Al-Kordi M, Fleti F, Farah M, Riaz IB, Banerjee I, Wang Z, Murad MH. Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk-of-bias assessment. BMJ Evid Based Med. 2024;29:394-8. PMID: 38383136. DOI: 10.1136/bmjebm-2023-112597.
Medline
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15:155-63. PMID: 27330520. DOI: 10.1016/j.jcm.2016.02.012.
Medline
Lai H, Ge L, Sun M, Pan B, Huang J, Hou L, Yang Q, Liu J, Liu J, Ye Z, Xia D, Zhao W, Wang X, Liu M, Talukdar JR, Tian J, Yang K, Estill J. Assessing the risk of bias in randomized clinical trials with large language models. JAMA Netw Open. 2024;7:e2412687. PMID: 38776081. DOI: 10.1001/jamanetworkopen.2024.12687.
Medline
Lai H, Liu J, Bai C, Liu H, Pan B, Luo X, Hou L, Zhao W, Xia D, Tian J, Chen Y, Zhang L, Estill J, Liu J, Liao X, Shi N, Sun X, Shang H, Bian Z, Yang K, et al. Language models for data extraction and risk of bias assessment in complementary medicine. NPJ Digit Med. 2025;8:74. PMID: 39890970. DOI: 10.1038/s41746-025-01457-w.
Medline
Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45:255-68. PMID: 2720055. DOI: 10.2307/2532051.
Medline
McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22:276-82. PMID: 23092060. DOI: 10.11613/BM.2012.031.
Medline
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. PMID: 33782057. DOI: 10.1136/bmj.n71.
Medline
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. PMID: 28935701. DOI: 10.1136/bmj.j4008.
Medline
Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R, ROBIS Group. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-34. PMID: 26092286. DOI: 10.1016/j.jclinepi.2015.06.005.
Medline