Can Journal Impact Predict the Trustworthiness of Individual Studies in Assisted Reproduction? Development and Pilot Evaluation of the TRUST-PAPER Framework

Jose G. Franco Jr; Claudia G Petersen; Laura D. Vagnini; Fabiana C. Massaro; Bruna Petersen; Andreia Nicoletti; Juliana Ricci; Camila Zamara; Renata A. Pouza; Bianca C. Matuella; Elisangela V. Espirito-Santo; Joao B. Meziara; Antonio H. Oliani; Joao Batista A. Oliveira

JBRA Assist. Reprod. 2026 2026;00(0):00-00
ORIGINAL ARTICLE
doi: 10.5935/1518-0557.20260123

Can Journal Impact Predict the Trustworthiness of Individual Studies in Assisted Reproduction? Development and Pilot Evaluation of the TRUST-PAPER Framework

Jose G. Franco Jr^1,2, Claudia G Petersen^1,2, Laura D. Vagnini^2,3, Fabiana C. Massaro¹, Bruna Petersen¹, Andreia Nicoletti¹, Juliana Ricci¹, Camila Zamara¹, Renata A. Pouza¹, Bianca C. Matuella¹, Elisangela V. Espirito-Santo¹, Joao B. Meziara¹, Antonio H. Oliani^3,4,5, Joao Batista A. Oliveira^1,2

¹Center for Human Reproduction Prof. Franco Jr, Ribeirao Preto, Brazil
²Paulista Center for Diagnosis - Research and Training, Ribeirao Preto, Brazil
³Department of Gynecology and Obstetrics, Sao Jose do Rio Preto School of Medicine (FAMERP), Sao Jose do Rio Preto, Brazil
⁴University of Beira Interior - Faculty of Health Sciences, Covilha, Portugal
⁵Reproductive Medicine Unit - University Hospital Centre Cova da Beira, Covilha, Portugal

Received March 05, 2026
Accepted August 27, 2026

CORRESPONDING AUTHOR
J.G. Franco Jr, Center for Human Reproduction Prof. Franco Jr, Ribeirao Preto, SP - Brazil
Email: franco@crh.com.br

CONFLICT OF INTEREST
The authors declare no conflict of interest related to this methodological study.

ABSTRACT
Objective: To develop and pilot-test TRUST-PAPER, a structured framework for evaluating the methodological trustworthiness of individual studies in assisted reproduction, and to examine whether papers published in higher-impact journals receive higher TRUST-PAPER and Practice Change Confidence (PCC-PAPER) scores than papers published in lower-impact journals.
Methods: Forty original studies in assisted reproduction were assessed using the frozen TRUST-PAPER framework and its complementary PCC-PAPER scale. Twenty papers originated from higher-impact journals and 20 from lower-impact journals. Journal-impact group allocation was disclosed only after all paper-level scores had been assigned and frozen. The primary comparison was TRUST-PAPER score between groups; PCC-PAPER and the association between both constructs were secondary analyses.
Results: Mean TRUST-PAPER was 72.15±9.36 in higher-impact journals and 56.00±12.08 in lower-impact journals (mean difference 16.15, 95% CI 9.22-23.08; Welch p=0.000035; Mann-Whitney p=0.000136; Hedges’ g=1.46). Mean PCC-PAPER was 3.90±1.25 versus 2.20±1.20 (difference 1.70, 95% CI 0.92-2.48; Welch p=0.000088). TRUST-PAPER and PCC-PAPER were strongly correlated (Spearman rho=0.868, p<0.001). Score distributions overlapped.
Conclusion: Journal impact was associated with higher average methodological trustworthiness and practice-change confidence, but did not reliably determine the trustworthiness of an individual paper. TRUST-PAPER may complement journal-level prestige metrics by providing an auditable paper-level assessment.

Keywords: assisted reproduction, artificial intelligence, evidence appraisal, journal impact, live birth, methodological quality

INTRODUCTION
Clinicians in assisted reproduction face a rapidly expanding literature in which journal prestige, study design, statistical significance, and author conclusions may all influence perceived credibility. Journal-level citation metrics are useful descriptors of visibility, but they were not created to determine whether an individual paper has a valid estimand, an appropriate denominator, adequate bias control, clinically relevant outcomes, or conclusions proportional to the data. The long-standing concern that journal impact factor should not be used as a surrogate for the quality of an individual article is reflected in both methodological criticism and the San Francisco Declaration on Research Assessment (Seglen, 1997; Cagan, 2013).
These limitations are especially relevant in assisted reproductive technology (ART). Outcomes occur along a sequence of biological and clinical gates, and the same intervention can appear more or less favorable depending on whether results are expressed per initiated cycle, oocyte retrieval, embryo assessed, embryo transfer, pregnancy, or patient. International terminology and core-outcome initiatives have emphasized standardized definitions and patient-important outcomes, including live birth, pregnancy loss, neonatal outcomes, and time to pregnancy leading to live birth (Duffy et al., 2020; Zegers-Hochschild et al., 2017).
Established instruments address important but different aspects of research quality. CONSORT (Schulz et al., 2010) and STROBE (von Elm et al., 2007) improve reporting of randomized and observational studies, while RoB 2 (Sterne et al., 2019) and ROBINS-I (Sterne et al., 2016) focus on risk of bias. They are not intended to provide a single ART-specific, decision-oriented score integrating estimand alignment, denominator integrity, clustering, critical-flaw ceilings, and practice-change confidence.
TRUST-MA was developed as an auditable framework for evaluating meta-analyses and separating methodological trustworthiness from Practice Change Confidence. Its conceptual architecture was informed by established systematic-review and evidence-appraisal frameworks, including PRISMA (Page et al., 2021), AMSTAR 2 (Shea et al., 2017), ROBIS (Whiting et al., 2016), and GRADE (Guyatt et al., 2008). TRUST-PAPER extends this architecture to the individual scientific article. Its research prototype uses ten weighted domains totaling 100 points, ten critical-flaw flags, ART-specific interpretation rules, and a complementary PCC-PAPER scale.
Recent studies have also shown that large language models can support structured risk-of-bias and critical-appraisal tasks, while emphasizing the need for reproducibility, human oversight, and transparent methodological constraints (Forero et al., 2025; Lai et al., 2024; Hasan et al., 2024; Lai et al., 2025). The present pilot study tested the discriminant behavior of TRUST-PAPER in 40 original ART studies. We hypothesized that papers from higher-impact journals would have higher average TRUST-PAPER scores, while journal prestige alone would remain insufficient to determine the reliability of an individual article.

MATERIALS AND METHODS

Study design and sample
This was a methodological pilot discriminant-validity study of 40 original papers in assisted reproduction. Papers were evaluated in eight consecutive batches of five. After all evaluations were completed and scores frozen, the journal-impact allocation code was opened: batches 1, 4, 6, and 8 comprised the higher-impact group (n=20), whereas batches 2, 3, 5, and 7 comprised the lower-impact group (n=20). Journal prestige was not used as a scoring criterion. The complete frozen 40-paper scoring dataset is provided in Supplementary Table S1. This dataset and the complete TRUST-PAPER and PCC-PAPER instruments used in this study can also be accessed through the QR codes shown in Figure 1. The study workflow is summarized in Figure 2.

Figure 1. QR codes providing access to the frozen 40-paper scoring dataset and the complete TRUST-PAPER and PCC-PAPER instruments used in this study via ChatGPT resources.

Figure 2. Study workflow. Journal-impact allocation was disclosed only after all paper-level TRUST-PAPER and PCC-PAPER scores had been assigned and frozen.

TRUST-PAPER
Before scoring, the evaluator applied an Estimand Lock identifying population, intervention or exposure, comparator, outcome, unit of analysis, denominator, and principal clinical claim. Outcome Hierarchy and Denominator Locks were then applied. TRUST-PAPER awards up to 100 points across ten domains. Score-based classifications are A/high (85-100), B/good (70-<85), C1/low (50-<70), C2/very low (25-<50), and C3/minimal (0-<25). Critical flaws are evaluated independently after the raw score; confirmed flaws may cap the final category, whereas uncertain flaws require a provisional P marker. The prototype explicitly prohibits rewarding statistical significance or treating non-significance as equivalence. The TRUST-PAPER domains and their maximum scores are summarized in Table 1. A summary of the TRUST-PAPER framework is provided in Supplementary Appendix S1.

Table 1. TRUST-PAPER
Note: ART, assisted reproductive technology; ITT, intention-to-treat

ART-specific safeguards and critical flaws
TRUST-PAPER explicitly evaluates denominator distortion, post-assignment selection, uncontrolled confounding, clustering and unit-of-analysis errors, multiplicity-driven claims, unsupported equivalence/non-inferiority claims, internal data inconsistency, PGT-A strategy fallacy, and definitive patient-level claims derived only from laboratory surrogates. Live birth and cumulative live birth outrank implantation, clinical pregnancy, and laboratory endpoints when the claim concerns the chance of taking home a baby. PGT-A studies must distinguish efficacy per transfer from efficacy of the testing strategy per woman, cycle, or retrieval.

PCC-PAPER
PCC-PAPER is independent of the TRUST-PAPER total and ranges from 0 to 10. Five dimensions are scored 0-2: clinical relevance, precision and magnitude, internal validity and robustness, information size, and replication/applicability. Scores of 0-3 indicate insufficient evidence for routine change; 4-6 selective use/shared decision; 7-8 sufficient evidence to influence practice in appropriate patients; and 9-10 strong practice-changing evidence. A single paper should rarely reach 9-10 without exceptional size, validity, and external confirmation. The PCC-PAPER scoring anchors are provided in Supplementary Appendix S2.

Score freezing and statistical analysis
TRUST-PAPER was prospectively treated as the frozen scoring version for this pilot study. No domain weights, category thresholds, ART-specific rules, or critical-flaw definitions were modified after application began. All 40 TRUST-PAPER and PCC-PAPER scores were frozen before disclosure of higherversus lower-impact allocation. The primary endpoint was the between-group difference in TRUST-PAPER score. Continuous results are summarized by mean±SD, median, and range. Groups were compared using Welch's t test and Mann-Whitney U test. The mean difference was reported with a Welch 95% confidence interval, and standardized separation with Hedges’ g. PCC-PAPER was analyzed similarly. The association between TRUST-PAPER and PCC-PAPER across all 40 papers was explored with Spearman and Pearson correlations. Two-sided p<0.05 was considered statistically significant. Because only published scientific articles were evaluated and no patient-level data were used, ethics approval and informed consent were not applicable. The complete statistical analysis is presented in Supplementary Table S2.

RESULTS

Primary TRUST-PAPER comparison
Higher-impact journal papers had a mean TRUST-PAPER score of 72.15±9.36 (median 73.5; range 43-84) compared with 56.00±12.08 (median 56.5; range 34-79) in lower-impact journals. The mean difference was 16.15 points (95% CI 9.22-23.08; Welch t=4.73, df=35.77, p=0.000035). The non-parametric analysis confirmed the separation (Mann-Whitney U=341.5, p=0.000136). Hedges’ g was 1.46, indicating a large standardized difference. Individual scores and the overlap between journal-impact groups are shown in Figure 3. The between-group summary statistics are presented in Table 2.

Table 2. Between-group comparison of TRUST-PAPER and PCC-PAPER scores.
Note: Group values are means±standard deviations. Differences are higher-impact minus lower-impact group means. CI, confidence interval. PCC-PAPER, Practice Change Confidence scale for individual papers.

Figure 3. Individual TRUST-PAPER scores by journal-impact group. The distributions were clearly shifted but overlapped, demonstrating that journal group was not a perfect classifier of individual-paper trustworthiness. Blue and orange dots represent individual papers from higherand lower-impact journals, respectively. Green triangles indicate group means. Horizontal lines within the boxes indicate medians, and the lower and upper box boundaries represent the first and third quartiles (interquartile range, IQR). Whiskers extend to the most extreme values within 1.5 × IQR of the box boundaries. Open circles identify observations beyond the whiskers; these observations are also displayed as colored dots and do not represent additional papers.

Category distribution
Fourteen of 20 higher-impact papers were category B, compared with 3 of 20 lower-impact papers. Lower-impact journals contained 10 C1 and 7 C2 papers, versus 5 C1 and 1 C2 in the higher-impact group. No paper reached category A or C3. The category distribution by journal-impact group is shown in Figure 4.

Figure 4. Distribution of TRUST-PAPER categories in higherand lower-impact journal groups. Score-based classifications are A/high (85-100), B/good (70-<85), C1/low (50-<70), C2/very low (25-<50), and C3/minimal (0-<25).

PCC-PAPER and relationship with TRUST-PAPER
Mean PCC-PAPER was 3.90±1.25 in higher-impact journals and 2.20±1.20 in lower-impact journals (difference 1.70, 95% CI 0.92-2.48; Welch p=0.000088; Mann-Whitney p=0.000273). Across all 40 papers, TRUST-PAPER and PCC-PAPER were strongly correlated (Spearman rho=0.868, p=3.89×10^-13; Pearson r=0.865, p=6.28×10^-13), but the constructs were not identical. The relationship between TRUST-PAPER and PCC-PAPER is shown in Figure 5.

Figure 5. Relationship between methodological trustworthiness and practice-change confidence across the 40 papers.

DISCUSSION
The main finding is that papers published in the higher-impact journal group had substantially higher average TRUST-PAPER scores, with a 16.15-point difference and a large standardized effect. PCC-PAPER showed a parallel separation. Nevertheless, individual score distributions overlapped. This is the clinically and methodologically important result: journal impact and paper trustworthiness were associated at group level but were not interchangeable at article level.
This finding is consistent with the conceptual criticism of using journal impact factor as a surrogate for individual-paper quality (Seglen, 1997; Cagan, 2013). A journal-level metric summarizes citation behavior across a publication venue; TRUST-PAPER instead asks whether the specific paper aligns its question, design, denominator, analysis, and conclusion. The present study therefore does not argue that journal impact is meaningless. Rather, it suggests that prestige contains some information at group level while remaining insufficient for the appraisal of an individual clinical study.
The ART-specific architecture appears particularly relevant. Several recurrent vulnerabilities identified during the pilot were not exotic statistical problems but clinically consequential framing problems: excluding patients who never reached transfer, conditioning analyses on pregnancy or eventual live birth, interpreting a per-transfer PGT-A result as superiority of the overall testing strategy, ignoring repeated embryos or cycles within a woman, deriving predictive cut-offs and evaluating them in the same sample, and promoting laboratory improvements as evidence of improved live birth. These are precisely the situations in which a structured Denominator Lock and Outcome Hierarchy Lock can alter interpretation.
A second recurring problem was the conversion of non-significance into equivalence. TRUST-PAPER distinguishes a properly designed non-inferiority or equivalence trial from an ordinary comparison that simply fails to reject the null hypothesis. This distinction is important because a wide confidence interval can remain compatible with clinically meaningful benefit or harm even when p>0.05.
The strong correlation between TRUST-PAPER and PCC-PAPER supports preliminary convergent behavior but also confirms that they should remain separate. TRUST-PAPER asks whether the study's principal inference is methodologically trustworthy. PCC-PAPER asks whether the result is sufficiently clinically relevant, precise, robust, adequately sized, replicated, and applicable to influence practice. A rigorous trial centered on a surrogate endpoint may therefore receive a high TRUST-PAPER score but only moderate PCC-PAPER.
The study has important limitations. First, this is a 40-paper pilot sample and not a definitive validation cohort. Second, the higherversus lower-impact contrast was simplified and journal impact factor was used as an external comparator, not as a gold standard for methodological quality. Third, design mix may differ between journal groups; higher-impact journals may preferentially publish randomized trials, larger multicenter studies, or prospectively registered research. The present comparison therefore demonstrates association, not causation. Fourth, the initial pilot relied on a single structured AI evaluator; inter-AI and human-AI reproducibility of TRUST-PAPER itself remains to be tested prospectively. Fifth, the instrument's domain weights, category thresholds, and critical-flaw ceilings are provisional and require external calibration.
These limitations define the next validation phase. TRUST-PAPER should be prospectively frozen and applied independently by multiple AI systems and human methodologists to a larger sample matched where feasible by design, year, topic, and sample size. Primary validation endpoints should include total-score ICC, weighted kappa for categories, item-level agreement, critical-flaw agreement, test-retest stability after model updates, convergent comparison with RoB 2/ROBINS-I where appropriate, and discriminant analyses across journals and specialties. Established guidance for ICC, concordance, Bland-Altman agreement, and kappa statistics can support this reproducibility phase (Koo & Li, 2016; Lin, 1989; Bland & Altman, 1986; McHugh, 2012). TRUST-PAPER is not proposed as a replacement for CONSORT, STROBE, RoB 2, ROBINS-I, peer review, or clinical judgment. Its proposed role is integrative: to make the path from the paper's estimand and denominator to its clinical conclusion explicit, reproducible, and auditable.

CONCLUSION
In this 40-paper pilot, studies from higher-impact journals had substantially higher average TRUST-PAPER and PCC-PAPER scores than studies from lower-impact journals. However, journal prestige did not reliably determine the trustworthiness of an individual article. TRUST-PAPER may complement journal-level metrics by providing a structured, ART-specific, auditable assessment of the paper itself. External validation, inter-rater reproducibility, and prospective calibration are required before routine use.
Take-home message: Journal impact and paper trustworthiness are related, but they are not interchangeable constructs.

Artificial-intelligence disclosure
ChatGPT was the structured methodological evaluator applying the prespecified TRUST-PAPER and PCC-PAPER frameworks and was also used to support statistical coding, figure generation, language editing, and manuscript drafting. The authors reviewed the evaluations and analyses, verified the source data, and assume full responsibility for the final content.

Supplementary Table S1. Frozen 40-paper scoring dataset. Study IDs preserve the original blinded batch order. Score-based classifications are A/high (85-100), B/good (70-<85), C1/low (50-<70), C2/very low (25-<50), and C3/minimal (0-<25)

Supplementary Table S2. Final statistical analysisA. Descriptive statistics by journal-impact group
Note: SD, standard deviation; CI, confidence interval; df, degrees of freedom. Mean differences are higher-impact minus lower-impact group means. Between-group comparisons include 20 papers per group; correlations use all 40 papers. PCC-PAPER, Practice Change Confidence scale for individual papers. The dash indicates a value not reported in the source table.

Supplementary Appendix S1. TRUST-PAPER summaryStatus: research prototype; assisted reproduction; not externally validated. Core rules: score explicit documentary evidence rather than overall impression; lock estimand, outcome hierarchy and denominator before scoring; do not reward statistical significance; do not convert non-significance into equivalence; assess CF1-CF10 independently; apply critical-flaw ceilings after the raw score; and retain a P marker for unresolved critical issues.

Supplementary Appendix S2. PCC-PAPER anchors
Interpretation: 0-3 insufficient for routine change; 4-6 selective use/shared decision; 7-8 sufficient to influence practice in appropriate patients; 9-10 strong practice-changing evidence.