ai-research-survey

v2-findings.md (8203B)

---

 # V2 Scan Findings: Methodological Quality in Agentic AI/LLM Research
 
 *Updated 2026-03-15 from 467 v2 scans (Opus-rated, two-field boolean instrument)*
 
 ## 1. Overall Quality Distribution
 
 - **N**: 467 papers (all scanned by Opus with v2 instrument)
 - **Mean**: 51.0%
 - **Median**: 50.0%
 - **Std dev**: 16.1%
 - **Range**: 10.0% – 93.3%
 
 | Tier | Count | % |
 |------|-------|---|
 | >= 70% | 55 | 12% |
 | 60-69% | 93 | 20% |
 | 50-59% | 94 | 20% |
 | 40-49% | 107 | 23% |
 | 30-39% | 68 | 15% |
 | < 30% | 50 | 11% |
 
 The typical paper satisfies exactly half of applicable criteria. Median declined from 53.3% (n=135, priority papers) → 55.6% (n=271) → 50.0% (n=467) as less prominent papers entered the pool, confirming that visibility correlates with quality. Numbers are stabilizing — findings consistent across three checkpoints.
 
 ## 2. The Methodological Theater Gap
 
 The headline finding from v2: **papers look like they have evaluations, but the evaluations lack rigor.**
 
 | Layer | Pass Rate | What It Measures |
 |-------|-----------|-----------------|
 | Base: evaluation_design | **86%** | Has baselines, metrics, breakdowns |
 | Conditional: experimental_rigor | **28%** | Seeds, search budgets, self-comparison bias |
 | Conditional: data_leakage | **15%** | Temporal/feature leakage, non-independence |
 | Conditional: survey_methodology | **15%** | PRISMA, quality assessment, publication bias |
 
 Papers have the *form* of rigorous evaluation without the *substance*. 86% include baselines and multiple metrics. But only 26% report seed sensitivity, search budgets, or address self-comparison bias. This is methodological theater.
 
 ## 3. Category Pass Rates
 
 | Category | Pass Rate |
 |----------|-----------|
 | evaluation_design | 86% |
 | setup_transparency | 70% |
 | data_integrity | 65% |
 | claims_and_evidence | 64% |
 | limitations_and_scope | 61% |
 | human_studies | 42% |
 | conflicts_of_interest | 41% |
 | statistical_methodology | 36% |
 | cost_and_practicality | 34% |
 | artifacts | 33% |
 | experimental_rigor | 28% |
 | contamination | 19% |
 | data_leakage | 15% |
 | survey_methodology | 15% |
 
 ## 4. Near-Zero Questions (The Damning Statistics)
 
 These questions are almost universally failed. Each is a single publishable finding.
 
 | Question | Pass Rate | N |
 |----------|-----------|---|
 | self_comparison_bias_addressed | **2.3%** | 3/132 |
 | financial_interests_declared | **3.3%** | 9/271 |
 | multiple_comparison_correction | **3.8%** | 3/80 |
 | hyperparameter_search_budget | **5.2%** | 7/135 |
 | sample_size_justified | **5.2%** | 11/212 |
 | quality_assessment_of_sources (surveys) | **8.3%** | 3/36 |
 | reproduction_instructions | **8.5%** | 21/247 |
 | environment_specified | **11.0%** | 25/227 |
 | pre_registered | **11.4%** | 5/44 |
 | non_independence_addressed | **11.5%** | 14/122 |
 
 **97%** of papers don't declare financial interests. **95%** of benchmark papers don't report hyperparameter search budgets. **91%** don't provide reproduction instructions. **89%** don't specify environments.
 
 ## 5. The Reproducibility Gap
 
 - Code released: **58.6%** (78/133 applicable)
 - Data released: **56.1%** (74/132)
 - Environment specified: **13.4%** (16/119)
 - Reproduction instructions: **10.9%** (14/129)
 - **Full reproducibility (all 4)**: **6.7%** (9/135)
 
 Most papers release code. Almost none tell you how to run it.
 
 ## 6. Statistical Negligence
 
 | Question | Pass Rate | N |
 |----------|-----------|---|
 | Confidence intervals / error bars | **24.0%** | 25/104 |
 | Variance reported | **26.0%** | 27/104 |
 | Significance tests | **31.4%** | 32/102 |
 | Sample size justified | **6.4%** | 7/109 |
 | Effect sizes reported | **84.5%** | 87/103 |
 
 Papers report that "X outperforms Y by 12%" (effect sizes: 84.5%) but don't report whether that 12% is within noise (CIs: 24%, significance tests: 31.4%, variance: 26%). Point estimates without uncertainty are not science.
 
 ## 7. The Contamination Blindspot
 
 | Question | Pass Rate | N |
 |----------|-----------|---|
 | Training cutoff stated | **16.7%** | 10/60 |
 | Benchmark contamination addressed | **33.3%** | 20/60 |
 | Train/test overlap discussed | **40.0%** | 24/60 |
 | Temporal leakage addressed | **39.0%** | 23/59 |
 | Feature leakage addressed | **20.0%** | 12/60 |
 | Non-independence addressed | **9.8%** | 6/61 |
 | Leakage detection method | **31.7%** | 19/60 |
 
 Two-thirds of benchmark papers don't address contamination at all. 90% don't check for non-independence between train and test sets. Results on public benchmarks may be meaningless.
 
 ## 8. Conflicts of Interest
 
 | Question | Pass Rate | N |
 |----------|-----------|---|
 | Affiliations disclosed | **98.5%** | 133/135 |
 | Funding disclosed | **32.6%** | 44/135 |
 | Funder independent of outcome | **24.6%** | 33/134 |
 | Financial interests declared | **1.5%** | 2/135 |
 
 Everyone lists their employer. Almost nobody declares financial interests. Only a quarter of funded papers establish funder independence. In a field where major companies fund research that evaluates their own products, this is a transparency crisis.
 
 ## 9. Scores by Methodology Tag
 
 | Tag | N | Mean | Median |
 |-----|---|------|--------|
 | rct | 9 | 63.4% | 60.3% |
 | observational | 19 | 60.9% | 62.5% |
 | qualitative | 22 | 56.0% | 61.1% |
 | benchmark-eval | 79 | 50.4% | 50.0% |
 | theoretical | 10 | 48.1% | 49.0% |
 | meta-analysis | 25 | 45.2% | 37.5% |
 | case-study | 14 | 43.1% | 41.6% |
 
 **RCTs score highest** (63.4%). The methodology that requires the most effort produces the best quality. **Meta-analyses score lowest** (45.2%) — survey papers that preach methodological rigor don't practice it themselves.
 
 ## 10. Quality Declining Over Time
 
 | Year | N | Mean | Median |
 |------|---|------|--------|
 | 2022 | 1 | 73.2% | 73.2% |
 | 2023 | 14 | 49.3% | 47.1% |
 | 2024 | 28 | 54.7% | 55.3% |
 | 2025 | 65 | 46.6% | 47.3% |
 | 2026 | 26 | 60.6% | 61.4% |
 
 2025 papers (n=65) average 46.6% — **8 points below 2024**. The field is growing faster than its standards. (2026 bump likely reflects selection bias toward higher-profile early-year papers.)
 
 ## 11. What Papers Do Well
 
 | Question | Pass Rate |
 |----------|-----------|
 | affiliations_disclosed | 98.5% |
 | abstract_claims_supported | 97.8% |
 | per_category_breakdown | 97.0% |
 | negative_results_reported | 92.4% |
 | failure_cases_discussed | 90.2% |
 | scaffolding_described | 88.4% |
 | data_collection_described | 88.6% |
 | multiple_metrics | 87.6% |
 
 Papers are good at the visible parts: disclosing affiliations, supporting abstract claims, reporting breakdowns. The failures are in the invisible infrastructure: statistical foundations, contamination controls, reproducibility details, financial transparency.
 
 ## 12. Headline Statistics for the Paper
 
 Across 467 papers in agentic AI/LLM programming research (2022-2026), scanned by Claude Opus 4.6 with a 67-question two-field boolean instrument:
 
 - The median paper satisfies **50%** of applicable methodological criteria
 - **86%** have formal evaluations; **28%** address experimental rigor — methodological theater
 - **97%** don't declare financial interests
 - **95%** don't report hyperparameter search budgets
 - **91%** don't provide reproduction instructions
 - **54%** report effect sizes with zero uncertainty (no CIs, no sig tests, no variance)
 - **84%** of benchmark papers don't address contamination
 - **84%** of code releases are "open source theater" — no env or instructions
 - Only **4%** achieve full reproducibility (code + data + environment + instructions)
 - Contamination awareness **collapsing**: 32.5% (2024) → 12.5% (2026)
 - Data leakage awareness **collapsing faster**: 24.2% (2024) → 9.3% (2026)
 - Even evaluation_design is **eroding**: 88.7% (2024) → 82.3% (2026)
 - One bright spot: statistical_methodology **improving** 30% → 42% (2023→2026)
 - Surveys score **lowest** (43.6%) — papers about rigor lack rigor
 - RCTs score **highest** (64.3%) — effort correlates with quality
 - Funding disclosure predicts quality: **14pp gap** (60% vs 46%)
 - Visibility correlates with quality: prominent papers scored higher than long tail
 - Findings **stable across three checkpoints** (n=135, 271, 467) — these are real