ai-research-survey

methodology.md (14379B)

---

 # Survey Methodology
 
 ## Precedent
 
 This project adapts systematic review methodology from medical research, particularly the Cochrane Review process and PRISMA reporting guidelines. These frameworks provide decades of refinement on how to:
 
 - Define inclusion/exclusion criteria defensibly
 - Score study quality on structured rubrics
 - Minimize reviewer bias through structured extraction
 - Report findings transparently
 
 The key adaptation is that we are reviewing *methodological quality*, not synthesizing effect sizes. We are not doing a meta-analysis of "how much does AI help"; we are asking "how well did each study support its claims."
 
 ## Quality Assessment Instrument
 
 ### Design: 50-question two-field boolean checklist
 
 We use a 50-question checklist with two boolean fields per question rather than subjective Likert-style scores or a three-way yes/no/na enum. This was a deliberate design decision refined across two calibration rounds.
 
 **Each question has:**
 - `applies` (boolean): Is this criterion relevant to this paper type?
 - `answer` (boolean): Does the paper satisfy this criterion? (Only meaningful when applies=true.)
 - `justification` (string): 1-3 sentence explanation citing specific paper sections.
 
 **Why two fields instead of yes/no/na:**
 The original design used a single `answer: yes/no/na` field. Calibration round 1 (93.2% agreement) revealed that 47-56% of Sonnet-Opus disagreements were "NA boundary errors" — the model conflating "the paper didn't do this" (should be no) with "this doesn't apply" (should be na). The two-field design forces explicit, separate decisions on applicability and compliance, eliminating this conflation.
 
 **Design principles:**
 - **Verifiable**: each question has a factually correct answer checkable against the paper
 - **Auditable**: justification text cites specific sections/quotes
 - **High inter-rater reliability**: booleans have much less variance than 0-3 scores
 - **Fast human calibration**: checking 50 booleans takes ~15 min, not hours
 - **Derived scores**: composite scores computed deterministically from boolean counts
 - **Separate denominators**: compliance rates computed only over papers where applies=true
 - **The questions are findings**: "only 34% of papers release code" is concrete and publishable
 
 **Previous designs (discarded):**
 1. 6-dimension 0-3 rubric — discarded because LLM-assigned subjective scores are hard to defend in a paper about methodological rigor.
 2. Single yes/no/na field — discarded after calibration showed NA boundary confusion was the dominant error mode.
 
 ### Categories (11 groups, 50 questions)
 
 1. **Artifacts** (4q): code released, data released, environment specs, reproduction instructions
 2. **Statistical methodology** (5q): CIs/error bars, significance tests, effect sizes, sample size justification, variance
 3. **Evaluation design** (9q): baselines, contemporary baselines, ablation, multiple metrics, human eval, held-out test, breakdowns, failure cases, negative results
 4. **Claims & evidence** (5q): abstract supported, causal claims justified, generalization bounded, alternatives discussed, proxy-outcome distinction
 5. **Setup transparency** (5q): model versions, prompts, hyperparameters, scaffolding, data preprocessing
 6. **Limitations & scope** (3q): limitations section, specific threats, scope boundaries
 7. **Data integrity** (4q): raw data available, collection described, recruitment described, pipeline documented
 8. **Conflicts of interest** (4q): funding disclosed, affiliations disclosed, funder independent, financial interests declared
 9. **Contamination** (3q): training cutoff stated, train/test overlap discussed, contamination addressed
 10. **Human studies** (7q): pre-registered, IRB, demographics, inclusion/exclusion, randomization, blinding, attrition
 11. **Cost & practicality** (2q): inference cost, compute budget
 
 Data integrity and conflicts of interest categories inspired by the Wakefield MMR case — "Is raw data available for independent verification?" would have caught the fabrication years earlier.
 
 ### Conditional modules (v2, 15 questions)
 
 V2 scans add conditional question modules activated by methodology_tags. These target systematic issues identified by meta-research papers in the corpus.
 
 **12. Experimental rigor** (8q, activated by `benchmark-eval`):
 - `seed_sensitivity_reported` — Henderson et al. (2018) showed RL results vary 2x across seeds
 - `number_of_runs_stated` — exact run count, not implicit
 - `hyperparameter_search_budget` — Dodge et al. (2019) showed search budget dramatically affects results
 - `best_config_selection_justified` — selection on validation set, not cherry-picked
 - `multiple_comparison_correction` — Bonferroni/Holm/BH for multiple tests
 - `self_comparison_bias_addressed` — Lucic et al. (2018) showed authors' baseline re-implementations systematically underperform
 - `compute_budget_vs_performance` — performance as function of compute, not just peak
 - `benchmark_construct_validity` — Kapoor & Narayanan (2024) documented widespread validity gaps
 - `scaffold_confound_addressed` — SWE-bench scores vary 2.7–28.3% for the same model depending on scaffold; scaffold effect often exceeds model effect
 
 **13. Data leakage** (4q, activated by `benchmark-eval`):
 - `temporal_leakage_addressed` — training data from after prediction target
 - `feature_leakage_addressed` — input features leak answer information
 - `non_independence_addressed` — train/test share structural similarities
 - `leakage_detection_method` — concrete detection (canary strings, n-gram overlap, etc.)
 
 Source: Kapoor & Narayanan (2024) leakage taxonomy.
 
 **14. Survey methodology** (3q, activated by `meta-analysis`):
 - `prisma_or_structured_protocol` — PRISMA or equivalent systematic protocol
 - `quality_assessment_of_sources` — quality scoring of included studies (Leech et al.)
 - `publication_bias_discussed` — funnel plots, negative-result underrepresentation
 
 Total: 51 base + 16 conditional = 67 max per paper. V1 scans remain valid (new fields optional).
 
 Full schema with evaluation criteria for each question: `schema/scan.schema.json`
 
 ### Answer rules
 
 - **`applies: true, answer: true`** = the paper clearly satisfies the criterion; you can point to where
 - **`applies: true, answer: false`** = the paper does not satisfy the criterion, or evidence is absent. Absence of evidence is `answer: false`, not `applies: false`.
 - **`applies: false, answer: false`** = the criterion is structurally inapplicable to this paper type (e.g., human_studies questions for a benchmark paper, contamination questions for a mining study)
 
 Each answer includes a 1-3 sentence justification citing specific paper sections.
 
 ### Model assignment
 
 - **Primary rater (v1)**: Sonnet — switched to Opus after Round 3 calibration showed persistent Sonnet generosity bias
 - **Primary rater (v2)**: Opus (Claude Opus 4.6) — all scanning from Round 3 onward
 - **Calibration rater**: Opus (independent re-evaluation of subset to measure inter-rater agreement)
 
 ### Calibration results
 
 **Round 1 (2026-02-28, yes/no/na format):** 8 papers, **93.2%** agreement (373/400). Two systematic issues:
 1. NA boundary errors (56% of disagreements): Sonnet confused "didn't do it" with "doesn't apply." Fixed by adding explicit NA guidance per question.
 2. Generosity bias (44%): Sonnet credited partial information. Fixed by adding "does NOT count" examples.
 
 **Round 2 (2026-02-28, yes/no/na format, post-fixes):** 10 papers, **96.2%** agreement (481/500). Improvement confirmed, but NA boundary errors still 47% of remaining disagreements. Led to two-field redesign (applies + answer) to structurally eliminate the conflation.
 
 **Round 3 (2026-02-28, applies/answer format):** 60 papers, **97.0%** agreement (2,911/3,000). Two-field design validated at scale. Remaining disagreements: applies_boundary 52%, sonnet_generous 36%, opus_generous 12%. Sonnet generosity bias persisted, leading to switch to Opus as primary rater.
 
 ## LLM-Assisted Systematic Review
 
 This survey is itself a contribution to the methodology of large-scale systematic reviews. The entire pipeline — from paper harvesting and PDF acquisition through structured quality assessment — is conducted using Claude Opus 4.6 (Anthropic) as the primary evaluation instrument, with human oversight for calibration and editorial judgment.
 
 ### Why this matters
 
 Traditional systematic reviews face a scale ceiling: human reviewers can assess tens to low hundreds of papers before fatigue and inconsistency degrade quality. This survey targets ~1,000 papers with a 65-question structured instrument — infeasible for a small research team using manual review alone.
 
 ### What the LLM does
 
 1. **Structured extraction**: Each paper is read in full and evaluated against the boolean checklist. The LLM cites specific sections, tables, and figures in its justifications — these are auditable by human reviewers.
 2. **Consistency at scale**: Unlike human reviewers who drift over hundreds of papers, the LLM applies the same criteria to paper #1 and paper #900. Calibration data (97% inter-rater agreement at 60 papers) quantifies this consistency.
 3. **Conditional evaluation**: V2 scans activate domain-specific question modules (experimental rigor, data leakage, survey methodology) based on paper type, applying targeted criteria from meta-research literature.
 
 ### What the LLM does not do
 
 - **Editorial judgment**: Decisions about what findings mean, which narrative threads to pursue, and how to position the work are human decisions.
 - **Instrument design**: The 65-question checklist was designed by humans, informed by meta-research literature (Henderson, Dodge, Kapoor, Leech, REFORMS). The LLM's role is to apply the instrument, not design it.
 - **Calibration**: Inter-rater reliability was measured by comparing independent Opus evaluations against prior scans. Disagreement analysis and instrument refinement were human-driven.
 
 ### Transparency
 
 - The scanning model (Claude Opus 4.6), prompt text (`agents/scan-agent.md`), and schema (`schema/scan.schema.json`) are all published as part of the replication package.
 - Every checklist answer includes a justification citing specific paper content, enabling human spot-checking at any granularity.
 - The calibration journey (93.2% → 96.2% → 97.0%) and the specific failure modes discovered (NA boundary confusion, generosity bias) are documented as methodological findings in their own right.
 - V1 vs V2 scan versions are tracked per paper, so analysis can control for instrument version.
 
 ### Limitations of LLM-assisted review
 
 - The LLM cannot verify claims against external sources (e.g., checking if a claimed GitHub repo actually exists and contains working code).
 - Fabricated data would not be detected — the checklist assesses transparency and methodology, not truthfulness of reported numbers (the Wakefield benchmark scored 45.7%, catching transparency gaps but not fabrication).
 - The LLM's training data includes many of the papers being reviewed, creating a potential bias toward charitable interpretation of familiar work. The strict "absence of evidence is false" rule and calibration process mitigate but do not eliminate this.
 
 ## Paper Selection
 
 ### Inclusion Criteria
 - Published 2023 or later (post-GPT-4, when agentic AI research accelerated)
 - Makes empirical claims about AI/LLM capability, productivity, or safety
 - Relevant to software development, code generation, or agentic workflows
 - Available in English
 
 ### Exclusion Criteria
 - Pure opinion pieces or blog posts (no empirical content)
 - Product announcements without methodology
 - Papers focused exclusively on non-code domains (unless methodology is transferable)
 
 ### Sampling Strategy
 - **Seed set**: Papers cited in existing survey documents and well-known references
 - **Forward/backward citation chasing**: Follow citations from seed papers
 - **Venue monitoring**: arXiv cs.SE, cs.AI, cs.CL; major ML conferences (NeurIPS, ICML, ACL)
 - **Community sources**: HuggingFace trending, Semantic Scholar alerts
 
 This is a purposive sample, not a random one. The goal is coverage of the most influential and most cited papers, not statistical representativeness of all papers published.
 
 ## PDF Acquisition Pipeline
 
 PDFs are obtained through a multi-stage automated pipeline before falling back to manual retrieval. All stages are fully documented for transparency in the PRISMA flow diagram.
 
 ### Automated stages (scripts/download-arxiv.py, scripts/download-doi.py)
 
 1. **arXiv direct download** — papers with `arxiv_id` downloaded from `arxiv.org/pdf/<id>.pdf`. Also catches arXiv DOIs (`10.48550/arXiv.*`) where the arXiv ID is embedded in the DOI.
 2. **Semantic Scholar open access** — queries S2 API for open-access PDF URL; also recovers arXiv IDs missed during harvesting.
 3. **Unpaywall** — queries Unpaywall API for green/gold OA versions.
 4. **CORE API** — queries CORE (core.ac.uk) for author manuscripts and institutional repository copies.
 5. **OpenAlex** — queries OpenAlex for additional OA links not indexed by Unpaywall.
 6. **Sci-Hub** — opt-in (`--scihub` flag); parses mirror HTML to find embedded PDF URL.
 
 ### Claude web search stage (scripts/run-pdf-finder.py / Agent tool)
 
 For papers that survive all automated stages without a PDF, Claude agents (Sonnet, WebSearch + WebFetch + Bash) perform targeted web searches:
 - DOI landing page crawl for embedded PDF links
 - Author institutional page and publication list
 - Preprint servers (arXiv, SSRN, bioRxiv, OSF)
 - ResearchGate and Semantic Scholar pages
 - Publisher "free access" or author-accepted-manuscript versions
 
 Each agent writes `papers/<slug>/pdf-finder-result.txt` with `FOUND <url>` or `NOT_FOUND`. The orchestrator updates registry status on success.
 
 **Observed hit rate**: ~50% of papers attempted via web search are found (primarily through author pages and preprint servers). The remaining failures are documented as genuinely paywalled with no open-access version available.
 
 ### Reporting
 
 Papers that could not be obtained are counted in the PRISMA flow diagram under "full text not available." The acquisition method (arXiv, OA repository, author page, etc.) is not tracked per paper but the overall breakdown is available from registry metadata.