# Methods, amendments and evidence hierarchy

Use this document together with the archived PROTOCOL.md, PHASE2-PLAN.md, LOG.md and implementation. Original documents are preserved without rewriting their historical claims.

## Authority order

For what ran, prefer per-call records and per-run headers, then the execution code, then the dated journal, then plans. An `invariants` object records intended common settings and sometimes conflicts with the actual fields. Use `responseFormatMode`, `maxTokensSent`, `reasoning`, `temperatureHonoured`, `requestParams` and `endpoint` for the metadata explorer. The stored `requestParams.response_format` is a descriptive string in some files; reconstruct the actual JSON schema from the exported request and runner, rather than sending that string to an API.

For numbers, use this package's JSON tables and define the denominator. `reproduction/reproduce.py` recalculates scores from raw grades, checks against saved scores, verifies the original analysis manifest and verifies all prompt exports. It copies precomputed 5,000-resample paired intervals only after their input hashes are verified. The archived `analyze.py` and `compare-production.py` recompute those intervals from the same records. The publication script independently computes the new weighting, rank-frequency, kappa and length analyses.

## Planned versus completed

| Concern | Planned/documented | What the saved experiment supports |
|---|---|---|
| Number of models | Drafts say 15 or 16; final prose lists 17 | 17 OpenRouter candidate files with 100 calls each |
| All models strict schema | Generic invariants retain strict JSON schema | Four actual JSON-object runs: both DeepSeek models, GLM and Qwen 27B |
| Universal token cap | Generic invariant is 1,536 | Six reasoning-enabled runs have 4,096; eleven have 1,536 |
| Reasoning off | Intended default | Six mandatory-reasoning configurations use low/minimal; no matched reasoning ablation in this lineup |
| Temperature 0.2 | Sent by candidate runner | Honoured flag differs by endpoint metadata; not independently experimentally measured |
| Human review before full grading | Planned gate 2.C precedes 2.F | Journal explicitly defers the 15-sheet review; review sheet has no completed corrections |
| Gold contributor sensitivity | Planned human-reviewed sensitivity | Human-reviewed analysis unavailable; later 3/3 core-consensus sensitivity is explicitly post-hoc |
| Gold source of truth | Plan mentions production-answer cross-check | Candidate prompts exclude production reference; no completed prod-reference fact-diff artifact is included here. Do not claim that step was completed |
| Headline criterion | Plan mentions ≤12 words | Actual judge rubric tests accuracy/main event/clickbait; it does not enforce the planned 12-word cap |
| Schema success | Plan describes all six fields | Parser only requires headline_en, summary_en and summary_ne, with normalization of other fields |
| Dates/consistency checks | Plan describes conversion/identity checks | Implementation uses deterministic token/surface heuristics. They do not establish semantic date conversion or cross-language factual equivalence |
| Bootstrap | Original scorer uses 1,000 resamples | Exploratory report uses 5,000; focused production also 5,000 with a separate seed; package rank analysis uses 2,000 |
| Calibration gate | Thresholds: ≥0.80 intra-label agreement and ≥0.70 inter-model ranking correlation | Saved calibration meets both; Astra has one missing grade and no second repeat |
| Judge retries | Initial approach preserves errors | Journal adds retries for judge truncation; one Gemini grade still missing. Candidate and judge retry policies must not be conflated |
| Endpoint snapshot | Snapshot at run/resume time | Runner refreshes a file's endpoint snapshot on resume. It is not a full immutable endpoint history for every request |
| Reproducible selection | Deterministic selected-row construction | Selected pool and selection files retained; original full eligibility database population not included |
| Public registration | Files use “pre-registered” internally | No independent public registry or immutable public timestamp supplied |

## Scoring exactly

Let `w(stated)=1`, `w(partial)=0.5`, `w(absent)=w(contradicted)=0`.

- Core recall: mean credit on core facts.
- Supporting recall: mean credit on supporting facts, reported outside the composite.
- Precision proxy: total credited coverage across all fact categories / (that credit + unsupported annotation count).
- Language quality: Nepali and English scores each 1–10.
- Composite: `40*core + 30*precision + 1.5*ne + 1.5*en`.
- Judge aggregation: average available valid judges per model-question.
- Effective model score: average over all 100 questions, using zero for candidate failure.
- Quality-only mean and auxiliary metrics: average over valid generations, with available grades.

The historical scorer computes recall over returned grade entries that match the salience group; duplicate entries can change denominators and extra IDs can affect credited precision. The sanitation sensitivity instead enumerates gold IDs, ignores extras, takes the lowest credit among duplicates and assigns missing IDs zero. It does not rewrite the official score or raw grades.

There is no conventional claim-by-claim precision denominator. Do not label the proxy “93% of statements are true.” The judge's `ne_quality` includes both language form and consistency; do not call it pure grammar. Coverage considers both summaries jointly; do not label it “Nepali factual recall.”

Entity scores use deterministic normalized surfaces/aliases and type matching against gold. Gold may contain entities not chosen for a concise English summary even though the generation prompt requests entities mentioned in that summary. The entity recall deficit therefore includes selection/coverage effects, not only extraction errors.

## New analysis decisions made during publication preparation

All items below are exploratory and were chosen after the original results existed:

1. Independent Python reconstruction and schema/metadata audit.
2. Five weight scenarios, explicitly retaining the original score as primary.
3. Conditional bootstrap rank frequencies, 2,000 draws with seed 20260919.
4. Within-model length/coverage correlations and complete-panel centering on 96 shared-success questions.
5. Cohen's kappa alongside existing exact coverage agreement.
6. Updated selected-corpus counts, publisher inventory and phase cost accounting from retained calls.
7. The four-model criterion analysis from the earlier exploration, 5,000 draws with seed 20260918.

The overall/NE-input pairwise tables use 5,000 draws with seed 20260917. Random seeds specify computation, not generation seeds. The candidate requests do not establish reproducible remote sampling from a fixed seed.

## Interpretation boundaries

Question bootstrap intervals have no adjustment for all pairwise comparisons or overlapping slices. Related saga questions may be correlated. Candidate outputs are sampled only once; judge repeats reassess the same answer. The bootstrap does not separate generation uncertainty from story variation. A confidence interval crossing zero neither proves equality nor validates an equivalence margin. A bootstrap top-frequency is conditional on this benchmark and is not the probability a model is best in general.

Single-judge results include judge/contestant and gold-contributor overlap. High label agreement is reliability evidence, not human validity. Consensus-gold scoring does not establish that consensus facts are correct. The pending review must be completed or prominently disclosed in the published version.

Costs sum values retained on saved calls. Null costs are counted as missing and contribute zero to the reported sum. Replaced retries and earlier experiments are not included. Latency is the stored call duration, not guaranteed full retry or queue latency. Data/cost figures apply to the saved endpoints and run period.

The protocol's descriptions of remote model capabilities are historical experiment records. No current catalogue verification is implied. A model display name, namespace and hosting provider must remain separate fields.

## Prior-publication comparison

The supplied live URL did not return a page through the available web reader. The author-owned local website source was inspected read-only. It declares v0.3, 15 models, 107 stories, 1,310 snippets, 23 publishers and a primary Opus 4.7 comparative judge with three language/coverage axes. The site also contains an older v0.2 blog/PDF. Those legacy version inconsistencies should not be copied into the new page's structured metadata.

No numerical “improvement since v0.3” can be inferred from its absolute scores. Preserve the old publication as an archive and describe methodological changes explicitly.