# Prespecified evaluation protocol, version 1.2

For the ENSOFORECAST core files, review edition `0.3.1-research-review`. Versions 1.0 and 1.1 were written on 22 October 2026; version 1.2 corrects the timing classification the same day (see Amendments). All three were written before Vultax computed any skill statistic from these files. Coverage, masks and target-month counts had been computed; they are not skill results. The SHA-256 of this file is recorded in the deposit-candidate manifest. Changes made after results are seen are logged below as dated amendments with reasons.

This is a plan. Nothing in it says how skilful LDEO1 or LDEO2 were.

## Questions

1. How closely did each product's fields match the archive's observations, by nominal lead and target season, on matched months?
2. Did either product beat simple baselines that use only information available by the nominal start?
3. How do conclusions change between periods with different evidence about when fields were made?

Out of scope: event probabilities, comparison with other centres' forecasts, and any claim of real-time skill without issue receipts.

## Data, masks and populations

- Only the nine documented core files, read from the verified transfer.
- **Mask.** One valid-cell mask, the same 1,077 cells, is shared by every month of all nine core files. This was verified from the source and is recorded in `data/coverage.json`; `validate.py --source-tgz` re-checks it. All metrics use this mask. If a future file breaks the invariant, the mask becomes the per-target intersection of valid cells across every product compared, and is reported per target.
- **Field-agreement population: 316 months.** The target months common to all nine files (1972-01 to 1998-04). Used for model-versus-observation agreement without baselines, so products and leads are compared on identical months.
- **Baseline-comparison populations.** Baselines cannot be computed for every early target (see Warm-up). For each lead, the population is the common months on which every baseline below is defined:

  | Nominal lead | First target | Months (to 1998-04) |
  |---|---|---|
  | 3 | 1977-06 | 251 |
  | 6 | 1977-12 | 245 |
  | 9 | 1978-06 | 239 |
  | 12 | 1978-12 | 233 |

- **Fully matched population: 233 months** (1978-12 to 1998-04), on which every product, lead and baseline is defined. Used for any comparison across leads or between products against baselines.
- **Secondary:** each product and lead on its full overlap with observations (316–338 months), reported separately and never ranked against the matched results.

## Regions and weights

- Full domain, area-weighted by cos(latitude), on the common mask.
- Equatorial strip: the 1°S and 1°N rows.
- A Niño-3.4-like box: cell centres 5°S–5°N, 172°W–120°W. Always labelled "Niño-3.4-like (archive grid, undocumented baseline)", never as the official index.

## Metrics

Continuous only: anomaly correlation (centred), root-mean-square error and mean error, by lead (3, 6, 9, 12) and target season (DJF, MAM, JJA, SON); box-mean time-series correlation and RMSE. No categorical or probabilistic scores.

## Information cutoff for baselines

For target month *t* and nominal lead *L*, the **baseline cutoff** is *c* = *t* − *L*. Every baseline uses only observed anomalies for months ≤ *c*, and every fitted coefficient uses only pairs whose months are both ≤ *c*.

- Under the convention in LDEO's 1996–1997 bulletins (ledger E3), the six starts are *t* − *L* − 5 … *t* − *L*, so *c* is the latest start month.
- Under the alternative hypothetical convention, in which the nominal lead counts from the earliest start, the starts would be *t* − *L* … *t* − *L* + 5, ending five months **later**. The cutoff *c* = *t* − *L* is then earlier than the model's latest start.
- So *c* never gives a baseline more information than the forecast had under either convention.
- The archive does not document its convention, and the observation file's vintage is unknown. The cutoff is a restriction on the archive's own monthly index, not evidence of what was historically available on any date.

## Baselines

All baselines use the archive's observation file and the cutoff above.

- **Zero anomaly**: climatology as defined by the archive's own undocumented baseline, which may overlap the evaluation years. Reported, but not used as the sole reference.
- **Persistence**: the observed anomaly at month *c*.
- **Lag-averaged persistence (bulletin convention)**: the mean observed anomaly over months *c* − 5 … *c*, mirroring the six-start composite.
- **Lag-averaged persistence (hypothetical earliest-start convention)**: the mean over *c* … *c* + 5. This uses months after the cutoff, so it is a labelled sensitivity analysis of the convention question, not a fair baseline. It is never chosen because it scores better or worse.
- **Damped persistence**: the anomaly at *c* times a lag-*L* correlation estimated, cell by cell and separately by calendar month of *c* if at least 20 pairs exist (otherwise pooled across calendar months), from pairs (*s*, *s* + *L*) with 1972-01 ≤ *s* and *s* + *L* ≤ *c*.

## Warm-up and unavailable baselines

Observations start in 1972-01, and the archive holds no earlier months.

- Persistence needs *c* ≥ 1972-01.
- Bulletin-convention lag-averaged persistence needs *c* − 5 ≥ 1972-01.
- Damped persistence needs at least 60 fitting pairs, which first holds at *t* = 1972-01 + 2*L* + 59 months (the first targets in the table above).

Earlier targets have no defined baseline. They are reported as unavailable, never filled from other data, and excluded from baseline comparisons, which use the populations above.

## Timing classes and context phases

Classify each target by its **target month** *t*, not by its nominal start. The cutoff *c* = *t* − *L* restricts what a baseline may use; it says nothing about when a model field was produced or issued.

- **Retrospective: *t* ≤ 1985-12, both products, every lead.** Lamont's first predictions of the future were made in early 1986 (ledger E5), so a field for a target month up to December 1985 was produced after its target. The exact production dates are unknown.
- **Issue time unverified: *t* ≥ 1986-01, both products, every lead.** The files carry no issue or production timestamps. A field for these targets may have been produced in real time or later. No target from 1986 onward is called retrospective or real-time without field-specific evidence (E7).

Within the issue-time-unverified targets, results may also be shown by **documentary context phase**, by target month. The phases follow the bulletins' account of when LDEO publicly issued its improved-initialization system, called LDEO2 there (E4). They are context, not issuance or production classes:

- the archive's LDEO2 files are not established to be that system;
- a first documented public issue date does not show when any experimental field was first generated;
- the phases apply to both products alike.

| Class or context phase (target month) | Of the 316 common targets, every lead | Of the 233 fully matched targets |
|---|---|---|
| Retrospective: 1972-01 to 1985-12 | 168 | 85 |
| Issue time unverified, context phase "before documented LDEO2 issuance": 1986-01 to 1995-06 | 114 | 114 |
| Issue time unverified, context phase "documented start of LDEO2 issuance, exact month unknown": 1995-07 to 1995-12 | 6 | 6 |
| Issue time unverified, context phase "after documented LDEO2 issuance began": 1996-01 to 1998-04 | 28 | 28 |
| Total | 316 | 233 |

In the per-lead baseline populations (251/245/239/233 for leads 3/6/9/12), the retrospective targets number 103/97/91/85 and the issue-time-unverified targets 148 at every lead. No result may be called real-time skill unless issue receipts are established for the fields used (E7).

## Uncertainty

Month-to-month anomalies are strongly autocorrelated and leads overlap. Use a moving-block bootstrap over target months with block lengths of 12 and 24 months (report both), 10,000 resamples, fixed published seed. For product differences, resample paired months jointly. Report effective sample sizes.

## Known unknowns that limit interpretation

The anomaly reference period (E8); the observation product and its vintage (E9); the SVD training period, which may overlap the evaluation years; the lead convention (E3); when fields for targets from 1986 onward were produced, and whether they were written as issued (E7); and the archive meaning of LDEO1 and LDEO2 (E4). Each is listed beside the results it affects.

## Reporting rules

- Publish every result produced under this protocol, in every period and lead.
- No single headline number and no "best model" statement. Differences between LDEO1 and LDEO2 are reported with paired intervals and period labels.
- Analysis code reads only the verified core files, records this protocol's hash and the transfer hash, and is published with the results.

## Review

The protocol may be shared with people who know the archive. Comments imply no endorsement; any change is logged below.

## Amendments

- **1.1, 22 October 2026, before any skill statistic was computed.** Following an independent review:
  - Damped persistence previously fitted on months "before each target", which could use months after the nominal start. All baselines now obey the cutoff *c* = *t* − *L*.
  - Warm-up rules and matched populations were added (233 months fully matched; 316 for field agreement only).
  - The five-month convention sensitivity was reversed: an earliest-start convention shifts the window five months later, not earlier. It is now a labelled hypothetical, not a baseline.
  - The claim that all LDEO2 targets before 1996 are retrospective was replaced by conditional and unresolved periods (itself corrected in 1.2).
  - The mask is now the verified common 1,077-cell mask rather than an inference from equal counts.
  - Version 1.0 SHA-256: `896383388a4b8cbcef328842a295e560505a5d48597f95efbd64c84843c807f5`.
- **1.2, 22 October 2026, before any skill statistic was computed.** A logic correction following a further review. The baseline cutoff *c* = *t* − *L*, the warm-up rules and the 233-month matched population are unchanged.
  - Version 1.1 assigned the retrospective period by nominal start (*c* ≤ 1985-12). The model chronology (E5) dates the first predictions of the future, so it supports the inference only for **target** months up to 1985-12. A field initialized in December 1985 for a December 1986 target could have been a real forecast issued in early 1986. Version 1.1 therefore over-classified 3/6/9/12 targets (from 1986) as retrospective at leads 3/6/9/12.
  - Version 1.1 labelled LDEO2 targets with starts from 1986-01 to 1995-06 "conditionally retrospective" because they preceded the bulletins' documented public issuance. A first public issue date does not establish when experimental fields were first generated, so those targets are issue time unverified. The bulletin dates now define labelled context phases only, by target month, for both products.
  - Classification is by target month: 168 retrospective targets of the 316 at every lead; 148 issue time unverified (context phases 114 / 6 / 28).
  - The displayed 1982 and 1997 maps keep their labels (retrospective; issue time unverified), and no source value changed.
  - Version 1.1 SHA-256: `8230004d840e618aa04e874b6ebc7777c1853b3c9c6c4bc4bc18263504d68a10`.