# Prespecified evaluation protocol, version 1.1 For the Gürkaynak–Wolfers auction-only edition `0.3.1-research-review`. Version 1.0 was written on 22 October 2026; version 1.1 corrects its definitions the same day (see Amendments). Both were written before Vultax computed any calibration, sharpness or scoring statistic from this edition. Full-sample descriptive counts had been computed (the 116 / 14 / 23 outcome classification, 32 totals that miss 100, and the data-validity diagnostics quoted below); none is a calibration result. The SHA-256 of this file is recorded in the deposit-candidate manifest so later results can cite the exact text they followed. This is a plan, not a result. Nothing in it says whether the auctions were well calibrated. The original authors report their own tests (FRBSF Working Paper 2005-26, section 5, pp. 23–30). An analysis under this protocol would be an independent check of the auction-only data with its own conventions, not a replication of the paper: the survey and market columns are excluded, and the conventions below differ from the paper's where stated. ## Questions 1. **Digital-call reliability.** For each finite strike, the auction implies a probability that the release exceeds it. How often did releases exceed strikes priced at each level? This parallels the paper's Figure 6 (p. 24). 2. **Realized cumulative positions.** Where did each announcement fall in its auction's distribution, and does the distribution of those positions look uniform by indicator? 3. **Dependence.** Are realized positions serially correlated within an indicator? Out of scope: comparison with survey forecasts (survey data are excluded); risk premia; any claim about present-day markets or trading. ## Notation For one release, ranges *k* = 1…*K* have bounds (*a*k, *b*k) with *a*1 = −∞ and *b*K = +∞ (the source codes these −9999 and 9999). Published masses *m*k are in percent, unchanged. *T* = Σ *m*k is the published total. *C*k = *m*1 + … + *m*k is the raw cumulative mass, with *C*0 = 0. The announced value is *y*. ## Probabilities: raw source versus derived normalized distribution Raw cumulative masses are not a probability distribution. Totals range from 99.4 to 100.3 (91 releases exactly 100, 37 above, 25 below), and one realized interval has a raw upper end above 100. Therefore: - **Source data stay unnormalized** in every file. Nothing is clipped or silently rescaled. - **Every probability test** (uniformity of positions, inverse-normal transforms, reliability of calls) uses an explicitly derived normalized distribution: *q*k = *m*k / *T* and *G*k = *C*k / *T*, so *G*0 = 0 and *G*K = 1. Results state that they use this derivation. - **Raw-mass diagnostics** (published totals, raw realized intervals) are reported separately as properties of the source, never tested as probabilities. - **Sensitivity:** repeat every test excluding the 32 releases whose totals differ from 100 by more than 0.05 points. ## Realized positions: three distinct treatments For *y* strictly inside range *k* (*a*k < *y* < *b*k): - **(I) Assumption-free interval.** The position lies in [*G*k−1, *G*k]. This uses only whole-range masses and needs no assumption about how probability is spread within a range. Primary. - **(II) Linear interpolation in outcome space.** *z* = *G*k−1 + *q*k · (*y* − *a*k) / (*b*k − *a*k), defined for finite ranges only. This is the paper's assumption that the state price is uniform within each strike range (p. 24). It is used as a comparison with the paper's approach, under this edition's normalization and boundary rules. - **(III) Randomized position.** *z* drawn uniformly on the open interval (*G*k−1, *G*k), with 1,000 replications from a fixed, published seed; statistics are summarized by their median and 2.5–97.5% range across replications. This is a randomized PIT for interval-valued positions, a different method from (II). The midpoint of the interval in (I) is **not** the paper's assumption and is not used. Example, December 2003 payrolls (raw percent): *y* = 1 thousand in the 0–25 range, preceding mass 2.00, range mass 1.03. Linear interpolation gives 2.0412; the interval midpoint gives 2.515; the assumption-free interval is 2.00–3.03. **Open-ended ranges.** (I) and (III) need no tail width: an outcome in the lowest range has interval [0, *G*1]. (II) is undefined there without a tail convention. Two stated sensitivity conventions: (S1) the paper's footnote 9 rule, which imputes a tail bound so that the bin midpoint equals the tail mean under a normal shape (p. 6; the paper states it for moments, and its treatment of tail positions is not stated, so this is a reconstruction); (S2) leave the 14 open-tail outcomes out of (II) and report the count. ## Exact boundaries (23 releases) When *y* equals an interior strike *a*k, the identifiable quantity is the **cumulative mass of the preceding ranges**, *C*k−1 (normalized *G*k−1). Reading it as the probability that the release is strictly below *y* would require the contracts' settlement convention for ties, and that no mass sits on the tied value. Neither is documented in the inspected package. - Under (II), the position is *G*k−1 whichever range the tie is assigned to, because the uniform-density assumption puts no mass on a single value. - Under (I) and (III), the side matters: report both, (B-lower) the interval [*G*k−2, *G*k−1] and (B-upper) [*G*k−1, *G*k]. Neither side is primary. - If the original authors document the settlement convention, an amendment makes it primary; earlier results stay published. ## Inverse-normal transforms and endpoints The inverse-normal transform Φ−1(*z*) (the paper's Berkowitz approach, p. 28) is applied only to positions strictly inside (0, 1): interpolated positions (II) of finite-range outcomes, and randomized draws (III) taken on the open interval. It is never applied to the endpoints of (I). Data check: no range has zero mass; the 14 realized intervals that touch 0 or 1 are all open-tail outcomes; no boundary position equals 0 or 1. Any position that equals 0 or 1 under a later convention is reported and excluded from the transform, never clipped. ## Digital-call reliability For a finite strike *s*, the derived normalized probability of the event *Y* > *s* is the sum of *q*j over ranges lying above *s*. The numerical event 1[*y* > *s*] is well defined: at a tie, *y* = *s*, it is 0. What is unverified is whether that numerical event matches the auction's payout at a tie (a call might have paid at *y* ≥ *s*). - **Primary:** exclude the 23 strike–outcome pairs with *y* = *s* and report the count. - **Sensitivity:** include them with indicator 0 (the numerical event), then with indicator 1 (a "pays at or above" convention). Raw masses above every strike stay within 0–100 in this edition, but the normalized values are used for consistency with the position tests. ## Announced values Use the announced values in the edition, the basis of contract payoffs (p. 5). Revised series are not substituted and are not used as a sensitivity check, because they answer a different question. ## Sample, dependence and uncertainty - All 153 releases; no exclusions beyond those stated above. Primary results are **by indicator** (payrolls 33, ISM 30, retail 26, claims 64); pooled results are secondary and shown beside them. Units are never mixed: pooling uses unit-free positions and probabilities only. - Claims are weekly and the indicators share news, so observations are not independent. Confidence intervals: moving-block bootstrap over release dates within each indicator, block length 3 months for monthly indicators and 13 weeks for claims, 10,000 resamples, fixed seed. Pooled intervals resample calendar blocks across indicators jointly. - Serial correlation: first-order autocorrelation of Φ−1 of (II) positions for finite-range outcomes and of (III) draws, by indicator. ## Statistics to report - Reliability table for digital calls: ten equal-count groups by implied probability; observed frequency with bootstrap intervals; by indicator and pooled. - Distributions of realized positions under (I) (lower ends and upper ends), (III) and (II), each labelled with its treatment, by indicator and pooled, with the uniform reference. - Counts excluded or handled by convention: open tails (14), boundaries (23), tie pairs (23), and releases whose totals miss 100 by more than 0.05 (32). - No single headline number, no pooled "accuracy", no ranking of indicators by skill. ## Reporting rules - Publish every result produced under this protocol, whatever its direction. - State where results differ from the paper and why: no survey data, normalization, interval treatment, boundary handling. - Results cannot establish that markets beat economists, that prices equal beliefs, or anything about current markets. - Analysis code must read only `data/auction-only-edition.json`, record this protocol's hash and the edition checksum, and be published with the results. ## Review The protocol can be shared with the original authors for comment before analysis. Comment is not required, implies no endorsement, and any change is logged below. ## Amendments - **1.1, 22 October 2026, before any calibration or scoring statistic was computed.** Following an independent review: the claim that the interval midpoint equals the paper's within-range uniform assumption was wrong and is removed; linear interpolation in outcome space, the assumption-free interval and randomized positions are now defined as distinct treatments. An explicit derived normalization is specified for all probability tests, because raw cumulative masses are not a distribution. Boundary wording now refers to the cumulative mass of preceding ranges, with the settlement convention unknown. The digital-call tie is described as a numerically defined event whose match to the payout convention is unverified. Endpoint handling before inverse-normal transforms is stated. Version 1.0 SHA-256: `a52c1015d13a1fadab5dec0a3c8d28e0d4563065643d588718f1053e50b74bc8`.