My model failed on the S&P 500

On 26 September I committed two gate files to my forecasting repository at 11:27 UTC. The S&P 500 price history went in at 11:30. Each gate said, in writing, what would count as the S&P confirming a decision I had already made on the Nifty 50, and what I expected to happen.

Neither rule could have been shaped by the data it would judge. The data was not in the repository yet.

Both came back with answers. One said no. The other said yes, and the yes is the less interesting of the two.

What did “replication” mean?#

Same start date, 1 January 1996. Same origin spacing, same origin counts, same levels, same seeds, same coverage bands, same significance level corrected for the number of tests. The primary model in each design carried over by name. Only the series changed: 7,734 S&P trading days from 2 January 1996, at 19.0% annualised volatility.

The tempting move was to re-pick the model on S&P development data. The gate says why not:

# configs/phase5a/gate.yaml, excerpt
chosen_by: >
  Re-selecting on S&P dev origins would answer a different and much
  weaker question, namely whether some model in the family fits this
  market, which is nearly always yes.

I wanted the harder question. Was the bet I had placed on the Nifty a fact about equity index volatility, or a fact about one Indian index?

The daily design failed by 0.0008#

The first design forecasts the single-day return one and five trading days ahead, with 80% and 95% intervals, at 400 test origins five days apart, from September 2018 to August 2026. The primary model is EWMA, a variance estimate that weights recent squared returns more heavily. Fourteen registered checks. Thirteen passed.

The one that did not was the independence test at h = 5 and the 80% level. It asks whether a miss makes the next miss more likely. It returned p = 0.0056 against a corrected threshold of 0.0064.

A margin of 0.0008, and I recorded it as a failure without qualification. Earlier that day I had built a control on synthetic data where I wrote the true variance process myself. Independence caught a model that ignored it in three seeds out of three. The coverage band and Kupiec’s test of the miss rate caught it once each. It is the one check with measured power against this error, so its rejections do not get discounted for being close.

What it saw was specific. The average was nearly perfect and the order was wrong.

h = 5, 80% level	Value
Misses	82 of 400, where 80 were expected
P(miss after a miss)	0.321
P(miss otherwise)	0.176
Longest run	5 consecutive origins, 25 trading days

Not one crisis, either. The misses came in recurring bursts across the whole test window, and no single year held more than 15 of them.

Was it the wrong model?#

That was my first question, and the run could answer it, because all six models in the registered family were scored on the same origins.

Model, h = 5, 80% level	Independence p
Flat variance, Normal tails	0.0001
Flat variance, empirical tails	0.0045
EWMA, the registered pick	0.0056
GJR-GARCH, asymmetric	0.0093
GARCH	0.0300
GARCH, Normal tails	0.115

All six have a lower p-value on this cell than at h = 1. The clustering at a week is something this market does, and the whole family registers it. Three members fail outright. GJR-GARCH clears the line by 0.0029, plain GARCH by a factor of five, and only the Normal-tailed GARCH is far from it.

So was the pick wrong? Partly. GJR-GARCH, which lets a fall move the variance more than a rise, had the best CRPS in the family at both horizons (CRPS scores the whole forecast distribution, lower is better): 0.0049 against EWMA’s 0.0051 at h = 1. It would have passed. On the Nifty the rule had picked EWMA over it on a 0.4% margin, which I had flagged as a risk at the time. The S&P says that margin was noise.

But swapping in the winner afterwards buys a pass by 0.0029, on a check where the family spans 0.0001 to 0.115. What I would defend is narrower than “wrong model”: at a week, on this market, most of the family leaves structure in the order of its misses, and the pick landed on the wrong side of a line its nearest siblings sit close to. The gate said a failure would not be followed by re-picking. It wasn’t.

Two things did replicate. Conditioning on volatility pays on both markets: every conditional model beat the flat interval on CRPS. And the flat interval is badly wrong on both. On the S&P it covered 0.880 at h = 1 against a nominal 0.80. The general finding travelled. The specific claim did not.

The monthly design, and the failure that came back#

The second design forecasts the cumulative return over 1, 5 and 20 trading days, at 200 test origins twenty days apart, with GJR-GARCH as the primary model. On the Nifty it had failed exactly once: the 80% interval at a month covered 0.865 against a band of [0.75, 0.85]. It was too wide. I built a correction for that, registered it, and it passed with four origins to spare.

The S&P replication carries the correction. The uncorrected forecasts sit in the same run, though, so I applied the old rules to them as a diagnostic. They failed exactly one check.

Uncorrected	Nifty 50	S&P 500
Checks passed	20 of 21	20 of 21
The failing check	80% coverage, h = 20	95% coverage, h = 20
Coverage against band	0.865 against [0.75, 0.85]	0.985 against [0.91, 0.98]
Kupiec on the same cell	p = 0.016, passes	p = 0.0080, passes

Same horizon, same direction, too wide. The coverage band carried the decision on both markets while Kupiec, on the same cell, lacked the power to see it. The level differs. Nothing else does.

1. 1The registered coverage band for that level. Inside it is a pass.
2. 2The one failing check on each market: coverage at h = 20 above the band, 0.865 against a ceiling of 0.85 on the Nifty and 0.985 against 0.98 on the S&P.
3. 3Kupiec’s test of the miss rate, on the same cell, passes on both markets against a corrected threshold of 0.0043. It lacks the power to see this.

I had not asked the S&P to show me this. The error the correction exists to fix turned up unprompted, so it was not a Nifty artefact. That is worth more than the pass that followed. A pass can be luck. A recurring shape is much harder to produce by accident.

What the correction does#

It watches how far outside its own interval the truth has been landing lately, and moves the interval by that much.

For each recent origin whose outcome is already known, score the interval that was issued:

$$E = \max(\mathit{lo} - y,\; y - \mathit{hi})$$

Positive when the truth fell outside, by how far. Negative when it fell inside, by the distance to the nearer bound. Take a high quantile of the last 60 scores, call it $Q$, and issue $[\mathit{lo} - Q,\ \mathit{hi} + Q]$ instead. A model that has been too wide collects mostly negative scores, so $Q$ is negative and the interval shrinks.

1. 1The interval the model issued at one past origin, from lo to hi.
2. 2The truth landed inside. The score is minus its distance to the nearer bound.
3. 3The truth landed outside. The score is how far past the bound it went.
4. 4Q is a high quantile of the last 60 scores: the 49th of 60 for the 80% interval, the 58th for the 95%. Positive Q widens both sides, negative Q shrinks them.

Nothing in that arithmetic cares which way the model was wrong. On the Nifty the correction had only ever shrunk intervals, so the S&P gate asked in advance for the other case: if the model is too narrow, the correction should widen it, and a pass there would be stronger evidence than a second shrink.

It did both, in one run. At h = 1 and h = 5, where the model was too narrow, it widened: at h = 5 and 95%, coverage went from 0.935 to 0.965 on intervals 1.24 times as wide. At h = 20 and 95%, where the failure was, it shrank the interval to 0.89 of its width and took 0.985 to 0.960. Twenty-one checks out of twenty-one.

1. 1Mean interval width after the correction, as a multiple of the width before it.
2. 2An overshoot at h = 1: 0.795 became 0.830. The registered band tolerates it; a tighter one would not.
3. 3Too narrow, so it widened: at h = 5 and 95%, 0.935 to 0.965, on intervals 1.24 times as wide.
4. 4Too wide, so it shrank: at h = 20 and 95%, where the uncorrected failure was, 0.985 to 0.960 on intervals 0.89 times as wide.

It is not a precise instrument. At h = 1 and 80% it overshot, and the 95% correction is the third-largest of 60 scores, so those widenings are mostly noise.

The cost came back too#

The correction makes the good model slightly worse and the bad one markedly better. At h = 20, GJR-GARCH’s CRPS went from 0.019439 to 0.019581, and the flat comparator’s from 0.021489 to 0.020588. The ratio that measures what conditioning on volatility buys moved from 0.905 to 0.951. On the Nifty it had moved from 0.911 to 0.972.

A width correction has more to fix in a badly calibrated model, so most of its benefit lands there. The measured edge of conditioning fell by half on the S&P and by about two thirds on the Nifty. With two markets behind it, that is a property of the method rather than an observation from one run.

Where I was wrong#

The gate registered a prediction for this run: it fails, or passes by a margin too thin to mean anything. It gave a reason, and the reason was a claim about the calendar.

# configs/phase5b/gate.yaml, expectation, excerpt
The S&P dev window under this design (1996 onwards, 100 dev origins
at step 20) contains 2000 to 2002, and the test window contains 2008
and 2020. That is the opposite ordering, volatile test after a
volatile dev, so the Nifty mechanism does not transfer

It passed, with seven origins to spare on the check that had carried the Nifty decision. The windows were these:

Warm-up	Development	Test
1998-01 to 2002-10	2002-11 to 2010-09	2010-10 to 2026-08

Development held 2008. Test was the calmer stretch after it, with one spike in 2020. That is the same ordering as the Nifty: a model shaped in a violent period meeting a quieter one, which is exactly the mechanism I had named for the Nifty’s over-coverage. Read correctly, the calendar predicted what happened. The same failure, recurring.

The reasoning was sound. The fact under it went unchecked, and checking it was a two-line query against the origin dates. A registered prediction is only worth something if its facts are right, and this one’s were not.

The defect that closed the study#

The gates read two interval levels, 80% and 95%. The forecast store holds five: 50, 80, 90, 95 and 99. A day after the decision, separate work on threshold probabilities read the whole grid and found intervals that crossed. A 99% band sitting inside the 95% one.

Each level had been corrected from its own scores, with nothing tying the levels together. A wider interval produces more negative scores, so a higher level shrinks further, and often enough it shrank past the level inside it. That was 19.3% of corrected rows on the Nifty and 28.9% on the S&P. No gate saw it because no gate looked there.

1. 1The 95% bounds, carried down through both panels.
2. 2The 99% band after the correction, narrower than the 95%. A higher level collects more negative scores, so it can shrink past the level inside it.
3. 3The repair pushes each outer bound out to meet the one inside it, and no further.
4. 4Inner levels never move. The outer correction comes from the most extreme of the same 60 scores, so it is the less reliable of the two.

The repair pushes each outer bound outward until the grid is ordered, and never moves an inner level. Rerunning both decisions on their stored forecasts moved one published number: 95% coverage at h = 20 on the S&P, from 0.960 to 0.970, inside the same band. No decision changed. The published tables stayed as they were, with a dated note underneath, because editing a recorded decision in place is how a record stops being one.

What this does not establish#

Two markets is two, and these two share their crises. The S&P’s 2008 and 2020 are the Nifty’s 2008 and 2020, so the markets are not independent samples the way two seeds of a simulation are, and the word “replication” promises more than it delivers here.

The simulator is also a choice. A later phase swapped independent resampling of residuals for a block bootstrap, a defensible alternative, and it moved a 20-day expected shortfall by 10% to 15%. Under it, the uncorrected S&P failure still recurred, but on a different cell.

The next test that means anything is a market whose 2008 happened in a different year.