Skip to content
forecastingstatisticstesting

Are your calibration tests blind?

I wrote a world where I knew the right answer, fed it a model with no conditional variance at all, and counted how often each standard interval check noticed. Two of the three almost never did.

Your test says the model is calibrated. How do you know the test can tell? I ran three calibration checks through three phases of a forecasting study before I asked that, and never once measured whether any of them could fail. When I did, two of the three turned out to be blind to the error I cared about most.

Write the world yourself

On real market data nobody knows the true volatility, so a check that passes there proves only that nothing is obviously broken. On 26 September I stopped asking real data this question and wrote the answer down first.

The series was a GARCH(1,1) process with parameters I picked. Each day’s variance is a small constant, plus a slice of yesterday’s squared return, plus most of yesterday’s variance:

σt2=ω+α rt−12+β σt−12,ω=1.6×10−6, α=0.09, β=0.90\sigma_t^2 = \omega + \alpha\, r_{t-1}^2 + \beta\, \sigma_{t-1}^2, \qquad \omega = 1.6\times 10^{-6},\ \alpha = 0.09,\ \beta = 0.90

α+β=0.99\alpha + \beta = 0.99, so a bad week echoes for months. The long-run daily standard deviation is 1.26%, about 20% a year. I generated 8,000 days after discarding a 2,000-day burn-in, at three seeds. A separate test holds the generator to its own parameters: sample variance near ω/(1−α−β)\omega / (1 - \alpha - \beta), squared returns autocorrelated, plain returns not. Without it the world could drift and leave the control passing while it tested nothing.

Writing the world hands you two models. garch_normal generated the data, so it is correct by construction. zero_return_fhs is wrong in exactly one way: one variance for every day, with tails taken from the historical distribution. It is what you would build if you believed volatility does not cluster.

Why a control needs two arms

A check that fails the correct model raises false alarms. A check that passes the wrong one is not measuring anything. The first arm is the one people build, because false alarms are loud.

It was also the arm I had been silently relying on. Every NO-GO the study had recorded, Phase 3’s included, assumed the gate was not simply too strict. It held. The correct model, run on the first seed, cleared every registered check in both shapes of backtest, at every horizon. The second arm is where the surprise was.

Three checks, in plain words

Each forecast is an interval: I am 80% sure the five-day return lands between here and here. When the actual lands outside, that is a miss, and a backtest becomes a row of covered and missed checks in time order.

  • Coverage band. Did the truth land inside about as often as promised? The study registered [0.75, 0.85] for the 80% interval and [0.91, 0.98] for the 95%.
  • Kupiec. Is the miss rate significantly off the promised one? A binomial likelihood ratio on the count.
  • Independence. Did the misses arrive scattered or in runs? It counts consecutive pairs and asks whether a miss is likelier straight after a miss. Its name is Christoffersen’s test.

Each ran at a Šidák-corrected level: 0.0064 when a phase ran eight tests, 0.0043 when it ran twelve.

One sequence of 20 checks read three ways: the coverage band reads the share covered, Kupiec reads the miss rate, and the independence test reads how often a miss follows a miss. one sequence, 20 checks of an 80% interval1coverage band reads the total0.700.750.800.850.9016 of 20 = 0.802Kupiec reads the rate0.20 missed, 0.20 promised, in any order3independence reads the orderthen coveredthen missmiss rateafter coveredafter a miss132220.130.50
  1. 1The band reads one number, the share covered: 16 of 20 is 0.80, inside [0.75, 0.85].
  2. 2Kupiec reads the same total as a miss rate and asks whether its distance from the promised 0.20 is more than chance. Sort the sequence, as here, and neither 1 nor 2 moves.
  3. 3Independence counts every consecutive pair into four boxes. A miss followed a covered check 2 times in 15 and followed a miss 2 times in 4, the arcs above. If misses arrived independently, both rows would sit near 0.20. The test is a likelihood ratio between two rates and one.
The same twenty checks, read three ways. The first two checks cannot tell this sequence from its sorted copy.

The first two read the same number. Shuffle the sequence and neither moves.

Two shapes

I ran the control in both shapes the real phases had been registered in, because I suspected the answer might depend on the shape. Shape A was daily returns, forecast origins 5 days apart, 400 test origins, horizons of 1 and 5 days. Shape B was cumulative returns, origins 20 days apart, 300 test origins, horizons of 1, 5 and 20.

What caught it

In shape A the flat model was caught at all three seeds, and the independence test caught it every time. A coverage band caught it once. Kupiec caught it once, on the same seed.

In shape B it was caught at two seeds of three, both times by independence. The bands and Kupiec caught it zero times, on every seed, at every horizon.

Detection of the flat model by each check over three seeds. Daily shape: coverage band 1 of 3, Kupiec 1 of 3, independence 3 of 3. Twenty-day shape: coverage band 0 of 3, Kupiec 0 of 3, independence 2 of 3. A: dailystep 5, 400 origins123B: 20-daystep 20, 300 origins123seedcoverage band1 of 30 of 3Kupiec1 of 30 of 3independence3 of 32 of 312
  1. 1Coverage bands and Kupiec in the 20-day shape: six chances, no detections, at any horizon.
  2. 2Seed 1 escaped everything in the 20-day shape. Its independence p-values stayed above the corrected threshold of 0.0043.
One square per seed, filled when that check caught the model with no conditional variance. Caught by anything: 3 of 3 daily, 2 of 3 at twenty days. The one daily seed where the band and Kupiec fired was a seed where the flat model was wrong on average as well as in sequence.

One trap nearly made this look better than it was. The study also scores CRPS as a ratio against the flat model, so for the flat model itself that ratio is exactly 1.0 and always fails. Counted as a detection, it would have made the negative arm pass for free. I excluded it and wrote the reason into the test.

Right on average, wrong in sequence

Why would a model with no volatility dynamics pass a coverage check? Because its errors cancel. Its variance is the long-run average. In a calm stretch the interval is too wide and it barely misses. In a storm it is too narrow and misses origin after origin. Add those up over 300 origins and the count lands near the promise. In shape B the flat model’s 80% coverage sat inside [0.75, 0.85] at every seed.

A flat variance model with empirical tails is right on average and wrong in sequence. Only one of the three checks reads the sequence, so only one can see the error.

Two rows of 200 interval checks with 40 misses each. In the first the misses are scattered; in the second they arrive in runs. Coverage and Kupiec are identical for both. 1scattered40 misses in 2002in runs40 misses in 2003
  1. 1Coverage 0.80, Kupiec p = 1.00, independence p = 0.075. A miss followed a miss 12 times in 40. Random misses clump a little too, and the test knows how much to expect.
  2. 2Coverage 0.80, Kupiec p = 1.00, independence p = 1.2 × 10-9. A miss followed a miss 23 times in 40, and followed a covered check 17 times in 159.
  3. 36 misses in a row. The first row's longest run is 4.
Two hundred checks of an 80% interval, drawn twice from a seeded simulation and wrapped at 100. Same count, same coverage, same Kupiec statistic. Only the first row is what a calibrated model produces.

The same day, this changed how I read a real result. Phase 5a, a replication on the S&P 500, failed one check of fourteen: independence at five days and 80%, p = 0.0056 against 0.0064. Coverage was fine, 82 misses where 80 were expected, and Kupiec returned 0.803 on the same cell. But a miss followed a miss 0.321 of the time, against 0.176 after a covered origin.

A margin of 0.0008 is the kind of thing I would once have called a near miss. By then it was the one check with measured power against exactly this kind of error, and I recorded a failure.

Validity and power pulled apart

Shape B exists because of a decision in Phase 3. At a 20-day horizon, forecasts from origins 5 days apart share 15 days of their windows, so their misses are dependent by construction and the independence test would flag clustering the model never caused. Spacing origins 20 apart made the test valid at 20 days.

It also meant that at 1 and 5 days the sequence was now read once every 20 days instead of every 5. A run of bad days shorter than the spacing puts one origin in the storm, maybe two. There were also fewer origins, 300 against 400.

I had reasoned about that trade when I made it. The control turned it into a number: one escaping seed in three. Seed 1 in shape B was caught by nothing, and its independence p-values never cleared 0.0043. It also told me Phase 2’s GO had been decided in a shape where the harness had more power than the one Phases 3 and 4 were decided in.

The negative arm asserts less than I measured: at least two seeds of three caught in shape A and at least one in shape B, against three and two observed. A test pinned to exactly the count you saw fails on the next library upgrade and gets deleted. The measured counts live in the docstring. A separate test pins the blindness itself, requiring the bands and Kupiec to stay silent on the flat model in shape B. If it ever fails, the harness has gained power, and the control is what gets updated.

How many paths is enough?

The same move answers a different question. The original plan proposed 1,000 Monte Carlo paths per forecast and asked whether that was enough. The spec answered by choosing 10,000, which is a choice and not an answer.

So I held the answer fixed and varied only the thing I wanted to measure. I fitted one GJR-GARCH model to the last 3,000 daily returns of the Nifty 50 and simulated 20-day cumulative returns from it 30 times at each path count, changing only the seed. Nothing about the model moved between reruns, so the spread is pure simulation noise.

PathsNoise on a 95% band edgeShare of band widthNoise on 5% ESShare of ES
1,000±0.01057.60%±0.012414.86%
10,000±0.00332.36%±0.00394.72%
100,000±0.00090.67%±0.00131.57%

Noise here is 1.96 standard deviations across the 30 reruns, in units of 20-day log return. Each tenfold increase in paths cuts it by roughly 10\sqrt{10}, which is what it should do.

Simulation noise on a 95% band edge falls from 7.60% of the band width at 1,000 paths to 2.36% at 10,000 and 0.67% at 100,000. A 1.8% effect sits below the noise at 10,000 paths. noise on a 95% band edge, as a share of band width0.5%1%2%5%10%1,00010,000100,000simulated paths, log scalePhase 4's width correction, 1.8%7.60%2.36%0.67%123
  1. 11,000 paths, the count the original plan proposed. Noise of ±7.60% of the band's width on a band edge, and ±14.86% of its own value on a 5% expected shortfall.
  2. 210,000, the count the study registered and ran. ±2.36%, still above the line.
  3. 3The noise falls below the 1.8% correction only somewhere between 10,000 and 100,000 paths.
Thirty reruns of one fitted model at each count, changing only the seed. Noise is the 95% range across reruns, as a half-width. At a single forecast origin, an effect under the curve is inside the simulation noise.

At 1,000 paths, every 5% expected shortfall you print carries about 15% of its own value in noise. And the Phase 4 decision turned on a width correction of 1.8% at 20 days, smaller than the 2.36% per-origin noise on a band edge at the 10,000 paths I actually ran. It survived only because the decision averaged over 200 origins, each with its own random stream, which shrinks that noise by roughly the square root of 200.

Every check has a power, and until I had counted it, a pass told me less than I thought.