Can you forecast the Nifty 50?

On 23 September I ran a registered backtest on daily closes of the Nifty 50, 1996 to 2026. A moving average at four window lengths, an autoregression, the historical mean and yesterday’s return each forecast every horizon from 1 to 20 trading days, against a forecast that the return would be zero. The question had a yes or no answer: does anything beat zero?

It did not. I had written down that it would not, in a file I could no longer edit without the report noticing, before a single test origin was scored.

Why a working backtest proves nothing#

Almost everyone who has touched market data has a backtest that worked. That is the default outcome, and it is not a good sign. A backtest is a program that holds the whole series and is asked to pretend it has only seen part of it. Pretending is where it breaks.

The mechanism is always the same. Information from after the forecast origin reaches the model, a fitted parameter, or a decision about which model to report. Nothing crashes. The error just gets smaller, and a smaller error is the one bug nobody goes looking for.

Where does the future get in?#

My design spec lists sixteen routes. Five of them sit on the path from raw data to a scored forecast, and each has a guard with a name.

1. 1Future values reach the model. Only the engine slices data; L1 overwrites everything after t and demands identical forecasts.
2. 2A transform or scale fitted on the whole series. Box-Cox λ, z-scores and the MASE scale are fitted per fold, on the slice.
3. 3A gap filled from the far side. Interpolation stays inside the slice; an origin with a missing last close is skipped.
4. 4Truth read one step off. The target period is computed once, in the engine; L4 checks it on y = t.
5. 5Choices made on the test origins. Window, reference and candidate are picked on dev origins; L5 poisons the test rows.

Future values. The model reads past the origin. The guard is structural: one function, run_backtest, is the only code in the repository that slices data, and every model receives the series up to $t$ and nothing after it. Test L1 checks that claim by overwriting everything after the origin with 1e9, then with NaN, then with a shuffle of the real values, and demanding that every forecast column comes out identical.

Global fits. A Box-Cox $\lambda$, a z-score or the scale in the denominator of MASE, computed on the full series, carries the future’s level and spread into the past. All of them are fitted per fold, on the training slice.

Gaps filled from the far side. Interpolating a missing close near the origin reads the next observed price, which is after $t$. Interpolation is confined to the slice, and an origin whose last value is missing is skipped and recorded as skipped.

Truth one step off. Score the forecast for $t+h$ against the value at $t+h-1$ and a model that copies the latest value looks prescient. The target period is computed once, in the engine, and L4 runs a series where $y_t = t$: the naive forecast at every origin must equal $t$ and the drift model’s errors must be exactly zero.

Choices made on the answer. Picking the moving-average window, the reference baseline or the model to report by looking at test results is the quietest leak of the five. All three were chosen on 20 dev origins that come before the 250 test origins. L5 poisons every numeric column of the test rows with NaN, 1e9, −1e9 and 0 and requires the same three choices.

Catching my own cheat#

A test that has never failed is a hypothesis about a test. So the suite carries two leaky components that exist nowhere else: a model that peeks at the next value through a back door to the full array, and a z-score transform fitted on the whole series. L2 asserts that L1 catches both.

The z-score one taught me something. Naive and drift forecasts are unchanged by an affine transform, so a check built around them would have passed with the leak in place, for the wrong reason. L2 observes it through a model that forecasts zero on the transformed scale, where a global mean cannot hide.

The planted components only prove the tests can see a leak in a test model. On 22 September I planted four leaks in the real engine, one at a time, and counted what failed:

Leak planted in the engine	Leakage tests that failed
Training slice includes $t+1$	10
MASE scale computed on the full series	8
Box-Cox $\lambda$ sees the next value	1
Selection reads every origin role, test included	4

The unmodified engine passed all 17. The Box-Cox leak tripped exactly one of them. One is enough to fail a build. It is also one test away from zero.

A canary that cannot be fed#

Planted cheats test the suite against leaks I thought of. The canary tests it against the ones I did not. It backtests 200 simulated random walks of 150 points each. On a random walk nothing can beat naive, so every “win” at a Holm-adjusted $p < 0.05$ is a false positive.

The share of walks with a win should sit at the 5% false-positive rate. A one-sided binomial test fails the build if it sits significantly above. A cheap model set runs in CI; every model, SARIMA included, is scheduled nightly because it takes about three hours on one core.

Built as specified, the canary failed. 33 of 200 walks were “beaten”, 16.5%, and the cause was not leakage. It was the Diebold-Mariano test with the Harvey-Leybourne-Newbold correction, the test that decides whether one forecast beats another. It estimates a long-run variance from $h-1$ autocovariances, and with 30 test origins at $h = 12$ it had 2.5 effective observations to do that with.

I measured it on the same walks: at $n/h = 2.5$ its one-sided false-positive rate was 9% to 15% against a nominal 5%, and at $n/h = 5$ it was 2.5% to 4.5%.

So the test now declines to run when $n/h$ is under 5, and the report prints “untestable” where a p-value would have been. The canary share fell to 13 of 200, 6.5%, binomial p 0.21.

Writing the rule down first#

Every guard above protects the pipeline from the data. None of them protects it from me. The last route is the researcher’s: change the threshold, drop a horizon, swap the reference after seeing which one flatters the result.

So the decision rule went into a YAML file before the run. This is the part for the Nifty, as committed:

registered: 2026-09-23
alpha: 0.05
primary_window: expanding           # verdicts use it; rolling answers RQ5
series:
  nifty50:
    decision_horizons: [1, 5, 20]   # day, week, month of trading days
    models: [zero_return, mean_return, last_return, sma, ar]
    expectation: >
      Nothing beats the zero-return forecast on the mean (U5). A win over 30% would be a
      leakage audit, not a result.

The run records the file’s sha256 and the commit that last touched it. The report checks both before it will issue a decision. A hash alone would not catch a file that was never committed, and a commit alone would not catch an edit made afterwards.

If either check fails, the report prints a reason in place of the decision: gate.yaml had uncommitted changes at run time (P1): no gate decision, or gate.yaml changed after this run (P1): no gate decision; rerun. A test edits the file after a run and asserts the refusal.

I amended the file once, before any test origin was scored, to name the 1996 start date and the 46 weekend sessions I had dropped from the data. The commit message says so, and the hash the run recorded, 22e458d1d926, is the amended one. Thresholds, horizons, model lists and alpha did not change.

What came back#

The reference chosen on dev origins was a 250-day mean return, which is a forecast of almost exactly zero. Against zero itself it scored a relative MAE of 1.002, 1.004 and 1.007 at 1, 5 and 20 days. The autoregression came closest to beating it: relative MAE 0.991 at $h = 1$, one-sided p 0.083 before correction and 0.250 after Holm. At 5 and 20 days its Holm-adjusted p was 0.539.

The predictable horizon $h^*$, the last horizon before the skill interval first touches zero, came out at 0 of 20. The registered expectation held. A negative result here is the design working: the harness was built to say no when the answer is no, and to say it in a way I could not argue with afterwards.

Two runs of the registered config, one from a clean tree and one with an uncommitted edit to the report code, produced the same 569,176 forecast rows across both series, identical once the run label and the timing column were set aside.

The part that could have embarrassed me was the positive control. A harness that finds nothing looks exactly like a broken harness until you show it finding something that is certainly there.

So the same run scored US electricity and gas output, a monthly series with seasonality you can see from across the room. At $h = 1$ all four seasonal models beat seasonal naive by 16% to 19% in MAE, and ETS and the combination stayed ahead at all 24 horizons. The raw one-sided p-values at $h = 1$ were 0.004 to 0.012.

The rule I had written said Holm over 15 tests. That put the smallest adjusted p at 0.066, above 0.05. The control, whose whole job was to prove the harness could see, came back NO-GO under my own rule.

I did not change the rule. The decision file records it as a power problem, 120 overlapping monthly origins against a 15-way correction, and says what to do next time: register fewer tests or more origins. It also says not to rerun this design hoping for a different p-value.

The one thing that moved#

The mean was flat. The variance was not.

On every test fold I took the autoregression’s one-step residuals from its training slice, a median of 6,956 of them, and ran Engle’s ARCH-LM test. It regresses the squared residuals on their own recent past and asks whether that regression explains anything:

$$e_t^2 = a_0 + a_1 e_{t-1}^2 + a_2 e_{t-2}^2 + a_3 e_{t-3}^2 + a_4 e_{t-4}^2 + u_t, \qquad LM = n R^2 \sim \chi^2_4$$

If yesterday’s size of move says nothing about today’s, $R^2$ stays near zero. It rejected at 5% at 250 of 250 test origins, and at every origin for every other model’s residuals too. The folds share most of their training data, so that is not 250 independent confirmations. It is still a result with no exceptions.

The intervals said the same thing from the other side. The autoregression’s 80% intervals, built from one unconditional variance, covered 94% of next-day returns: too wide for the calm stretches because they were sized for the loud ones.

I could not say where the next close would be. The data were quite sure about how far it was likely to move.