A Close Fit Is Not Yet a Forecast

BSTVC & Spatiotemporal Interpretability · Part 12 · Chao Song · 5 November 2026

Suppose a model reproduces a district’s annual outcome closely. Its fitted values follow the historical pattern, and the coefficient maps offer a useful account of how relationships vary across places and years. Can it predict next year?

That requires another evaluation. Historical fit describes agreement with observations used in estimation. A forecast must use the information available at the date it is issued, before the target outcome is observed. For BSTVC and other spatiotemporal models, this difference determines what a performance claim means.

Start with an information cutoff

Hiding one district-year observation while retaining later years tests reconstruction using surrounding information. It does not recreate forecasting from the past.

Bürkner, Gabry and Vehtari’s work on leave-future-out cross-validation explains why ordinary leave-one-out evaluation can be optimistic for this forecasting goal. Future observations may inform the model that predicts an earlier held-out value.

Define the forecast origin first: the date when a prediction would actually be issued. Then specify the outcome, locations and horizon. Predicting next month and predicting the next twelve months are different tasks. If the aim is future prediction across known districts, later outcome observations must be unavailable across those districts, rather than withheld from only one location.

The cutoff concerns availability, not merely the year printed beside a value. An annual statistic released months later cannot be treated as known at the beginning of that year.

Move the forecast origin forward

Rolling-origin evaluation repeats a chronological exercise:

1. Fit using information available at an earlier origin.
2. Predict the intended future horizon.
3. Compare predictions with the outcomes subsequently observed.
4. Move the origin forward and repeat under the same declared procedure.

An expanding training window retains earlier observations. A fixed-length window uses a recent portion of the history. Choose and report a window that matches the intended application, rather than assuming either design is universally preferable.

Evaluate the horizons that matter. Strong one-step performance does not establish equally good performance several years ahead. Errors from overlapping forecast windows can be dependent. Uncertainty summaries must account for the evaluation design rather than assume every error is an independent observation.

Figure 1. Conceptual teaching illustration of chronological evaluation. This simplified timeline assumes no reporting delays: at each forecast origin, training information is already available and evaluation outcomes follow the cutoff. Real data must be split by availability date, including any release delays. This is not a BSTVC experiment, software screenshot or measured performance result.

In this expanding-window example, the three origins use four, five and six training periods. Each is followed by two held-out future periods. Teal cells show available training information; orange hatching marks future outcomes for evaluation; gray cells mark later periods. Earlier evaluation outcomes can enter a later training window only after they become available.

Check what enters the model

Future predictors deserve particular attention. Calendar variables may be known in advance. Future weather, economic conditions or healthcare resources may require separate forecasts, suitable lags or explicit scenarios.

Hyndman and Athanasopoulos distinguish ex-ante forecasts from ex-post predictions that use subsequently observed predictor values. A retrospective evaluation supplied with actual future predictors answers a different question from a forecast issued before those predictors were known. Report that distinction. Intervals conditional on a predictor scenario do not automatically include uncertainty about the scenario itself.

Apply the cutoff throughout the analysis. Learned scaling, imputation, feature selection and other preprocessing belong inside each training split. The official scikit-learn guidance explains how using evaluation data during these steps creates leakage. The principle applies beyond any particular software package.

Keep model selection separate from final assessment

Rolling-origin results can help select model settings. Once they do, they are part of model development.

Reserve a later evaluation period, or use an appropriate nested temporal design, to assess the selected procedure. Cawley and Talbot explain how selection using noisy evaluation results can bias reported performance.

A predeclared sequential procedure may update the model when earlier outcomes become available. That is legitimate if it matches deployment. Changing settings after inspecting final-test performance is a different operation and must not be presented as untouched evaluation.

Report evidence at the right scale

Compare candidate models with suitable simple benchmarks on identical origins, locations and horizons. Depending on the outcome, a last-observation or seasonal-naïve forecast may provide a useful reference.

Report point errors by horizon and state how locations are weighted. For probabilistic forecasts, assess predictive interval coverage together with width and an appropriate proper scoring rule. Narrow intervals alone do not establish reliable uncertainty.

Keep posterior uncertainty about coefficients or latent means distinct from uncertainty about future observations. A coefficient credible interval is not a forecast interval.

For spatiotemporal interpretability, useful forecasting evidence connects a clear question to an honest information boundary. Historical explanations and future predictions can complement each other, provided their evidence is reported separately.

中文速读

历史拟合良好，只说明模型与参与估计的数据较吻合，不能直接证明预测未来的能力。时间验证应先明确预测发出日期、预测多少期、哪些地区，以及当时真正可获得的信息。滚动预测起点让每次训练只使用过去的数据；未来协变量、数据预处理和模型选择也必须遵守这一边界。最终测试应与调参分开，比较相同预测期下的误差与不确定性。本文介绍验证设计，不报告已执行的 BSTVC 预测实验，也不声称 Desktop 已自动实现这些步骤。

Read, verify and use

Continue with Part 11: What Does Leaving One Observation Out Actually Test? and Part 10: STVPI, Bayesian R² and Predictive Accuracy.

For implementation, consult the public BSTVC-R documentation, verify the actual workflow and check its current licence. This article credits the BSTVC research team and the original methodological authors; it does not establish a software feature or a measured forecasting result.

Further reading: author-maintained leave-future-out explanation; Gneiting, Balabdaoui and Raftery on calibration and sharpness.