Model outputs and validation
A trained model returns a stack of views, and every one of them exists to answer a single question: does this model predict outcomes on data it never saw during training. Read them together and they triangulate that answer. Which views appear depends on the family you picked on the training page: a regression reports one set of things, a classifier another. In-sample fit is not among them, on purpose. A model can fit the history it was trained on with no error and still fall apart the moment a new session opens. Out-of-sample behaviour is the evidence. In-sample fit is not.
What "validated" means here
Validated here means one thing. The model held up on data it was not trained on, and it held up more than once.
That is a narrower claim than "the model works," and the gap between the two is where money is lost. A model can pick up a relationship that is real and still not be tradable, because the relationship is buried under enough noise that a live position cannot lean on it. The whole point of these views is to separate those two cases: a genuine, repeatable edge from a number that looked good on one slice of history and nowhere else. The validation layer frames the entire model stage around that distinction, and the outputs below are how you apply it to one specific model.
The fold table
The fold table is the first thing to read and the hardest to fake. Walk-forward validation trains the model on one window of history and scores it on the next window it has never seen, then slides forward and repeats. Each pass is a fold, and each fold is one row.
The columns are the same for every model type:
- The training window: the date range the model learned from in that fold.
- The test window: the out-of-sample date range it was then scored on.
- The out-of-sample score: how well it predicted outcomes in that test window. For a regression the score is a fit metric; for a classifier it is a discrimination metric. The family you chose decides which.
Read down the score column, not across any single row. Consistency is the signal. If the model scores steadily across every fold, it is learning something that persists as the market moves through different periods. If one fold carries the whole result and the rest are flat or negative, the model found a pocket of history it happened to fit and nothing more. A strong average built on one dominant fold is a warning, not a result. This column is what confirms the model is generalizing rather than memorizing the noise in any one stretch of history.
Coefficient stability (linear and logistic) or feature importance (forests)
The next view tells you whether each input signal actually carries weight, and it takes one of two forms depending on the model family.
For a linear or logistic model, you get a coefficient-stability view. For each input signal it plots the weight the model assigned to that signal in every fold. Stable weights, close in size and holding the same sign fold to fold, mean the model found a consistent relationship it can rely on. Weights that swing wildly in magnitude, or flip sign between folds, mean the opposite: the model is not learning a durable relationship for that feature, it is chasing whatever fit the last window. A coefficient that is large in one fold and reversed in the next is telling you that feature is noise dressed as signal.
For a random forest (regression or classifier), coefficients do not exist, so the same slot shows feature importance instead: how much each input signal contributed to the ensemble's predictions. It answers the same question the coefficient view answers for linear models, "does this feature actually matter," through a different mechanism. A feature that ranks high is one the forest leaned on to make its splits; a feature near zero is one the model could drop without noticing.
Both views serve the same purpose. Before you trust any headline score, check that the features driving it are the ones you expected, and that their influence is steady rather than an artifact of a single window.
Regression outputs: the out-of-sample scatter and fit metrics
A regression model, linear or random forest, predicts a value, so its verdict is a predicted-versus-actual scatter drawn from every out-of-sample test window pooled together. Each dot is one out-of-sample prediction: the value the model forecast on the horizontal axis, the value that actually occurred on the vertical. A y=x reference line marks perfect prediction.
Alongside the scatter sit two fit metrics: the share of variance the model explains, and the size of its prediction error. A positive trend in the cloud, sloping up toward the reference line, confirms the forecasts are directional in a way that survived out of sample.
Read the cloud, not the line. A thin diagonal draws the eye and invites over-reading. What matters is how tightly the dots hug the trend. A visible slope with wide scatter around it is a weak relationship, whatever the headline number says, because any single prediction can land far from the actual. And watch the extremes: a handful of influential points at the edges of the range can tilt the whole fit and manufacture a trend the bulk of the data does not support. A model that predicts a return dataset well over the middle of its range, and only breaks at the tails, is more tradable than one whose entire signal lives in three outliers.
Classification outputs: probabilities, calibration, and discrimination metrics
A classifier, logistic or random forest, predicts a class probability, so it reports a different set of views. The raw output is the predicted probability for each point in time, for example the model's estimated chance that a given trade wins. Two things tell you whether those probabilities are worth acting on.
The first is the calibration curve. It bins the predictions by their stated probability and, for each bin, plots the predicted probability against the frequency the event actually occurred. A well-calibrated model sits on the 45-degree line: when it says 70%, the outcome happens about 70% of the time. A curve that bows above or below the line means the probabilities are systematically too low or too high, so a 70% call cannot be sized as a 70% call.
The second is a set of discrimination metrics: AUC, log-loss, accuracy, and a confusion matrix of predicted versus actual classes. These measure whether the model separates winners from losers at all, independent of whether its stated probabilities are honest.
Read calibration and discrimination together, because they fail independently. A model can be perfectly calibrated and still useless: if it says 55% on every trade and 55% of every kind of trade wins, its probabilities are honest and its AUC is near a coin flip, so it separates nothing. Calibration without discrimination is an honest model that cannot tell one outcome from another. You need both, the probabilities to be truthful and the model to actually pull the classes apart.
The final model summary
The last view is the fitted model over all available history, not a fold. This is the model that would actually run: the one a backtest or an automation uses to generate a live prediction. For a linear or logistic model it reports the per-signal weights and their confidence statistics; for a forest it reports the fitted ensemble and its feature-importance ranking.
Read it last, and only after the earlier views have earned it. The final summary is trained on everything, which makes its numbers look the most convincing and carry the least evidence: there is no held-out window here to check it against. Strong final weights sitting on top of an unstable fold table are a mirage. The fold table and the stability view are what license you to believe the summary. Trust the summary because the folds held, never the other way around.
Telling real from overfit
Put the views together and the two cases look nothing alike.
A real edge reads as: stable coefficients or importances across folds, a fold table that scores consistently out of sample rather than on one dominant window, and a believable regression scatter or a calibration curve that tracks the diagonal. The signal shows up in independent test windows and the features driving it hold their weight.
An overfit model reads as: a great in-sample number that collapses out of sample, a fold table where one period carries the whole result, coefficients that flip sign fold to fold, and a scatter whose trend lives in a few influential points. The headline looks strong and every out-of-sample check undercuts it.
This is the "real but noisy versus actually tradable" split applied to a model. A relationship can clear the first bar, genuinely present in the data, and still fail the second, too weak or too unstable to trade on. The views are built to show you which side of that line a model is on before you wire it into a position.
Out-of-sample is only honest if the split was look-ahead-safe
Every claim on this page rests on one condition: the out-of-sample windows have to be genuinely out of sample. If a feature or the target could read a value that was not visible yet at that timestamp, the split leaks, and the fold table reports a fit the model would never reproduce live.
Out-of-sample evidence is only honest if the training split was look-ahead-safe. Every feature and the target are aligned on the union spine with backward as-of fill, and every rolling window is observation-count, not calendar. A daily or bucketed input is visible only at its session-close label, so it cannot feed a prediction earlier in the same session. When any of that breaks, a feature quietly reads the future, in-sample and out-of-sample fit both inflate, and the fold table stops meaning anything. Confirm the split held before you trust a single score above.