Bruno Hebling Vieira·
Weblog

ML Model Performance Visualizer

Performance metrics are usually met as formulas, and a formula is a poor place to build intuition. Every metric below is instead drawn as a shape on a scatter plot: a length, an area, a set of connecting segments. Once you can see the shape, you can see what makes it grow — and, more usefully, what leaves it completely unmoved.

The data are synthetic and every panel on this page reads from the same two samples, so a point you drag in one figure moves in all the others. That is the whole point: one nudge, several verdicts.

Regression metrics

The regression sample has 24 observations. The true value sits on the horizontal axis and the model's prediction on the vertical one; both are standardised, so the units are standard deviations. The dashed diagonal is the line of perfect prediction, where every point would lie if the model were exactly right. Drag any point up or down to change what the model predicted for that observation.

Randomise everything throws the sample away and draws a fresh one: the true values move as well as the predictions. Dragging is the gentler tool — points slide vertically, so the truth stays put and only the prediction changes.

Mean absolute error

The residual of an observation is the vertical gap between its point and the diagonal. Take the length of each gap, ignore its sign, and average:

\[ \mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n} \lvert \hat{y}_i - y_i \rvert \]

MAE is measured in the units of the outcome, which makes it the easiest metric here to state in a sentence: on average the model is off by this much. Because each error contributes its own length and nothing more, one badly predicted observation moves MAE by exactly its share, 1/n of the damage.

Figure 1. Each segment is one residual. MAE is the average segment length; drag a point away from the diagonal and the metric grows in proportion to the distance you travel.

Mean squared error

Square the residual instead of taking its absolute value and you get an area rather than a length:

\[ \mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n} (\hat{y}_i - y_i)^2 \]

Each shaded square has the residual as its side, so doubling an error quadruples its contribution. This is the practical difference between the two: MAE primarily reflects the typical magnitude of prediction errors because all residuals contribute linearly. MSE increasingly emphasizes the largest prediction errors because their contribution grows quadratically, making the metric much more sensitive to the tails of the error distribution. Drag one point far from the diagonal and watch its square swell past everything else on the plot — that single observation now dominates the score.

Figure 2. The same residuals as Figure 1, squared. MSE is the average area. A point twice as far from the diagonal contributes four times as much.

Concordance index

Both metrics above ask how close the predictions are. The concordance index asks something weaker and often more relevant: does the model put the observations in the right order? Take every pair of observations with different true values and check whether the predictions rank them the same way.

\[ \mathrm{C} = \frac{\#\,\text{concordant} + \tfrac{1}{2}\,\#\,\text{tied}}{\#\,\text{comparable pairs}} \]

A model that ranks perfectly scores 1, coin-flip ordering scores 0.5, and perfectly reversed ordering scores 0. Note what the C-index ignores: add a constant to every prediction, or multiply them all by two, and the ranking is untouched while MAE and MSE change substantially. It is a rank statistic, blind to calibration and to scale.

Figure 3. One segment per comparable pair, coloured by whether the predictions agree with the true ordering. The C-index is the fraction of segments that are concordant.
  • Concordant pair
  • Discordant pair
  • Observation

One sample, three verdicts

Here are all three on the same sample, still linked — drag a point in any panel and the other two answer immediately.

Perfect drops every prediction onto the diagonal. Move together is a toggle: while it is on, dragging any point slides the whole set of predictions by the same amount, which shifts them without ever reordering them.

Figure 4. The same 24 observations under three metrics. Dragging a point in any panel updates the other two.

Things to try

  1. Drag a single point far from the diagonal without crossing its neighbours. MSE jumps, MAE rises modestly, the C-index does not move at all.
  2. Now take a point that sits close to the diagonal and drag it just past a neighbour. MAE and MSE barely register it; the C-index drops.
  3. How large can you make the mean squared error while the C-index stays at 1? Start from Make predictions perfect, switch on Move predictions together, and push the whole set away from the diagonal.
  4. And the other way round: how far can you drive the C-index down while MAE stays under, say, 0.3? Small swaps only — you are allowed to ruin the ordering, but not to move anything very far.
What you should find

A group shift is rigid: it cannot change which prediction is larger, so the C-index sits at 1 no matter how far you drag, while MAE grows with the shift and MSE with its square. Ranking metrics are blind to any offset you care to add. The reverse experiment is just as lopsided — a scatter of hair-thin swaps near the diagonal costs almost nothing in MAE or MSE and walks the C-index down toward the 0.5 of a coin flip. Distance and order are simply different questions.

Classification metrics

For binary outcomes the picture changes. The true class is 0 or 1 — the two horizontal reference lines, with the points jittered slightly so they do not overlap — and the model returns a probability, plotted along the horizontal axis. Drag any point left or right to change the predicted probability for that case; the class it truly belongs to is fixed.

Randomise everything draws a fresh sample, true classes included, so cases can switch sides. Dragging moves points horizontally — only the predicted probability changes.

Brier score

The Brier score is mean squared error applied to probabilities, with the outcome treated as 0 or 1:

\[ \mathrm{BS} = \frac{1}{n}\sum_{i=1}^{n} (p_i - o_i)^2 \]

The squares are back, now measuring the horizontal distance from each point to its own class line. A confident mistake — probability 0.95 on a case whose outcome is 0 — produces a large square, while an unconfident one near 0.5 stays small. The Brier score is a proper scoring rule: it is minimised by reporting your honest probability, so it rewards calibration and discrimination together.

Figure 5. Each square is one squared probability error. Confident mistakes are punished quadratically; hedging toward 0.5 keeps the squares small but never gets them to zero.

Calibration error

Calibration asks a different question: among the cases where the model said 70%, did roughly 70% turn out positive? Sort the predictions, cut them into equal-sized bins, and compare each bin's mean prediction with the frequency actually observed in it:

\[ \mathrm{MCE} = \sum_{b=1}^{B} w_b \left( \bar{p}_b - \bar{o}_b \right)^2 \]

The dotted verticals are the bin edges, and each square spans the gap between a bin's average prediction and its observed frequency. A model can be badly calibrated and still rank cases perfectly, or be beautifully calibrated and useless at telling the classes apart — which is why this metric never travels alone.

Figure 6. Five equal-count bins. Each dot is a bin's mean prediction against its observed frequency; the squares are the squared calibration gaps, weighted by bin size.

Area under the ROC curve

AUC is the C-index again, restricted to the pairs that matter for classification: every positive case paired with every negative one. It answers "if I draw one case from each class, how often does the model give the positive one the higher probability?"

\[ \mathrm{AUC} = \frac{1}{n_{+}n_{-}} \sum_{i \in +} \sum_{j \in -} \left[ \mathbf{1}(p_i > p_j) + \tfrac{1}{2}\mathbf{1}(p_i = p_j) \right] \]

Because it only compares probabilities against each other, AUC is invariant to any monotone transformation of the predictions. Push every probability toward 0.5 and the Brier score and calibration error both deteriorate while AUC stays exactly where it was.

Figure 7. One faint segment per positive–negative pair. Green segments are correctly ordered; AUC is the fraction of them, counting ties as half.

The classification trio

Again, all three linked. Drag the point with the most extreme probability toward the middle of the plot and watch the three metrics disagree about whether you have improved the model.

Perfect sends every probability to its own true class. Move together is a toggle: dragging then shifts every probability by the same amount on the log-odds scale, which keeps them inside 0 and 1 and leaves their ordering — and so the AUC — untouched.

Figure 8. Discrimination, calibration and overall accuracy of the same 24 predicted probabilities.

Things to try

  1. Switch on Move predictions together and slide everything toward one end of the axis. The Brier score and the calibration error deteriorate steadily; the AUC readout never so much as flickers.
  2. Suppose the Brier score is stuck at a distinctly unimpressive 0.25 — the value you would get by predicting the base rate for every case. How good can you make the AUC then? And the calibration error? Can you have both at once?
  3. Now the mirror image: make the predictions perfect, then break the calibration on purpose while keeping AUC at 1. How low can the Brier score stay?
What you should find

Either one, never both. Squeeze the sample into a narrow band parked far from the base rate, positives just above negatives, and the AUC reaches 1 while the calibration error is dreadful. Pile every prediction onto the base rate instead and the calibration error vanishes while the AUC settles at 0.5. With balanced classes, 0.25 is exactly the Brier score of knowing nothing, and the score splits into a calibration penalty minus whatever discrimination you have earned: holding the total at 0.25 means every ounce of ranking has to be paid for in miscalibration, and vice versa. It is the clearest argument on this page for never reporting one of these numbers alone.

Taking stock

Each of these numbers compresses a whole sample into a single value, and each throws away something different on the way. Rank metrics discard the scale; squared metrics let one observation speak for the sample; calibration metrics ignore ordering entirely. Reporting one alone is a choice about which failures you are willing not to see — which is the argument for reporting at least two, chosen so that their blind spots do not coincide.