Bruno Hebling Vieira·
Weblog

ML Model Performance Visualizer

Performance metrics are usually introduced as formulas, which is not a great way to build intuition for how they behave. Here each metric is instead drawn directly on a scatter plot, as lengths, areas, or segments between points. Seeing the geometry makes it much easier to understand what makes a metric grow and what it ignores.

The data are synthetic, and every panel on the page reads from the same two samples, so a point you drag in one figure moves in all the others. This lets you watch several metrics react to the same change.

Regression metrics

The regression sample has 24 observations. The true value sits on the horizontal axis and the model's prediction on the vertical one; both are standardised, so the units are standard deviations. The dashed diagonal is the line of perfect prediction, where every point would lie if the model were exactly right. Drag any point up or down to change what the model predicted for that observation.

Randomise everything throws the sample away and draws a fresh one: the true values move as well as the predictions. Dragging, by contrast, only slides points vertically, so the true values stay fixed and only the prediction changes.

Mean absolute error

The residual of an observation is the vertical gap between its point and the diagonal. Take the length of each gap, ignore its sign, and average:

\[ \mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n} \lvert \hat{y}_i - y_i \rvert \]

MAE is measured in the units of the outcome, which gives it a direct interpretation: on average, the model is off by this much. Because each error contributes linearly, a single badly predicted observation only shifts MAE by its 1/n share.

Figure 1. Each segment is one residual. MAE is the average segment length; drag a point away from the diagonal and the metric grows in proportion to the distance you travel.

Mean squared error

Square the residual instead of taking its absolute value and you get an area rather than a length:

\[ \mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n} (\hat{y}_i - y_i)^2 \]

Each shaded square has the residual as its side, so doubling an error quadruples its contribution. This is the practical difference between the two metrics: MAE reflects the typical error magnitude, while MSE weights large errors quadratically and is therefore much more sensitive to the tail of the error distribution. Drag one point far from the diagonal and its square quickly comes to dominate the average.

Figure 2. The same residuals as Figure 1, squared. MSE is the average area. A point twice as far from the diagonal contributes four times as much.

Concordance index

Both metrics above measure how close the predictions are to the truth. The concordance index instead asks whether the model puts the observations in the right order, which is a weaker requirement and often the more relevant one. Take every pair of observations with different true values and check whether the predictions rank them the same way.

\[ \mathrm{C} = \frac{\#\,\text{concordant} + \tfrac{1}{2}\,\#\,\text{tied}}{\#\,\text{comparable pairs}} \]

A model that ranks perfectly scores 1, random ordering scores 0.5, and perfectly reversed ordering scores 0. Since only the ordering matters, adding a constant to every prediction, or multiplying them all by two, leaves the C-index unchanged even though MAE and MSE change substantially. It is a rank statistic, and carries no information about calibration or scale.

Figure 3. One segment per comparable pair, coloured by whether the predictions agree with the true ordering. The C-index is the fraction of segments that are concordant.
  • Concordant pair
  • Discordant pair
  • Observation

All three together

Here are all three on the same sample, still linked; drag a point in any panel and the other two update.

Perfect drops every prediction onto the diagonal. Move together is a toggle: while it is on, dragging any point slides the whole set of predictions by the same amount, which shifts them without ever reordering them.

Figure 4. The same 24 observations under three metrics. Dragging a point in any panel updates the other two.

Things to try

  1. Drag a single point far from the diagonal without crossing its neighbours. MSE jumps, MAE rises modestly, the C-index does not move at all.
  2. Now take a point that sits close to the diagonal and drag it just past a neighbour. MAE and MSE barely register it; the C-index drops.
  3. How large can you make the mean squared error while the C-index stays at 1? Start from Make predictions perfect, switch on Move predictions together, and push the whole set away from the diagonal.
  4. And the other way round: how far can you drive the C-index down while MAE stays under, say, 0.3? You can ruin the ordering, but no point may move very far. This turns out to be a much harder constraint.
What you should find

A group shift cannot change which prediction is larger, so the C-index sits at 1 no matter how far you drag, while MAE grows linearly with the shift and MSE quadratically. Rank metrics are insensitive to any constant offset. The reverse direction behaves differently. Swapping a point across the diagonal is cheap in MAE and MSE, but it flips only the one pair it crosses, and 24 observations form 276 pairs: even twelve disjoint neighbour swaps (every point nudged past another) leave the C-index at about 0.96. Getting down to 0.5 means making half of those 276 pairs discordant, and a pair can only become discordant if the two errors together exceed the gap between the true values. Reordering on that scale costs more squared error, in the units of these plots, than ignoring the input entirely and predicting the mean for every observation. The asymmetry only goes one way: errors can grow arbitrarily large while the ordering stays perfect, but destroying the ordering necessarily costs a lot of error.

Classification metrics

For binary outcomes the picture changes. The true class is 0 or 1 (the two horizontal reference lines, with the points jittered slightly so they do not overlap), and the model returns a probability, plotted along the horizontal axis. Drag any point left or right to change the predicted probability for that case; the class it truly belongs to is fixed.

Randomise everything draws a fresh sample, true classes included, so cases can switch sides. Dragging moves points horizontally; only the predicted probability changes.

Brier score

The Brier score is mean squared error applied to probabilities, with the outcome treated as 0 or 1:

\[ \mathrm{BS} = \frac{1}{n}\sum_{i=1}^{n} (p_i - o_i)^2 \]

The squares are back, now measuring the horizontal distance from each point to its own class line. A confident mistake (probability 0.95 on a case whose outcome is 0) produces a large square, while an unconfident one near 0.5 stays small. The Brier score is a proper scoring rule: it is minimised by reporting your honest probability, so it rewards calibration and discrimination together.

Figure 5. Each square is one squared probability error. Confident mistakes cost quadratically more, while predictions near 0.5 keep the squares small but never zero.

Calibration error

Calibration asks a different question: among the cases where the model said 70%, did roughly 70% turn out positive? Sort the predictions, cut them into equal-sized bins, and compare each bin's mean prediction with the frequency actually observed in it:

\[ \mathrm{MCE} = \sum_{b=1}^{B} w_b \left( \bar{p}_b - \bar{o}_b \right)^2 \]

The dotted verticals are the bin edges, and each square spans the gap between a bin's average prediction and its observed frequency. A model can be badly calibrated and still rank cases perfectly, or be well calibrated while providing no separation between the classes, so calibration error should always be reported alongside a discrimination metric.

Figure 6. Five equal-count bins. Each dot is a bin's mean prediction against its observed frequency; the squares are the squared calibration gaps, weighted by bin size.

Area under the ROC curve

AUC is the C-index again, restricted to the pairs that matter for classification: every positive case paired with every negative one. It answers "if I draw one case from each class, how often does the model give the positive one the higher probability?"

\[ \mathrm{AUC} = \frac{1}{n_{+}n_{-}} \sum_{i \in +} \sum_{j \in -} \left[ \mathbf{1}(p_i > p_j) + \tfrac{1}{2}\mathbf{1}(p_i = p_j) \right] \]

Because it only compares probabilities against each other, AUC is invariant to any monotone transformation of the predictions. Push every probability toward 0.5 and the Brier score and calibration error both deteriorate while AUC stays exactly where it was.

Figure 7. One faint segment per positive–negative pair. Green segments are correctly ordered; AUC is the fraction of them, counting ties as half.

The classification trio

Again, all three linked. Drag the point with the most extreme probability toward the middle of the plot: the three metrics will disagree about whether the model got better or worse.

Perfect sends every probability to its own true class. Move together is a toggle: dragging then shifts every probability by the same amount on the log-odds scale, which keeps them inside 0 and 1 and leaves their ordering (and so the AUC) untouched.

Figure 8. Discrimination, calibration and overall accuracy of the same 24 predicted probabilities.

Things to try

  1. Switch on Move predictions together and slide everything toward one end of the axis. The Brier score and the calibration error deteriorate steadily, but the AUC does not change at all.
  2. Suppose the Brier score is stuck at 0.25, the value you would get by predicting the base rate for every case. How good can you make the AUC then? And the calibration error? Can you have both at once?
  3. Now the mirror image: make the predictions perfect, then break the calibration on purpose while keeping AUC at 1. How low can the Brier score stay?
What you should find

You can get either one, but not both. Squeeze the predictions into a narrow band far from the base rate, positives just above negatives, and the AUC reaches 1 while the calibration error becomes large. Set every prediction to the base rate instead and the calibration error vanishes while the AUC falls to 0.5. With balanced classes, 0.25 is exactly the Brier score of an uninformative model, and the score decomposes into a calibration penalty minus a discrimination term: if the total is held at 0.25, any gain in ranking has to be offset by miscalibration, and vice versa. This is a good argument for never reporting one of these numbers alone.

Summary

Each of these metrics compresses a whole sample into a single number, and each discards different information in the process: rank metrics discard the scale of the errors, squared metrics can be dominated by a single observation, and calibration metrics ignore the ordering. Any single metric therefore leaves some failure modes invisible, which is a good reason to report at least two whose weaknesses do not overlap.