What is a confusion matrix?
A confusion matrix is a table that compares a classifier’s predicted labels with the labels that were actually observed. Instead of reducing model performance to one number, it preserves the direction of every mistake. In a binary problem, the four cells are true positives, false positives, false negatives, and true negatives. In a multiclass problem, the same idea expands to an n × n grid: rows represent true classes, columns represent predicted classes, and the diagonal contains correct predictions.
That structure makes the matrix a better starting point than accuracy alone. A fraud model can be 99% accurate by predicting “not fraud” for nearly every transaction, yet still miss most fraud cases. The matrix exposes those false negatives immediately. It also lets you calculate the family of metrics used to discuss different error costs: precision, recall, specificity, F1, Matthews correlation coefficient, and Cohen’s kappa.
Use the direct 2 × 2 mode when you already know TP, FP, FN, and TN. Use the two-column mode when you have rawy_true and y_pred values. The latter automatically discovers numeric class labels and works for multiclass classification. Rows in the heatmap follow true labels and columns follow predicted labels, matching the orientation returned by scikit-learn and the @kanaries/ml Metrics API.
How the main classification metrics are calculated
Accuracy: the overall hit rate
Accuracy is the number of correct predictions divided by all predictions. For a binary matrix, that is(TP + TN) / (TP + TN + FP + FN). It is easy to communicate and useful when classes are reasonably balanced and different mistakes have similar costs. It becomes misleading when one class dominates. Always inspect support per class and compare accuracy with recall, precision, or macro F1 before declaring a model successful.
Precision: how trustworthy positive predictions are
Precision answers: “Of everything predicted as positive, how much was truly positive?” Its binary formula isTP / (TP + FP). Choose precision as a primary metric when false alarms are expensive. Examples include a system that automatically blocks legitimate payments, removes lawful content, or sends scarce sales leads to a human team. High precision means the positive queue contains relatively little noise, although the model may still fail to find many real positives.
Recall: how many real positives were found
Recall, also called sensitivity or the true-positive rate, answers: “Of all real positives, how many did the model detect?” The formula is TP / (TP + FN). Recall matters when missing a positive is dangerous or costly, such as disease screening, safety-event detection, or fraud review. Raising recall often lowers precision because a more permissive decision threshold captures more positives and more false positives. The correct trade-off depends on the action triggered by a prediction, not on a universal target.
Specificity: how well negatives are rejected
Specificity is the true-negative rate: TN / (TN + FP). It complements recall by measuring performance on the negative class. A medical screening test, for example, can have high sensitivity but poor specificity, catching most cases while also producing many unnecessary follow-ups. For multiclass input this calculator treats each class as positive in turn, computes the one-vs-rest specificity, and reports the unweighted macro average.
F1 score: balancing precision and recall
F1 is the harmonic mean of precision and recall. Unlike an arithmetic mean, the harmonic mean stays low when either component is low, so a model cannot compensate for terrible recall with excellent precision or vice versa. The compact matrix formula is 2TP / (2TP + FP + FN). F1 ignores true negatives, which is useful when the positive class is rare but means it does not describe every classification problem. If correct rejection of negatives matters independently, read F1 alongside specificity or MCC.
Micro, macro, and weighted F1 for multiclass models
Multiclass evaluation produces one precision, recall, and F1 value per class. An average is needed when a dashboard or experiment table requires a single summary. Macro averaging calculates every class score independently and then gives each class equal weight. A class with 20 examples therefore matters as much as a class with 20,000 examples. This is a strong default when minority-class quality is a product requirement.
Weighted F1 also calculates each class separately but weights the results by true support. It reflects the class mix in the evaluated dataset and can be easier to compare with overall accuracy. However, a very large majority class can hide weak minority performance. Micro F1 pools all per-sample decisions before calculating the metric. In ordinary single-label multiclass classification, every error creates one false positive and one false negative, so micro F1 is numerically equal to accuracy.
There is no universally best average. Report macro F1 when every class deserves equal attention, weighted F1 when the observed class distribution is operationally meaningful, and micro F1 when each individual prediction should carry equal weight. The per-class bars above remain essential because two models can share the same average while failing on different classes.
Why MCC and Cohen’s kappa are included
Matthews correlation coefficient summarizes the entire confusion matrix and remains informative when class sizes are very different. It behaves like a correlation between true and predicted labels: 1 means perfect agreement, 0 means no better than the relevant chance structure, and -1 represents complete disagreement in the binary extreme. The multiclass formulation uses all rows and columns rather than averaging a collection of binary F1 scores.
Cohen’s kappa measures observed agreement after subtracting the agreement expected from the label marginals. It is often used for annotator agreement but also helps describe classifiers when the frequency of predicted classes differs from the frequency of true classes. Kappa and MCC answer different statistical questions, so agreement between them is reassuring while a large difference is a reason to inspect the matrix and class supports more closely.
JavaScript implementation and reproducible Python comparison
Every result above is calculated locally with the JavaScript implementation in @kanaries/ml. The code panel emits the exact label arrays currently in the calculator and calls confusionMatrix,accuracyScore, precisionScore, recallScore, and f1Score; the page derives weighted F1, MCC, and kappa from that matrix with the corresponding sklearn formulas. The Python tab sends the same data to scikit-learn. This side-by-side form makes the tool useful both for a quick answer and for moving a verified calculation into a browser application, Node.js service, notebook, or test suite.
The CSV export includes the displayed matrix and headline metrics. The PNG export captures the heatmap for reports and presentations. Because the calculation happens in the browser, sensitive labels are not transmitted to a calculation API. For a production evaluation pipeline, keep class order explicit, record the positive label and decision threshold, and save the dataset version alongside the exported scores so later comparisons remain reproducible.