Model Evaluation Explained: How It Works and Why It Matters in 2026

Frequently Asked Questions

01.What is model evaluation in simple terms?

Model evaluation is testing a trained model on data it has not seen, with a metric that reflects what errors cost the business. Offline evaluation happens before launch on stored, labelled examples; online evaluation follows the live outcome after launch. A sound report states the sample size and margin of error beside every score.

02.How large should a model test set be?

Large enough to resolve the smallest score gap that would change your decision. By standard sampling arithmetic, a five-hundred-question test at ninety percent accuracy carries a margin of a little over two and a half points either way at ninety-five percent confidence, and resolving one point takes roughly three and a half thousand independent questions.

03.Does the EU AI Act require model evaluation?

For high-risk AI systems, yes. Article 9 requires providers to test them before they are placed on the market, against prior defined metrics and probabilistic thresholds, and Article 15 requires accuracy levels and metrics to be declared in the instructions of use. After the Digital Omnibus amendments these duties apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems.