Skip to main content

AI glossary Operations

What is AI model evaluation?

← All glossary terms
In short

The process of measuring how well an AI model or application performs: accuracy, relevance, safety, bias, cost and other criteria.

Explained

Evaluation is the systematic measurement of how well an AI model or AI application performs against the criteria that matter for its use: accuracy, relevance, groundedness, safety, bias, robustness, latency and cost. For language models it combines automatic metrics, test sets with known answers, judgements by other models and review by people. Public benchmarks compare models in general; your own evaluations tell you whether a system works for your task and data. Solid evaluation is needed before a model goes to production and again every time the model, prompt or data changes.

How AI systems are evaluated

  • Reference-based tests: a set of inputs with expected outputs, scored automatically for exact match, similarity or correctness.
  • Model-graded evaluation: another model scores answers against a rubric, useful for open-ended text but itself in need of checking.
  • Human review: experts rate a sample of outputs, the most reliable signal for quality and the most expensive.
  • Adversarial testing: red teaming and stress tests to find unsafe or broken behaviour.
  • Online evaluation: signals from real use in production, such as user feedback, escalations and task completion.

Benchmarks vs your own evals

Public benchmarks measure models on shared tasks so that they can be compared. They are useful to shortlist models, but they rarely reflect your documents, your users or your definition of a good answer, and models can be tuned to score well on them.

Your own evaluation set is built from real cases in your domain, with the answers your experts would accept. It is the only way to know whether a change of model, prompt or retrieval makes your system better or worse.

Evaluating agents and RAG

For retrieval-augmented systems, evaluate the parts separately: did retrieval find the right passages, and did the answer stay faithful to them? Errors in one look like errors in the other unless you measure both.

For agents, the final answer is not enough. Look at the path: which tools were called, with what arguments, whether the steps were safe and whether the agent stopped when it should have. Cost and number of steps matter too.

Frequently asked questions

What are evals in AI?

Evals are the tests used to measure an AI system's behaviour: a set of inputs, expected outcomes or scoring rules, and a way to run them repeatedly. They work like a test suite for software, but they score quality rather than only pass or fail.

How do you evaluate an LLM?

Define what a good output means for your use case, collect representative test cases, and score outputs with a mix of automatic checks, model-graded rubrics and human review. Run the same set whenever you change the model, prompt or data, and compare.

What is LLM-as-a-judge?

It is the practice of using a language model to grade the outputs of another model against a rubric. It scales better than human review, but it has its own biases, so its scores should be checked against human judgements on a sample.

What metrics are used to evaluate AI models?

It depends on the task. Common ones are accuracy, precision and recall for classification; relevance and faithfulness for retrieval and generation; plus safety, bias, robustness, latency and cost. Frameworks such as Stanford's HELM measure several of these together.

Why is evaluation important before deploying AI?

Because a model that looks good in a demo can fail on real cases, in ways that harm users or the business. Evaluation gives evidence of quality and risk before launch and a baseline to detect regressions afterwards. Risk frameworks such as NIST's AI RMF treat measurement as a core function.

Decision guides that use this term

How to put it to work

Where it shows up at Thinkia

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert