Skip to main content

AI glossary Responsibility

What is explainability in AI (XAI)?

← All glossary terms
In short

The ability to explain, in terms people can follow, how an AI model reached a given output or decision (also called explainable AI, XAI).

Explained

Explainability is the capacity to describe how an AI model arrives at its outputs in a way that the people affected can understand. It covers global explanations (which factors drive the model overall) and local ones (why this customer was declined). It matters for trust, for debugging models, for detecting bias and for regulatory compliance, since laws such as GDPR and the EU AI Act give people and authorities rights to meaningful information about automated decisions. Explainable AI (XAI) is the set of methods used to produce those explanations.

Main techniques

  • Inherently interpretable models: linear models, decision trees, scorecards.
  • Feature attribution: methods such as SHAP and LIME that estimate how much each input contributed to a prediction.
  • Counterfactuals: what would have to change for the decision to be different.
  • Example-based explanations: similar past cases that the model treated the same way.
  • For language models: citing the sources used, showing the steps taken by an agent and logging tool calls.

Explainability vs interpretability

The terms are often used as synonyms. When they are distinguished, interpretability refers to how understandable the model itself is, and explainability to the explanations produced about its behaviour, which can apply even to a black-box model.

NIST's principles add a useful test: an explanation should exist, be meaningful to its audience, accurately reflect how the system works, and the system should say when it is operating outside its knowledge limits.

When it matters most

  • Decisions about people: credit, insurance, hiring, healthcare, public services.
  • High-risk systems under the EU AI Act, which require transparency and human oversight.
  • Models that must be audited or defended before a regulator or a court.
  • Any system where users need to know when to trust the output and when to check it.

Frequently asked questions

What is explainable AI in simple terms?

It is AI that can show why it gave a particular answer. Instead of only saying "declined", an explainable system can point to the factors that weighed most in that decision.

Why is explainability important in AI?

It lets people trust and challenge AI decisions, helps teams find errors and bias, and is required in many regulated settings. Without it, it is hard to know whether a model is right for the right reasons.

Does the EU AI Act require explainability?

The EU AI Act requires high-risk systems to be transparent enough for deployers to interpret their output and use it properly, and to allow human oversight. It also gives affected people a right to an explanation of certain decisions taken with high-risk systems. GDPR adds its own rules on automated decision-making.

What are SHAP and LIME?

They are two widely used methods for explaining individual predictions. LIME fits a simple model around one prediction to approximate the complex one locally; SHAP assigns each input feature a contribution based on game theory. Both work with many kinds of models.

Can large language models explain their answers?

They can produce a plausible explanation, but that text is not guaranteed to reflect how the model actually computed the answer. In practice, explainability for LLM systems relies on citing sources, logging the steps of an agent and evaluating outputs, rather than trusting the model's own account.

How to put it to work

Where it shows up at Thinkia

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert