Skip to main content

AI glossary Responsibility

What is explainable AI (XAI)?

XAI (Explainable AI)

← All glossary terms
In short

Methods and practices that make an AI system's outputs understandable to people, so they can see why it reached a decision and judge whether to trust it.

Explained

Explainable AI (XAI) covers the methods and practices that let people understand why an AI system produced a given output. An explanation might show which inputs weighed most in a credit decision, which passage of a document supported an answer or how a model behaves across different cases. Explainability builds trust with users, helps teams find errors and bias, and supports regulatory requirements for transparency in high-stakes decisions. An explanation is only useful if it is accurate and suited to the person who receives it, which is harder than it sounds.

How AI decisions are explained

  • Inherently interpretable models: simple models such as decision trees or linear scores, whose logic can be read directly.
  • Local explanations: methods that explain a single prediction, for example by approximating the model around that case with a simpler one, as LIME does.
  • Feature attribution: scores showing how much each input contributed to an output.
  • Example-based explanations: similar past cases, or what would have to change for the decision to change.
  • For generative AI: citing the sources used, logging the steps an agent took and showing the instructions it followed.

What makes a good explanation

NIST has proposed four principles for explainable AI: a system should deliver an explanation for its outputs; the explanation should be meaningful to its audience; it should accurately reflect how the system produced the output; and the system should operate only within the conditions it was designed for, signalling when it is outside them.

In practice, a data scientist debugging a model, a customer refused a loan and an auditor need different explanations of the same decision.

Limits to keep in mind

Post-hoc explanations are approximations and can be misleading, especially for large models. A plausible explanation is not proof that the model reasoned that way. For large language models, the reasoning a model writes out is generated text, not a faithful trace of its internal computation. Explainability therefore works best combined with testing, monitoring and human oversight, not as a substitute for them.

Frequently asked questions

Why is explainable AI important?

Because people need to understand and challenge decisions that affect them, and organisations need to detect errors and bias before they cause harm. In regulated sectors such as finance, insurance and healthcare, being able to explain a decision is often a legal or supervisory expectation.

What is the difference between explainability and interpretability?

The terms are often used interchangeably. When distinguished, interpretability usually refers to a model whose workings can be understood directly, while explainability refers to explanations produced for any model, including complex ones, often after the fact.

What are examples of XAI techniques?

LIME and SHAP are widely used methods that estimate how much each input contributed to a prediction. Others include counterfactual explanations, which show what would need to change for a different outcome, and simple surrogate models that approximate a complex one.

Does the EU AI Act require explainable AI?

The AI Act sets transparency and human oversight obligations, particularly for high-risk systems, so that deployers can interpret outputs and use them appropriately. It does not prescribe a specific explainability technique. Check the regulation's text and your legal advisers for the obligations that apply to a given system.

Can large language models be explained?

Partly. You can show the sources an answer was grounded in, log prompts and tool calls, and test behaviour systematically. Explaining the internal computation of a large model is still an open research area, known as mechanistic interpretability.

Where it shows up at Thinkia

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert