Skip to main content
Contact

AI how-to guides Risk, compliance and governance

How to red team your AI agents automatically

Short answer

Write down what each agent must never do, then use a battery of attacks, partly generated by another model, to try to make it do exactly that, and score every run automatically. Run the battery on every change to prompts, tools or models, and block releases that regress. Automation gives coverage and repeatability; people still design the threat model and look for the attacks nobody has scripted yet.

Updated: · Thinkia

What AI does

  • Generates and mutates adversarial prompts: injections, jailbreaks, role-play and multi-turn manipulation.
  • Plants malicious instructions in the documents, emails and tool outputs the agent reads.
  • Plays the attacker over many turns and adapts to the agent's answers.
  • Scores each run against your policies: data leaked, forbidden tool called, unsafe content produced.
  • Re-runs the whole battery on every change and reports regressions.

What stays with people

  • Building the threat model: what the agent can reach, who might attack it and what harm matters most.
  • Deciding what counts as a failure and which failures block a release.
  • Manual, creative red teaming to find new attack patterns the automated suite does not cover yet.
  • Fixing root causes: permissions, tool design and approvals, not only prompt patches.

How to put it in place

  1. Map the agent and its blast radius

    List the agent's tools, data sources, permissions and the actions it can take on its own. That defines what an attacker could make it do and where the damage would land.

  2. Turn policies into testable failure cases

    Write concrete cases: leaks personal data, calls a payment tool without approval, follows an instruction found in an email, reveals its system prompt. Each case needs a clear pass or fail check.

  3. Build the attack library

    Start from public taxonomies of attacks on language models and agents, then add cases specific to your domain. Include indirect attacks through documents and tool outputs, not only direct chat prompts.

  4. Automate attackers and judges

    Use an attacker model to generate and vary attacks over several turns, and automated checks or a judge model to score the results. Calibrate the judge against a sample reviewed by people so its verdicts can be trusted.

  5. Wire it into the release pipeline

    Run the suite on every change to prompts, tools, models or retrieval sources in a sandbox with fake data. Block releases on critical failures and keep each run's results as evidence.

  6. Keep testing in production

    Monitor real traffic for attack patterns and unusual tool calls, and feed what you find back into the library. Repeat manual red teaming periodically and after major changes.

What you need

  • A sandbox copy of the agent with its tools pointed at test systems and synthetic data.
  • A written policy of what the agent may and may not do, per tool and per data type.
  • An attack library and an attacker model, plus a way to score results automatically.
  • A CI/CD pipeline where the suite can run and block a release.
  • Logs of agent conversations and tool calls in production.
  • A named owner who decides which failures are acceptable and tracks the fixes.

When it is not worth it

  • Prototypes with no access to real data or tools, where a quick manual review is enough.
  • Agents whose policies are not defined yet: without a clear notion of failure, scoring is noise.
  • When nobody will act on the findings: a report that does not change permissions or approvals adds no safety.

Risks and how to control them

  • False sense of security: a passing suite only shows the attacks you tried; keep manual red teaming and update the library.
  • Unreliable judges: a judge model can miss failures or flag harmless output; calibrate it against human review.
  • Testing against real systems: attacks can trigger real actions; run in a sandbox with test credentials and fake data.
  • Fixing symptoms only: patching prompts while the agent keeps excessive permissions; reduce tools, scopes and autonomy instead.
  • Regulatory exposure: if the agent falls in an AI Act high-risk area (employment, credit, essential services), testing and robustness evidence matter; check obligations with your legal team.

How Thinkia does it

We start by mapping every AI use case and classifying it by EU AI Act risk tier, with accountable owners. Lifecycle gates, documentation and monitoring are then aligned to the regulation and your internal policy, and a subset of models and vendors runs through the full workflow first, with gaps turned into a backlog.

For higher-risk systems we design the human oversight layer: who reviews, what triggers review and how decisions are logged, with an audit trail across AI systems so every decision can be explained. It is operational delivery, not legal advice.

The Thinkia solution behind this guide

Thinkia products involved

Frequently asked questions

What is AI red teaming?

Deliberately attacking an AI system, the way an adversary would, to find where it breaks: leaking data, following injected instructions, producing harmful content or taking actions it should not. The aim is to fix those weaknesses before someone else finds them.

Can red teaming be fully automated?

Not fully. Automation covers known attack patterns at scale and catches regressions on every change. People are still needed to model threats, judge borderline results and invent new attacks.

Why are agents riskier than chatbots?

Because they act. An agent with tools can send emails, change records or move money, so a successful attack becomes a real action rather than a bad answer. That is why permissions and approvals matter as much as the model.

What is indirect prompt injection?

An attack hidden in content the agent reads, such as a web page, an email or a document, instead of in the user's message. The agent may treat that text as an instruction unless it is designed to treat it as data.

How often should we run it?

Automatically on every change to prompts, tools, models or data sources, plus continuous monitoring in production. Manual exercises are worth repeating periodically and after significant changes.

Does the EU AI Act require red teaming?

The regulation sets testing, accuracy and robustness requirements for high-risk systems and adversarial testing for general-purpose models with systemic risk. Whether and how they apply to your agent depends on its use; check with your legal team.

Related guides

Related decisions

Key terms

Thinkia articles

Sources

Want to do this in your organisation? Let's talk.

Talk to an AI Expert