AI Red Teaming: A Practical Pre-Launch Testing Guide

AI red teaming tests whether an AI system can be pushed into unsafe behaviour, exposing private information or taking actions it should not. Before launch, it helps you discover failures that ordinary acceptance testing may miss. This guide explains how to scope the assessment, agree safe testing boundaries, evaluate findings and make an evidence-based release decision.

What AI red teaming tests—and why it matters before launch

Functional testing asks whether a system completes its intended tasks. AI red teaming asks what happens when someone challenges its instructions, supplies misleading information or tries to use its capabilities against the organisation’s interests.

The target is the whole application, not just its underlying model. An assistant may combine a model with document retrieval, conversation history, identity controls and tools that change business records. Each connection introduces different risks.

A model that refuses an inappropriate request in isolation may behave differently when the request appears inside a retrieved document. An assistant that answers safely may still have excessive permissions to act.

AI red teaming does not replace security testing, privacy assessment or quality evaluation. It connects them around a specific question: can this AI system be induced to cause harm in its actual operating context?

For the broader organisational context, this guide to when red teaming services are useful explains how an assessment differs from routine security checks.

Step 1: Define the system and the launch decision

Start with a short system brief that the business owner, engineering lead and security team can all understand.

Record:

• Purpose: the tasks the system is intended to perform.
• Users: employees, customers, citizens, partners or a combination.
• Information access: the documents, records and conversation history it can use.
• Actions: whether it only provides answers or can modify records, send messages or initiate workflows.
• Authority: whose permissions apply and where approval is required.
• Consequences: what could happen if an answer or action is wrong.

Then state the decision the assessment must support. For example: can a staff assistant launch with access to internal guidance, or must access to sensitive case records remain disabled?

This prevents an ambiguous outcome such as “the model was tested”. The deliverable should support a defined release scope, including features that may need to be restricted.

Step 2: Turn business harms into test objectives

Do not begin with a generic collection of challenging prompts. Begin with the harms your organisation needs to prevent.

For a public-facing information assistant, a serious failure might be presenting invented eligibility guidance as official policy. For an internal case assistant, it might be exposing records outside the user’s role. For an agent that changes business data, it might be acting without valid authorisation.

Translate each harm into a test objective:

• Can the assistant distinguish approved guidance from unsupported claims?
• Does access control remain effective throughout a conversation?
• Can untrusted information influence decisions beyond its legitimate role?
• Does the agent stop when authority or intent is unclear?
• Can a reviewer reconstruct what information supported an action?

Prioritise by consequence and exposure. A cosmetic response problem should not receive the same attention as unauthorised disclosure or an irreversible action.

Output: a risk register connecting each harm to a test objective, an owner and an expected safeguard.

Step 3: Agree safe rules of engagement

The assessment itself must not put operations or sensitive information at unnecessary risk.

Agree the following before testing:

• The environments, application components and user roles in scope.
• Permitted testing activities and explicit exclusions.
• Approved test data, using synthetic or appropriately protected information.
• Restrictions on external communications and consequential actions.
• Stop conditions and an escalation contact.
• How evidence will be stored, accessed and deleted.

Use an isolated environment that represents the intended release as closely as practical. Differences in permissions, retrieval sources or approval workflows can make results misleading, so document them.

Where a live environment is necessary, require explicit approval and additional safeguards. Testing authority should never be inferred from a general instruction to assess the AI.

Step 4: Assess the complete workflow

Organise testing around the places where the system receives information, makes decisions and exercises authority. Keep scenarios relevant to real use rather than judging success by the number of unusual prompts submitted.

Inputs and retrieved information

Assess how the system handles conflicting instructions and untrusted material from documents, messages or other connected sources. Information supplied for analysis should not automatically gain authority over the application’s rules.

Check whether responses remain appropriately bounded when source material is incomplete, contradictory or outside the assistant’s approved knowledge.

Identity and information boundaries

Evaluate whether users can access only the information they are entitled to see. Include conversation history and generated summaries in the assessment, not just the original source records.

A refusal written by the model is not a substitute for enforced access control in the application and its connected systems.

Actions and human approval

For agentic systems, assess what happens after the answer. Confirm that tool permissions, approval requirements and action boundaries match the intended business process.

Ask whether approval covers the actual proposed action, whether the system handles ambiguous requests safely, and whether failures leave the workflow in a controlled state.

IntellifyAi’s Cyber Security offering includes AI model and LLM red teaming as well as agentic AI red teaming. The distinction matters: testing generated answers alone does not establish whether an action-taking system is safe to release.

Step 5: Record findings as business risks

A useful finding is more than a transcript showing an unexpected answer. It should explain what boundary failed, why it matters and what evidence supports that conclusion.

For each finding, capture:

• The affected feature and tested configuration.
• The expected behaviour and observed deviation.
• The potential consequence and affected users or information.
• The conditions under which the issue appeared.
• Evidence sufficient for an authorised team to investigate.
• A remediation owner and retest requirement.

Separate observed facts from possible consequences. If testing demonstrated inappropriate disclosure in a controlled environment, say so; do not imply that a real breach occurred.

Avoid reducing the assessment to a single pass rate. AI behaviour can vary, and an aggregate result can conceal a serious failure in a consequential workflow.

Step 6: Fix causes, retest and set release conditions

Choose safeguards that address the failed boundary. Revised model instructions may help, but some problems require application-level changes: narrower data access, restricted tools, clearer approval steps or a reliable escalation path.

Retest the original failure and related workflows. Confirm that the fix preserves legitimate tasks rather than merely causing the assistant to refuse everything.

The release review should record:

• Which findings have been resolved and retested.
• Which risks remain and who accepts them.
• Which capabilities or users remain outside the approved scope.
• What monitoring and incident handling must be ready.
• Which changes will trigger reassessment.

A restricted launch can be more defensible than a binary go-or-no-go decision. For example, an assistant may be permitted to draft a proposed update while direct execution remains disabled.

Red teaming provides evidence, not a guarantee. Changes to models, information sources, permissions and workflows can alter the risk after launch.

Frequently asked questions

Is AI red teaming the same as penetration testing?

No. Penetration testing generally examines exploitable weaknesses in applications and infrastructure, while AI red teaming also examines model behaviour, instruction handling and AI-enabled actions. The two can overlap and should inform each other when an AI application connects to business systems.

When should AI red teaming happen?

Start once the intended workflow and risk boundaries are clear enough to assess. Complete a representative assessment before launch, then revisit it when material changes affect the model, connected data, permissions or available actions.

Can an organisation test an AI system internally?

Yes, provided the team has suitable expertise, clear authorisation and safe testing conditions. Independent review can help challenge assumptions that the delivery team shares, but it does not remove the need for internal ownership of fixes and release decisions.

Does passing an assessment mean an AI system is safe?

No assessment can establish safety in every future situation. A defensible outcome is documented evidence about the tested scope, verified safeguards, explicit residual risks and a plan to detect and respond to failures after release.