In October 2025, researchers from OpenAI, Anthropic, and Google DeepMind released a study titled The Attacker Moves Second. Subsequently published at USENIX Security 2026, the study examined 12 defenses against LLM jailbreaks and prompt injection. By adapting their attacks to each defense, the researchers achieved attack success rates above 90% against most of them.
Most of those defenses had reported near-zero attack success rates in their original evaluations. The adaptive methods included gradient-based optimization, reinforcement learning, search, and human-guided exploration. The defenses had not been weak by the standards of their own papers. They had been measured against attackers who did not adapt.
A second data point comes from the other end of the scale. In a competition run by Gray Swan with the UK AI Security Institute in March and April 2025, almost 2,000 participants submitted about 1.8 million attacks against 22 frontier AI agents across 44 realistic deployment scenarios, and more than 60,000 of them produced policy violations. Gray Swan reported that every model was successfully attacked, though some held up better than others. The targets were not simple chatbots. They were agents with tools and memory, behaving like financial assistants and shopping bots.
Together these results explain why AI red teaming exists as a discipline. A fixed list of test prompts, or a vendor's claim of a low attack success rate, says little about how a system behaves against someone who is trying, adapting, and trying again. Red teaming supplies that someone, under controlled conditions and with permission, before a real attacker does.
This guide is an introduction. It covers what AI red teaming is and how it differs from benchmarking and from conventional security testing, which attack categories a red team works through, why agents need particular attention, how language changes the picture for Indian deployments, how an engagement is structured, what a useful report contains, and where Indian expectations are heading. It is written for CISOs, product security leads, and engineering teams who are about to ship, or have already shipped, AI features that touch real data and real actions.
What AI Red Teaming Is, and Is Not
AI red teaming is goal-driven, adaptive, adversarial testing of an AI system as it is deployed. The tester starts from an objective, such as extracting confidential data, making an agent take an unauthorized action, or bypassing a policy the business depends on, and uses whatever combination of techniques achieves it. Success is defined by the objective, not by coverage of a checklist.
Talk to Our Security Experts →
Microsoft's AI Red Team, which has red-teamed more than 100 generative AI products, summarized its experience in eight lessons. Several shape this guide: understand what the system can do and where it is applied; AI red teaming is not safety benchmarking; breaking an AI system does not require computing gradients; automation extends coverage but human judgment remains essential; and the work of securing AI systems is never complete.
AI Red Teaming and Its Neighbors
| Practice | Core question | Limitation |
|---|---|---|
| Safety benchmarking | How does the model score on a fixed set of prompts? | Static tests can be learned around, and a good score says little about an adaptive attacker |
| AI security testing | Which known weakness categories exist in the application? | Strong on coverage, less focused on a determined attacker's chosen goal |
| AI red teaming | Can an adversary achieve a specific objective against this system? | Narrower by design and more dependent on tester skill |
| Traditional red teaming | Can an attacker breach the organization and avoid detection? | Does not examine model behavior, prompts, retrieval, or agent tools |
The categories overlap, and mature programmes use all of them. Security testing finds the known problems efficiently. Red teaming finds out what happens when someone combines those problems, or invents a new one, in pursuit of a goal.
Why AI Safety Benchmarks Cannot Replace Red Teaming
The Attacker Moves Second study makes a methodological point that applies well beyond its twelve defenses. In conventional security, defenses are expected to withstand an attacker who knows how they work. In the LLM field, the authors argue, evaluations have too often used static sets of known attacks or low-effort optimization, which produces impressive numbers and little assurance. When attackers adapted to each defense's design, none of the four families of technique tested held up.
Three practical consequences follow. A guardrail or filter's published attack success rate should be treated as a starting hypothesis, and a buyer is entitled to ask how it was measured and whether the attacker was allowed to adapt. Defenses should be layered, with the assumption that any single one can be bypassed. And the most reliable protection is to limit what a successfully manipulated model can reach, since that does not depend on stopping the manipulation.
The Gray Swan results point the same way for agents. Even the more resilient models in the competition were broken by a large enough, creative enough crowd. That does not mean deployment is unwise. It means the evidence that a given deployment is resilient has to come from adversarial testing against that deployment, not from a general reputation for safety.
AI Red Teaming Attack Surface: Key Risks to Test
The categories below describe objectives and the controls they test. They are deliberately conceptual. A red team's value lies in how techniques are chosen, combined, and adapted to a specific system, not in any list of prompts.
| Category | What the red team is trying to achieve | Control being tested |
|---|---|---|
| Jailbreaks | Make the system produce content or behavior its policy forbids | Alignment, content filters, system-level policy enforcement |
| Direct prompt injection | Override the operator's instructions through user input | Instruction hierarchy and input handling |
| Indirect prompt injection | Plant instructions in documents, emails, web pages, or tool outputs the system will read | Treatment of untrusted content and separation from instructions |
| Data leakage and extraction | Recover system context, retrieved documents, other users' data, or memorized training content | Context protection, retrieval access control, tenant isolation |
| Tool and agent abuse | Make an agent take actions beyond what its task requires | Tool permissions, approval gates, authorization outside the model |
| Memory and retrieval poisoning | Insert content that persists and influences later behavior | Integrity of knowledge bases, memory, and ingestion pipelines |
| Supply chain compromise | Tamper with a model, plugin, tool, or dataset the system depends on | Provenance, version pinning, vetting of third-party components |
| Resource abuse | Drive up cost or exhaust capacity | Rate limits, budgets, timeouts |
| Multi-turn and multi-modal manipulation | Build toward a goal gradually, or use images, audio, or files to carry the attack | Controls that hold across a conversation and across input types |
Two frameworks give red teams a shared structure. The OWASP GenAI Red Teaming Guide, first released in January 2025, organizes the work into model evaluation, implementation testing, infrastructure assessment, and runtime behavior analysis. MITRE ATLAS is a living knowledge base of adversary tactics and techniques against AI-enabled systems, modeled on ATT&CK, whose 2025 and 2026 releases added techniques aimed at generative AI and agents, among them retrieval poisoning. Mapping findings to both makes results comparable and easier for governance teams to use.
For practitioners seeking a more detailed engagement methodology, OWASP's GenAI Red Teaming Manual Version 1.0, released in 2026 as a draft for public comment, provides an eight-phase approach spanning planning, reconnaissance, attack-surface mapping, exploitation, persistence, impact assessment, reporting, and remediation. It complements the broader GenAI Red Teaming Guide by offering a more operational structure for conducting assessments.
AI Agent Red Teaming: Testing Tools, Permissions and Memory
Agents change the stakes because they act. A chatbot that is manipulated says something wrong. An agent that is manipulated can send an email, modify a record, call an internal service, or spend money, and it can do so in a chain of steps where an early error carries forward. OWASP's 2026 LLM list places Excessive Agency third, and the project published a separate Top 10 for Agentic Applications in December 2025.
Red teaming an agent asks questions that a chatbot test never reaches.
| Test dimension | Question the red team asks |
|---|---|
| Permissions | What can each tool actually do, and how does that compare with what the task needs? |
| Instruction hierarchy | Can content the agent retrieves override instructions from the operator or the user? |
| Memory | Can poisoned content persist across sessions and shape later actions? |
| Delegation | In a multi-agent setup, does one compromised component spread its influence to the others? |
| Limits | What stops a loop, a runaway task, or an expensive action? |
| Human approval | Does the approval step show a person what will really happen, or a summary an attacker can shape? |
| Identity | Does the agent act with its own tightly scoped credentials, or with a broad shared account? |
Findings in this area tend to be architectural. The fix is rarely a better prompt. It is narrower credentials, authorization enforced in the backing service, approval gates for irreversible actions, and logging that lets a reviewer reconstruct what the agent did and why.
The scenario below shows how these test dimensions combine along a single attack path, from initial access through remediation and retest.
| Stage | Example scenario | What the red team validates |
|---|---|---|
| 1. Initial access | An employee asks an AI assistant to summarize a document from an approved internal repository. | Whether the assistant processes untrusted document content safely. |
| 2. Indirect prompt injection | The document contains instructions attempting to redirect the assistant's behavior. | Whether document content can override the intended task or influence tool selection. |
| 3. Privilege boundary | The assistant attempts to access information outside the employee's authorized scope. | Whether authorization is enforced by the underlying data service rather than trusted to the model. |
| 4. Unauthorized action | The assistant attempts to transmit sensitive information through an integrated tool. | Whether data-loss controls, tool permissions, and approval requirements prevent the action. |
| 5. Detection and containment | The test team examines logs and attempts to reconstruct the interaction. | Whether the organization can identify the event, trace the tool calls, and determine the potential impact. |
| 6. Remediation and retest | The team introduces appropriate authorization and execution controls and repeats the scenario. | Whether the attack path is closed and related workflows remain secure. |
Multilingual AI Red Teaming: Hindi, Hinglish and Indian Languages
Indian deployments face a testing question that many global methodologies treat lightly: how do safeguards hold up in Indian languages and in code-mixed text such as Hinglish? The evidence is mixed, and the mix is informative.
Early work found that translating unsafe English inputs into low-resource languages could bypass GPT-4's safeguards, with combined low-resource languages reaching roughly a 79% bypass rate. A May 2026 study of four African languages found that simple translation no longer reliably defeats current commercial models, but that multi-turn red teaming by native speakers raised the average jailbreak rate from 59.8% to 75.8% compared with automated translation. Translation quality, not language alone, determined success. For Indian languages, a recent preprint benchmark, IndicSafeEval, tested persuasion-style attacks in Bengali, Hindi, Marathi, and Punjabi and reported that the open models it evaluated were noticeably more vulnerable in Indian languages than in English, with one model above 60% attack success in Marathi and Punjabi against 27% in English. That work covers smaller open models and is a preprint, so it should be read as a signal to test, not as a verdict on every system.
The practical lesson is to avoid both assumptions. English results do not automatically transfer to Hindi, Tamil, or Hinglish, and weaknesses found in older models do not automatically persist in newer ones. A credible Indian engagement tests in the languages the users actually speak, with native-speaker testers who can write natural, context-aware attacks and judge whether a response is harmful.
AI Red Teaming Methodology: Seven Steps from Scoping to Retesting
The structure below follows the published frameworks and the lessons above. Individual engagements adapt it to the system and its risks.
| Step | What happens | What it produces |
|---|---|---|
| 1. Objectives and threat model | Agree which harms matter most: data exposure, unauthorized actions, policy bypass, cost, reputation | A short list of attacker goals tied to business impact |
| 2. Scope and rules of engagement | Fix authorization, environments, test accounts, data handling, and safeguards for tests that can trigger real actions | A signed scope and a safe-execution plan |
| 3. Reconnaissance | Map the model, instructions, tools, data sources, identities, and integrations | An architecture picture and a prioritized attack map |
| 4. Attack campaign | Pursue each objective with manual and automated techniques, including multi-turn, indirect, and tool-based paths | Evidence of what worked, what failed, and why |
| 5. Measurement | Repeat promising attacks to estimate success rates, then rate severity by what the attacker can reach | Rates, not anecdotes, and severity tied to impact |
| 6. Reporting | Document reproducible findings, root causes by layer, and prioritized fixes | A report engineering and governance teams can both act on |
| 7. Retest and regression | Re-run the attacks after fixes and keep them as a regression suite | Confirmation that fixes hold, and a guard against regressions |
Two features deserve emphasis. The second step carries more weight for AI than for most systems, because a manipulated agent can take real actions during a test, so isolated environments and scoped credentials are the default. In India, written authorization also matters legally: Sections 43 and 66 of the Information Technology Act, 2000 treat unauthorized access to a computer system as a civil and, with dishonest intent, criminal matter.
The fifth step reflects a property of language models that changes testing practice: output varies from run to run, so one failed attempt proves little. Microsoft's team built its open-source PyRIT framework to scale attack generation and scoring, and its lessons are equally clear that automation supports human testers and does not replace them. Human judgment decides what to try, how to chain weaknesses, and whether an output is harmful in its context.
What a Useful Red Team Report Contains
| Element | Why it matters |
|---|---|
| Findings stated as achieved objectives | Leadership can see which business harms were demonstrated, not only which techniques exist |
| Success rates across repeated trials | Separates a reliable attack from a lucky one |
| Reachable impact | Severity depends on what the attacker gains, not how clever the technique was |
| Reproduction detail for the engineering team | Enables verification and fixing without re-engaging the tester |
| Mapping to OWASP, ATLAS, and NIST AI 100-2 | Supports comparison across engagements and use by governance teams |
| Root cause by layer | Shows whether the fix belongs in the model, the prompt, the architecture, the tool, or monitoring |
| Retest results | Confirms that fixes worked |
| Residual risk statement | States plainly what remains exposed after the fixes |
The report is itself a sensitive document, because it describes how the system can be broken. It needs the same access control and retention discipline as any other security finding.
Example Finding: Unauthorized Data Access Through an AI Agent
The table below shows how a single finding can be documented using the elements above.
| Field | Example |
|---|---|
| Finding title | Unauthorized Data Access Through an Overprivileged AI Agent |
| Affected component | Enterprise AI assistant and integrated document retrieval service |
| Security objective | Determine whether an authenticated user can obtain information beyond their authorized access. |
| Attack scenario | The red team supplies untrusted instructions through a document processed by the assistant and attempts to induce retrieval of restricted information. |
| Expected behavior | The retrieval service enforces the user's authorization boundaries, regardless of the instructions supplied to the model. |
| Observed behavior | Hypothetical example: the assistant retrieves restricted content because the backing service uses broader permissions than the requesting user possesses. |
| Business impact | Potential exposure of confidential information and unauthorized cross-user or cross-role access. |
| Root cause | Authorization is not enforced consistently at the data-access layer. |
| Recommended remediation | Enforce user-context authorization at the retrieval service, restrict service identities, and validate access decisions independently of model instructions. |
| Retest criteria | Repeat the original attack and related authorization tests; verify that restricted content remains inaccessible. |
Note: This is an illustrative finding, not a claim that SecNinjaz or another testing organization has observed this issue in a client environment.
Where Indian Expectations Are Heading
CERT-In's May 2026 guidance, CISG-2026-02, addresses reducing exposure to AI-assisted vulnerability exploitation across digital infrastructure. Its recommendations include adversarial AI testing as part of a broader approach to security validation and resilience. Organizations should distinguish these recommendations from specific statutory or regulatory obligations applicable to their sector.
On the data protection side, Section 8(5) of the DPDP Act, which takes effect in May 2027, requires reasonable security safeguards, and Rule 6 of the DPDP Rules 2025 sets minimum measures that include access control and monitoring. A red team exercise that tests whether an AI system leaks personal data produces direct evidence about whether those safeguards operate.
The Business Risks of Skipping AI Red Teaming
| Gap | Consequence |
|---|---|
| Reliance on a vendor's reported attack success rate | Confidence built on a non-adaptive test, the pattern the Attacker Moves Second study exposed |
| Testing only the model, not the deployed system | Missed tool, retrieval, and identity weaknesses where most real impact sits |
| Agents granted broad permissions untested | A single manipulated output can trigger a real action |
| English-only testing for Indian users | Gaps in Indian languages and code-mixed text go unseen |
| No retest after changes | Fixes that do not hold, and silent regressions after model or prompt updates |
| No evidence of tested safeguards | Difficulty showing an auditor or regulator that protections were examined |
Cost data points the same way. IBM's 2026 breach research reports average losses of $5.89 million for prompt injection incidents involving organizations' own AI, and found that 92% of organizations with such a breach lacked proper AI access controls.
A Maturity Model for AI Red Teaming
Level 1: AI features ship after functional testing, with security assumed to come from the model provider ↓ Level 2: A one-off manual attempt at prompt injection runs shortly before launch ↓ Level 3: Structured testing against OWASP categories runs with a documented scope and repeated trials ↓ Level 4: Objective-driven red teaming covers tools, retrieval, memory, and languages, with human and automated testing, fixes verified by retest ↓ Level 5: Red teaming runs on a cadence tied to change, with a regression suite in the release pipeline, and results feed governance and regulatory reporting
The step from Level 2 to Level 3 matters most, since it replaces an anecdote with a method that can be repeated and compared between releases.
AI Red Teaming Readiness Checklist
- Inventory every AI feature and agent in scope, including third-party and embedded ones.
- Write down the attacker goals that would hurt most. Data exposure, unauthorized actions, and policy bypass are different objectives with different tests.
- Map the architecture before testing. Tools, data sources, identities, and trust boundaries decide where the attacks go.
- Set rules of engagement in writing. Include environments, test accounts, data handling, and a safeguard for actions that could be real.
- Include human testers, and native speakers for the languages in use. Automation expands coverage and humans find the context-specific chains.
- Measure with repeated trials and rate severity by reachable impact.
- Fix at the right layer. Prefer deterministic controls, narrow permissions, and approval gates over trying to harden a prompt.
- Retest every fix and re-run the suite whenever the model, prompt, tools, or data change.
Common Mistakes and Edge Cases
Treating a benchmark score as proof of resilience. A fixed test measures performance on that test.
Testing the model in isolation. Most serious findings sit in how the model is connected to tools, data, and identities.
Counting on a single attempt. With variable output, one failed attack is weak evidence of safety.
Using the same credentials for test and production agents. A test that can reach production creates the harm it was meant to prevent.
Skipping non-English testing. Safeguards tuned mostly on English cannot be assumed to behave the same way elsewhere, and the reverse assumption is just as unsafe.
Publishing exploit detail in the report body. Reproduction detail belongs in controlled, access-limited annexes.
Treating the exercise as finished at the report. Without retest, there is no confirmation that anything was fixed.
When to Use What: A Few Decision Points
Automated tooling versus human-led red teaming. Automation gives breadth and regression coverage. Human testers produce the creative, application-specific attacks. Mature programmes use both.
Model-level versus system-level scope. An organization consuming a hosted model usually has little control over its internals, so system-level scope, covering prompts, retrieval, tools, and identities, delivers more value. A team training or fine-tuning its own model needs model-level testing as well.
Pre-production versus production testing. Staging is safer and can differ from production in data and permissions. Production testing is more accurate and needs stricter safe-execution rules.
A one-time engagement versus a recurring programme. A rarely changed internal tool may justify periodic testing. A product whose models, prompts, or tools change often needs testing tied to releases.
In-house versus external red team. Internal teams know the system best. An external team adds independence, outside experience, and credibility with auditors and regulators.
How SecNinjaz Fits Into This
Securing AI systems requires more than testing a model's ability to resist jailbreaks. Organizations must also evaluate how AI applications handle sensitive data, interact with external tools, enforce access controls, and behave when an attacker manipulates their inputs. SecNinjaz approaches AI security through a combination of adversarial testing, application security assessment, and governance-focused assurance.
AI Red Teaming and Security Testing
SecNinjaz's cybersecurity practice brings together AI and LLM security testing, AI red teaming, application and API penetration testing, infrastructure security assessments, and conventional red teaming. This enables organizations to evaluate AI systems within the context of their broader technology environments, including the APIs, identity mechanisms, retrieval systems, integrations, and infrastructure on which they depend.
Assessments can examine prompt injection, jailbreaks, sensitive information disclosure, insecure tool execution, excessive agent permissions, retrieval and memory weaknesses, and other risks relevant to the deployment. For AI agents, the focus extends beyond model responses to whether an attacker can induce unauthorized actions or cross established trust boundaries.
The objective is to identify exploitable weaknesses, establish their potential business impact, and provide actionable remediation guidance. Retesting then helps verify whether the implemented fixes address the demonstrated attack paths.
Connecting Technical Findings With AI Governance
Technical testing is most valuable when its results inform risk management and organizational decision-making. SecNinjaz's GRC, AI governance, audit, and regulatory compliance practices help organizations connect security findings with their broader risk and assurance activities.
Depending on the engagement scope, findings can inform AI risk assessments, control-gap analysis, remediation tracking, and evidence collection for relevant governance and compliance requirements. Organizations can use this information to understand residual risks, assign remediation responsibilities, and establish a repeatable process for evaluating AI systems as models, prompts, tools, and integrations change.
SecNinjaz also holds ISO/IEC 42001:2023 and ISO/IEC 27001:2022 certifications, subject to their applicable certification scopes. These standards address AI management systems and information security management systems, respectively, and provide relevant management-system frameworks for organizations seeking to strengthen their governance and security practices.
From Assessment to Continuous Improvement
AI security is not a one-time exercise. Changes to models, system instructions, connected tools, data sources, and agent permissions can introduce new attack paths or reintroduce previously resolved weaknesses.
SecNinjaz helps organizations approach AI security as an ongoing process: identify risks, test realistic attack scenarios, prioritize remediation, validate fixes, and incorporate lessons into subsequent security assessments. The resulting evidence supports more informed decisions about deployment readiness, risk acceptance, and the security controls required to operate AI systems responsibly.
Planning to deploy an AI assistant, LLM-powered application, or autonomous agent? An AI red-team assessment can help determine whether your security controls withstand realistic adversarial scenarios before those weaknesses are exploited in practice.
Contact SecNinjaz to discuss an AI security assessment tailored to your architecture, threat model, and business requirements.
Talk to Our Security Experts →
Frequently Asked Questions
What is AI red teaming?
AI red teaming is goal-driven, adaptive, adversarial testing of an AI system as it is deployed. Testers pursue objectives such as extracting confidential data or making an agent take an unauthorized action, using whatever techniques work, with permission and under controlled conditions, to find weaknesses before real attackers do.
How is AI red teaming different from AI safety benchmarking?
Benchmarking scores a model on a fixed set of prompts. Red teaming attacks the deployed system adaptively toward a specific goal. Microsoft's AI Red Team lists this distinction among its lessons from red teaming more than 100 generative AI products, and the Attacker Moves Second study shows why it matters: by adapting their attacks to each defense, researchers achieved attack success rates above 90% against most of the defenses tested, many of which had looked near-perfect against static tests.
Why do AI agents need separate red teaming?
Agents use tools, hold memory, and take actions, so a successful manipulation can have real effects in connected systems. Red teaming an agent examines tool permissions, instruction hierarchy, memory poisoning, delegation between agents, approval gates, and credentials. In a large public competition run with the UK AI Security Institute, all 22 agents tested were successfully attacked.
Can jailbreaks and prompt injection be fully prevented?
No complete prevention is known. Published defenses have been bypassed by adaptive attackers, so practical protection layers several controls and limits what a manipulated model can reach, using least-privilege tools, output validation, and approval for consequential actions. Red teaming measures how well that layered design holds.
Should AI red teaming include Indian languages?
Yes, where users speak them. Evidence is mixed: older studies found large gaps in low-resource languages, a 2026 study found simple translation no longer reliably bypasses current models but that native-speaker multi-turn testing raised success substantially, and a recent preprint benchmark found higher attack success in some Indian languages for the open models it tested. Testing with native speakers in the languages in use settles the question for a given system.
Is AI red teaming required in India?
There is no general statutory mandate. CERT-In's May 2026 guidance, CISG-2026-02, includes adversarial AI testing among its recommendations for security validation and resilience, and organizations should distinguish these recommendations from specific statutory or regulatory obligations applicable to their sector. Data protection duties under the DPDP Act, which take effect in May 2027, make evidence that AI systems do not leak personal data useful.
How often should AI systems be red teamed?
A full engagement is normally repeated at a fixed interval and after major change, and a regression suite built from earlier findings should re-run whenever the model, prompt, tool set, or retrieval data changes. Fast-moving products benefit from testing wired into the release process.
Where should an organization start with AI red teaming?
Start with an inventory of AI features and agents, write down the attacker goals that would cause the most harm, and map each system's tools, data sources, and identities. Agree rules of engagement in writing, test in an isolated environment with scoped credentials, measure with repeated trials, and retest every fix.
