On 4 August 2026, the OWASP GenAI Security Project published a new edition of its Top 10 for Large Language Model Applications, a widely used reference list for scoping AI security testing. The reordering is informative. Excessive Agency, the risk of a model holding more capability and permission than is safe, moved from sixth place to third. System prompt leakage gave way to a broader entry called Hidden Context Exposure. Improper Output Handling, which ranked fifth a year earlier, dropped to tenth. Prompt injection and sensitive information disclosure kept the first two positions. Taken together, the changes show the risk moving away from what a model says and toward what a connected application lets it do.
That last point is the one most AI security programmes miss. A model can pass every published safety benchmark and still sit inside an application that leaks customer records, because a connected tool holds broader permissions than it needs or because a retrieval index ignores who is asking. AI security testing exists to find those failures before launch, by treating the whole application, not just the model, as the thing under test.
Indian guidance now names the practice explicitly. CERT-In's May 2026 guidance on AI-assisted threats covers organizations' own AI deployments, including prompt injection, model theft, training-data poisoning, and the governance of autonomous agents, and its phased rollout ends with red teaming and adversarial AI testing. This guide covers what AI and LLM security testing is, which parts of an application it examines, how a structured engagement runs, what the findings usually look like and how they get fixed, where Indian and international requirements are heading, and how to build a programme that stays current as models and prompts change. It is written for CISOs, product security leads, and the engineering teams shipping AI features.
Talk to Our Security Experts →
What AI and LLM Security Testing Is
AI and LLM security testing is a structured, adversarial evaluation of an AI-enabled application. It examines how the application behaves under hostile or unexpected input and reviews the architecture around the model, with the goal of finding exploitable weaknesses before users or attackers do. The subject is the system as deployed: the model, the instructions that steer it, the data it retrieves, the tools it can call, the identities it runs under, and the downstream systems that consume its output.
It sits alongside several neighboring practices and is easily confused with them.
AI Security Testing and Neighboring Practices
| Practice | Primary question | How it differs |
|---|---|---|
| AI safety evaluation | Does the model produce harmful, biased, or policy-violating content? | Focuses on model behavior and content, not on the security of the surrounding application |
| Traditional application penetration testing | Can the application's code, APIs, and infrastructure be exploited? | Covers the conventional attack surface but not the model's susceptibility to manipulation |
| AI security testing | Can the AI application be manipulated into leaking data, taking unauthorized actions, or failing in ways that harm the business? | Combines adversarial testing of model behavior with review of tools, data, access control, and architecture |
| AI red teaming | Can a determined adversary achieve a specific goal against the AI system while evading detection? | Objective-driven and often broader in scope, usually run after baseline security testing is in place |
| AI governance / AI assurance | Are AI systems governed, controlled, monitored, and assessed against organizational, regulatory, and risk-management requirements? | Focuses on governance, accountability, risk management, policies, controls, documentation, and assurance evidence rather than directly attempting to exploit the AI system. |
One property of language models changes how testing has to be done. Outputs are not fully deterministic, so the same input can produce different results on different attempts. A single failed attack attempt is therefore weak evidence of safety. Credible testing repeats each scenario enough times to estimate how often it succeeds, and reports a rate instead of a yes or no.
The Attack Surface of an LLM Application
Most AI applications have more moving parts than the chat window suggests. Each layer carries its own failure modes, and a test plan that covers only the first one leaves most of the exposure unexamined.
| Layer | What can go wrong | Typical test focus |
|---|---|---|
| User input | Crafted input changes the model's behavior or overrides its instructions | Direct manipulation attempts and repeated trials to estimate success rates |
| Retrieved content | Documents, emails, web pages, or database records carry instructions the model follows | Indirect injection through every source the application ingests |
| Hidden context | System prompts, policy text, and tool definitions are exposed or relied on as a security boundary | Extraction attempts and review of what secrets or logic the context contains |
| Retrieval and embeddings | A vector store returns documents the requesting user should never see | Access control at retrieval time and isolation between tenants |
| Model and supply chain | A third-party model, plugin, dataset, or library is tampered with or unvetted | Provenance, version pinning, and review of serialized model files |
| Tools and agents | The model can take actions with broader permissions than the task requires | Authorization on every tool call and behavior when output is manipulated |
| AI gateway / model gateway | Weak authentication, policy bypass, unrestricted model access, inadequate rate limiting, missing logging, or inconsistent security controls across model providers | Authentication and authorization, model access controls, policy enforcement, rate limits, logging, DLP, routing controls, and abuse scenarios |
| MCP / agent communication layer | An agent or MCP server exposes excessive tools, accepts untrusted instructions, leaks credentials, or allows unintended actions across connected systems | MCP server and tool trust, capability boundaries, tool authorization, credential handling, input/output validation, cross-agent communication, and privilege escalation paths |
| Output consumers | Model output is rendered or executed by another system without validation | Treatment of output as untrusted input by browsers, interpreters, and databases |
| Identity and infrastructure | Weak authentication, exposed APIs, or open storage surround the AI components | Conventional API, cloud, and secrets testing applied to the AI stack |
| Cost and availability | Unbounded requests exhaust budgets or degrade service | Rate limits, quotas, and behavior under abusive load |
What Changed in the OWASP LLM Top 10 for 2026
The 2026 list, published in August, is the most current structure for scoping a test. The table below shows the new ranking, where each entry stood in the 2025 edition, and what a tester examines.
| 2026 entry | 2025 position | What testers examine |
|---|---|---|
| LLM01 Prompt Injection | 1 | Direct and indirect manipulation of model behavior across every input channel |
| LLM02 Sensitive Information Disclosure | 2 | Leakage of personal data, secrets, and proprietary content through outputs |
| LLM03 Excessive Agency | 6 | Capability, permission, and autonomy granted to the model and its tools |
| LLM04 Supply Chain | 3 | Third-party models, datasets, plugins, and dependencies |
| LLM05 Data and Model Poisoning | 4 | Integrity of training, fine-tuning, and retrieval data |
| LLM06 Unbounded Consumption | 10 | Resource exhaustion, runaway cost, and denial of service |
| LLM07 Misinformation | 9 | Confident but wrong output that downstream decisions rely on |
| LLM08 Hidden Context Exposure | 7, as System Prompt Leakage | Exposure of system prompts, retrieved policy text, and tool schemas |
| LLM09 Vector and Embedding Weaknesses | 8 | Retrieval-layer access control and embedding integrity |
| LLM10 Improper Output Handling | 5 | Downstream systems that trust model output without validation |
The Hidden Context Exposure entry carries a design lesson testers can apply immediately. OWASP's guidance treats everything placed in the model's context as potentially discoverable, so credentials and security-critical rules should never live there, and authorization should be enforced outside the model in deterministic, auditable components. The 2026 release also includes an appendix on application architecture and threat modeling, which gives a useful starting template for the scoping stage described below.
How an AI Security Test Runs
A well-run engagement moves through seven stages, and skipping the early ones tends to weaken everything after them.
| Stage | What happens | What it produces |
|---|---|---|
| 1. Scoping and architecture review | Map components, data flows, trust boundaries, tools, and identities, and build a threat model | A documented scope and a prioritized list of attack paths |
| 2. Rules of engagement | Agree authorization, test accounts, rate limits, data handling, and safeguards for tests that can trigger real actions | A signed scope and a safe-execution plan |
| 3. Baseline capture | Record intended behavior, policies, and normal outputs for each function | A reference against which deviations are judged |
| 4. Adversarial testing | Probe each OWASP category with repeated trials, including indirect paths through retrieved content | Success rates and evidence for each weakness found |
| 5. Architecture and control testing | Verify authorization on tool calls, API security, secrets handling, network egress, and logging | Findings on the conventional controls surrounding the model |
| 6. Risk rating and reporting | Rate severity by what an attacker can reach, reproduce findings, and map to OWASP, NIST, and MITRE ATLAS | An evidence-backed report with a remediation roadmap |
| 7. Retest and regression | Fix, re-run the same scenarios, and keep them as a regression suite | Confirmation that fixes work and keep working |
Two points deserve emphasis. The second stage matters more for AI than for most applications, because a manipulated agent can send an email, change a record, or call an internal service during a test. Testing in an isolated environment with scoped credentials is the safe default, and any testing of a third-party hosted model needs to respect the provider's terms. In India, the same legal logic that governs penetration testing applies: Sections 43 and 66 of the Information Technology Act, 2000 treat unauthorized access to a computer system as a civil and, where dishonest intent is present, criminal matter, so written authorization is the foundation of the engagement.
The sixth stage deserves a note on severity. A clever attack that reaches nothing sensitive is a lower-priority finding than an unremarkable one that reaches customer data or triggers a payment. Severity follows reachable impact, not novelty. The vocabulary in NIST's adversarial machine learning taxonomy, AI 100-2 E2025, which covers evasion, poisoning, privacy, and misuse attacks and extends to prompt injection, supply chain attacks, and agents, gives testers and engineering teams a shared language for describing what was found.
Common Findings and How They Get Fixed
The pattern across engagements is consistent. The model is rarely the only problem, and the strongest fixes sit in deterministic layers that do not depend on the model behaving well.
| Finding | Why it happens | Typical fix |
|---|---|---|
| The model follows instructions embedded in a retrieved document or message | Retrieved content is mixed with trusted instructions in the same context | Treat retrieved content as untrusted data, limit what the model can do after reading it, and require human approval for consequential actions |
| An agent can call tools with broad permissions | Credentials are shared or scoped to the whole system, not to the task | Least-privilege credentials per tool, allow lists, and authorization enforced in the backing service |
| Credentials or business rules sit in the system prompt | Developers treat the prompt as private | Move secrets out of the context entirely and enforce rules in code |
| Model output is rendered or executed downstream without checks | Output is trusted because it came from the application's own model | Validate, encode, and parameterize output exactly as for any untrusted input |
| Retrieval returns documents the user should not see | The vector store has no per-user access control | Filter at retrieval time using the requester's identity, and isolate tenants |
| Requests are unlimited | No quotas, budgets, or timeouts | Per-user rate limits, spend caps, and request size limits |
| A third-party model or plugin is unvetted | No inventory or provenance checks | Maintain an AI bill of materials, pin versions, and scan serialized model files |
| Personal data appears in prompts, logs, or fine-tuning sets | Data minimization was not applied to the AI pipeline | Redaction, retention limits, and access controls on logs and training data |
A caution belongs here. Prompt injection has no complete fix. NIST's taxonomy discusses mitigations for each attack class together with the limitations of those mitigations, and the practical position is layered defense that lowers the likelihood and, more importantly, the impact of a successful manipulation. That is exactly why the findings above focus on limiting what a manipulated model can reach rather than on trying to make the model unmanipulable.
Why This Is Becoming a Compliance Question
Indian expectations have moved from general principle to specific practice during 2026. CERT-In published guidance on 25 May under the reference CISG-2026-02, which Infosecurity Magazine's coverage summarizes as devoting particular attention to securing organizations' own AI deployments: prompt injection, model theft, training-data poisoning, and the governance of autonomous agents. It recommends AI bills of materials and a three-phase rollout that begins with governance, exposure reduction, and multi-factor authentication and ends with red teaming and adversarial AI testing. CERT-In describes the timelines as indicative, not binding, and the existing six-hour incident reporting requirement remains in force.
The wider policy direction is consistent. MeitY's AI Governance Guidelines, released on 5 November 2025, take a principles-based approach, designate CERT-In to monitor vulnerabilities in AI systems across critical sectors, and assign the AI Safety Institute technical validation and safety research. On the data side, Section 8(5) of the DPDP Act, which takes effect in May 2027, requires data fiduciaries to implement reasonable security safeguards, and Rule 6 of the DPDP Rules 2025, which commences on the same date, sets minimum measures that include access control and monitoring. An AI system that processes personal data through prompts, retrieval, logs, or training sets sits inside that duty, and a documented test showing that the system does not leak such data is concrete evidence of safeguards operating.
The international picture shifted this year as well. The EU's Digital Omnibus on AI, approved by the Council on 29 June 2026, moved the application date for stand-alone high-risk obligations from 2 August 2026 to 2 December 2027, and for AI embedded in regulated products to 2 August 2028. The amendments entered into force in July. The deferral reflects missing technical standards, not weaker expectations, and the requirements themselves, including the Article 15 expectation that high-risk systems demonstrate accuracy, robustness, and cybersecurity, remain part of the Act. Organizations building toward that standard can use testing evidence now, and the same evidence supports an ISO/IEC 42001 management system, where risk assessment and impact assessment records are core artifacts.
What Skipping Pre-Deployment Testing Costs
| Gap | Consequence |
|---|---|
| No adversarial testing before launch | Weaknesses surface in production, where IBM's 2026 data puts prompt injection incidents at an average of $5.89 million |
| No access controls on AI models and data | The condition present in 92% of organizations that reported an AI-related breach |
| Agents granted broad tool permissions | A single manipulated output can trigger a real action in a connected system |
| Secrets or authorization rules stored in prompts | Exposure of credentials and logic through the context, which OWASP advises assuming is discoverable |
| No retest after model, prompt, or corpus changes | Silent regressions, since behavior can shift with any of those inputs |
| Personal data flowing through prompts, logs, or retrieval without safeguards | Exposure under the DPDP Act, where failure to implement reasonable security safeguards carries a penalty of up to ₹250 crore once the provision takes effect |
A Maturity Model for AI Security Testing
Most organizations sit somewhere on a five-level path between untested launch and continuous assurance.
Level 1: AI features ship after functional testing only, and security is assumed to come from the model provider ↓ Level 2: A one-time manual check for prompt injection runs shortly before launch ↓ Level 3: Structured testing against the OWASP LLM categories runs with a documented scope and repeated trials ↓ Level 4: Testing covers architecture, tools, retrieval, and access control, every finding is tracked to a verified fix, and a regression suite re-runs on every change ↓ Level 5: Adversarial testing runs continuously in the delivery pipeline and is paired with production monitoring and mapped to governance and regulatory obligations
The move from Level 2 to Level 3 is the most valuable, since it replaces an anecdotal check with a repeatable method that can be compared across releases.
A Readiness Playbook for Testing Before Deployment
- Inventory every AI-enabled application, model, and data source in scope, including third-party services and features embedded in other products.
- Draw the architecture and trust boundaries before testing begins. Most serious findings sit at a boundary the diagram would have shown.
- Decide which actions the AI may take without human approval, and test those first. Autonomy is where impact concentrates.
- Move authorization, secrets, and policy enforcement out of the prompt and into deterministic components that can be audited.
- Build the test plan around the current OWASP LLM categories, with repeated trials to account for non-deterministic output.
- Test retrieval and tool integrations with realistic, least-privilege accounts, never with administrator credentials that mask permission problems.
- Rate severity by reachable impact and retest every fix, keeping the scenarios as a regression suite.
- Re-run the suite on every change to the model, prompt, tool set, or retrieval corpus, and schedule a full engagement at a fixed interval.
Common Mistakes and Edge Cases
Treating the model provider's safety claims as proof the application is secure. Provider evaluations cover the model. They say little about how an organization wired it to tools, data, and users.
Testing only the chat interface. Indirect paths through documents, emails, and web content are often easier to exploit and harder to notice.
Putting secrets or access rules in the system prompt. Anything in the model's context should be assumed retrievable by someone who asks the right way.
Declaring a pass after a single attempt. With non-deterministic output, one failed attack proves little, and the result needs a success rate across repeated trials.
Relying on a fixed list of known attack strings. Such lists age quickly, and defenses tuned to them can pass the list while remaining open to variations.
Using the same credentials for test and production agents. A test that can reach production data or actions creates the harm it was meant to prevent.
Skipping the retest after a model upgrade. A new model version can reopen findings that were closed under the old one.
When to Use What: A Few Decision Points
Automated scanning versus manual expert testing. Automation gives broad, repeatable coverage and suits regression runs. Manual testing finds the chained, application-specific paths that scripts miss. Mature programmes combine both.
Pre-production versus production testing. A staging environment is safer but can differ from production in data, permissions, and traffic. Production testing gives the most accurate result and needs the strictest safe-execution rules.
A hosted model API versus a self-hosted or open-weight model. Hosted models shift some risk to the provider and add provider terms that constrain testing. Self-hosted models add supply chain and file-integrity concerns that need their own review.
A one-time pre-launch test versus continuous testing. A rarely changed internal tool may justify periodic testing. An application whose prompts, tools, or models change frequently needs testing wired into the release process.
In-house testing versus external specialists. Internal teams know the application best, while an independent team brings adversarial experience and credibility with auditors and regulators.
How SecNinjaz Fits Into This
AI and LLM Security Testing and AI Red Teaming sit within SecNinjaz's Cybersecurity practice alongside Vulnerability Assessment, Penetration Testing, Red Teaming, Purple Teaming, Breach and Attack Simulation, and AI SOC Automation. That arrangement lets an AI engagement draw on conventional API, cloud, and identity testing for the surrounding stack, and move to red teaming or a purple team exercise when the first round of fixes is in place.
The governance side connects to SecNinjaz's GRC and DPDP practice, including AI Governance, Audit and Gap Assessment, and Regulatory Compliance, which map test results to DPDP safeguard duties and to AI management system requirements. SecNinjaz holds ISO/IEC 42001:2023 for AI management systems alongside ISO/IEC 27001:2022 for information security, the combination most relevant to handling both the AI governance record and the sensitive findings an AI security test produces. For organizations that want to find AI security weaknesses before users do, that pairing of adversarial testing and documented governance is the one worth looking for, whether the organization works with SecNinjaz or anyone else.
Talk to Our Security Experts →
Frequently Asked Questions
What is AI security testing?
AI security testing is a structured, adversarial evaluation of an AI-enabled application that examines how it behaves under hostile input and reviews the architecture around the model. It covers the model, instructions, retrieved data, tools, identities, and downstream systems, with the aim of finding exploitable weaknesses before deployment.
How does LLM security testing differ from a traditional penetration test?
A traditional penetration test examines code, APIs, and infrastructure. LLM security testing covers that conventional surface and adds the model's susceptibility to manipulation, the data it retrieves, and the actions it can take through tools. Because model output is not fully deterministic, it also repeats scenarios to estimate success rates instead of relying on a single attempt.
What is prompt injection, and can it be fully prevented?
Prompt injection occurs when crafted input, whether typed by a user or hidden in retrieved content, changes a model's behavior against the application's intent. No complete prevention is known, and NIST's adversarial machine learning taxonomy discusses the limitations of available mitigations. Practical defense limits what a manipulated model can reach by enforcing least-privilege tool access, validating output, and requiring approval for consequential actions.
What changed in the OWASP Top 10 for LLM Applications in 2026?
The 2026 edition, published on 4 August 2026, keeps Prompt Injection and Sensitive Information Disclosure at the top, moves Excessive Agency up to third place, replaces System Prompt Leakage with the broader Hidden Context Exposure, and moves Improper Output Handling down to tenth. The shifts reflect growing risk from agents and tool access.
Does AI security testing need to examine the model itself?
Testing examines the model's behavior under adversarial input, but most organizations do not control the model's internals, so the emphasis falls on the application built around it. IBM's 2026 findings point the same way, with many AI-related breaches tracing to access control gaps, exposed APIs, and cloud misconfiguration rather than model flaws. Supply chain review covers the models and datasets an application depends on.
How often should AI security testing be repeated?
A full engagement is normally repeated at a fixed interval and after major architectural change. A regression suite built from earlier findings should re-run whenever the model version, system prompt, tool set, or retrieval corpus changes, since any of those can alter behavior. Fast-moving applications benefit from testing built into the release pipeline.
What does CERT-In's 2026 guidance say about AI testing?
CERT-In's guidance CISG-2026-02, published on 25 May 2026, covers securing organizations' own AI deployments, including prompt injection, model theft, training-data poisoning, and autonomous agent governance. Its three-phase rollout ends with red teaming and adversarial AI testing. CERT-In describes the timelines as indicative rather than binding, and the six-hour incident reporting rule remains in force.
Where should an organization start with AI security testing?
Start with an inventory of AI-enabled applications and a diagram of each one's architecture, data sources, and tool permissions. Decide which actions the AI may take without human approval and test those first, using least-privilege test accounts in an isolated environment. Findings are then rated by reachable impact, fixed, and retested.
