The organizations in Canada and the United States now shipping AI features rarely built the model. They wired a commercial or open model into their own application: a chat interface, a retrieval layer that feeds it internal documents, a set of tools or functions it can call, an agent that chains those calls, and the authentication and data flows around all of it. That assembled system, not the model in isolation, is what an attacker reaches, and it is where the exploitable weaknesses live. An AI penetration test assesses that whole deployed system for weaknesses a real attacker could use, with evidence.
This guide explains what an AI penetration test actually covers, how it differs from AI red teaming and model evaluation, what it can and cannot prove, what you receive, and how to prepare. It is not a catalogue of theoretical AI risks. It reflects how these systems are tested in practice, across LLM-integrated applications, retrieval-augmented systems, and tool-using agents in Canada and the United States. If you want the working tool our consultants use to run these engagements, the free AI Penetration Testing Playbook is available below.
What AI Penetration Testing Is
AI penetration testing is a structured, authorized attempt to compromise an AI-enabled system using the same techniques a real attacker would use, performed with written permission and defined rules of engagement. It treats the system as a whole: the application and its model interface, the retrieval layer, the tools and agents the model can invoke, the data flows, the authorization, and the surrounding infrastructure. The goal is to find weaknesses an attacker could exploit, chain them into a validated path to impact such as data exposure or an unintended action, and hand back evidence-based remediation.
It is not model evaluation, and it is not a generic prompt-injection demo. Model evaluation asks whether a model behaves safely in the abstract; an AI penetration test asks whether your specific deployment can be made to do something harmful, and proves it against your application, your data, and your integrations. That distinction, testing the deployed system rather than the model, is what makes the findings actionable for the team that owns the product.
AI Penetration Testing vs AI Red Teaming, Agentic AI Red Teaming, Model Evaluation, and Governance
These terms are used loosely, and choosing the wrong one wastes budget or gives false assurance. Each answers a different question about a different thing.
| Activity | What it assesses | Output |
|---|---|---|
| AI penetration testing | The deployed AI-enabled system for exploitable weaknesses | Proven exploit chains, evidence, remediation |
| AI red teaming | Model and system behaviour under adversarial pressure, often ongoing | Safety and robustness findings over time |
| Agentic AI red teaming | Autonomous, tool-using agents specifically | Agent-layer attack findings |
| Model evaluation | A model's behaviour and safety in the abstract | Benchmarks and scores |
| AI governance / risk assessment | Policies, controls, and process | Control gaps and recommendations |
- AI penetration testing is a scoped, point-in-time engagement that proves what an attacker can do to your deployed system, with reproducible evidence. It is the subject of this guide.
- AI red teaming is broader and often continuous, probing model and system behaviour under adversarial pressure for safety and robustness. Our agentic AI red teaming guide covers the autonomous-agent case in depth, and a penetration test links into that work rather than replacing it.
- Model evaluation benchmarks a model in isolation; it does not test your application, data, or integrations, so it cannot tell you whether your deployment is exploitable.
- AI governance and risk assessment examine policy and control maturity. A penetration test provides the technical evidence those programs need, but it is not a substitute for them.
Free download: the AI Penetration Testing Playbook
The article explains the method. The Playbook is the working toolkit our consultants run an engagement with: an architecture and data-flow scoping worksheet, a model-interface and prompt-injection test checklist, a retrieval and tool-use test matrix, an authorization and data-exposure matrix, an evidence template, and a remediation prioritization matrix. Confirm your email and we send a secure download link.
Privacy notice: Double opt-in - we email a confirmation link and the playbook downloads only after you confirm. We use your name and email to deliver the playbook and, only if you opt in above, to send marketing communications you can unsubscribe from at any time. We never sell your information. To unsubscribe or request deletion, email info@cybersecpentesting.com. See our Privacy Policy.
The AI-Enabled System Is the Attack Surface
Testing only the model misses where AI systems actually break. The model is one component in a chain, and each link in that chain is a distinct surface with distinct tests. Mapping the whole system, and the data flows through it, is the first move in any serious AI penetration test.
| Surface | What we test for | Concrete failure |
|---|---|---|
| Application and model interface | Direct prompt injection, system-prompt disclosure, jailbreak | User input overrides the developer's instructions |
| Retrieval layer (RAG) | Indirect prompt injection, poisoned or over-broad retrieval | Attacker-controlled content is trusted as instruction or fact |
| Tools, functions, and agents | Insecure tool use, excessive agency, unsafe chaining | A text-level attack triggers a real action or request |
| Data flows and authorization | Sensitive-data exposure, tenant isolation, over-broad access | The model reaches data the caller was not entitled to |
| Output handling and infrastructure | Unsafe rendering of output, the APIs and services around the model | Model output injects into a downstream system |
The Application and Model Interface
The interface is where direct prompt injection lives: user input that overrides the developer's system prompt, extracts it, or bypasses a guardrail. We test whether the boundary between instructions and user data holds under adversarial input, and whether the system prompt or restricted behaviour can be coaxed out. Single-LLM application injection is covered in depth in our LLM prompt injection analysis.
The Retrieval Layer, RAG, and Vector Stores
When the system retrieves documents to answer a question, whatever it retrieves is treated with trust, and that is the opening for indirect prompt injection: instructions planted in a document, a web page, a support ticket, or any source the retriever might pull. We test whether attacker-influenced content in the retrieval corpus can steer the model's behaviour, and whether the vector store leaks data across users or tenants through over-broad retrieval. This is where an external attacker with no direct access to the model can still reach it.
Tools, Agents, and Excessive Agency
The moment a model can call a tool, send an email, query a database, hit an API, a text-level weakness becomes an action-level one. Excessive agency is the recurring problem: the model is granted more capability, or broader permissions on a tool, than the task requires, so a successful injection does not just produce bad text, it performs an operation. We test what each tool can do, under whose authority, and whether a manipulated model can be made to misuse it. The agent and tool layer is covered in our AI agent prompt injection and MCP tool poisoning deep-dives.
Data Flows, Authorization, and Infrastructure
An AI feature usually sits on top of the same data and identity model as the rest of the application, and it often gets a broader view than any single user should have. We test whether the model can be used to reach data the caller is not authorized to see, whether tenant isolation holds in retrieval and tool calls, and whether the surrounding APIs and infrastructure introduce their own weaknesses. Many of the most serious AI findings are, at heart, authorization failures expressed through a new interface.
What a Professional AI Penetration Test Covers
A credible engagement is a structured examination of how the assembled system can be manipulated, driven by the architecture and the data flows, not a scripted set of jailbreak prompts. It runs as a repeatable sequence, and each phase feeds the next.
Scope and Rules of Engagement
The AI-enabled system in scope, its components, the environments, the user roles, and any restrictions are named explicitly, along with the rules of engagement. Because testing consumes model calls, the scope also sets expectations on rate limits and cost, and identifies anything with persistent state, such as a memory or vector store, that must not be polluted in production. Good scoping aligns the engagement to your obligations, whether that is the NIST AI Risk Management Framework, SOC 2, or PIPEDA.
Methodology
A professional test follows a published methodology rather than a tester's memory. The OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS provide the categories and adversary techniques to cover, and the NIST AI Risk Management Framework provides the governance language to report against. The categories that matter most when the target is a deployed system are prompt injection (LLM01), sensitive information disclosure (LLM02), improper output handling (LLM05), excessive agency (LLM06), system prompt leakage (LLM07), and vector and embedding weaknesses (LLM08). These are used to structure concrete test actions against your system, not as a checklist to tick, which is the difference between a penetration test and a risk questionnaire.
Manual Testing vs Automated Tooling
Tooling such as PyRIT and Garak can generate and replay large batches of adversarial prompts and is valuable for coverage and regression. But the findings that matter, an indirect injection that reaches a tool, an authorization gap exposed through retrieval, an exploit chain across components, are built by a human who understands the specific architecture and adapts to it. An engagement that is only an automated prompt battery is a fuzzing run with a penetration-test invoice.
| Automated prompt tooling | Manual penetration test |
|---|---|
| Replays known adversarial prompts at scale | Builds exploit chains across the whole system |
| No awareness of your architecture or data | Tests retrieval, tools, and authorization in context |
| Reports which prompts elicited a response | Proves a real action or data exposure with evidence |
| Cannot follow an injection into a tool call | Follows the chain from injection to impact |
| Severity by prompt category | Severity by business impact and reach |
Evidence and Exploit-Chain Validation
A single manipulated response is not a finding; a proven consequence is. Every weakness is developed into a reproducible exploit chain, from the input the attacker controls to the action or data it reaches, and captured with evidence. Findings are prioritized by validated impact, not by whether a guardrail could be tripped. The engagement also records what could not be confirmed, so a limitation reduces confidence in coverage rather than being read as an assurance of safety.
Common AI Attack Paths We Prove
Across engagements, a handful of chains recur, and naming them helps product teams recognize their own exposure. These are the paths we most frequently develop from attacker input to demonstrated impact:
- Indirect injection to tool action: attacker-controlled content in the retrieval corpus instructs the model, which then calls a tool and performs an unintended operation.
- Excessive agency to real-world effect: an over-permissioned tool lets a manipulated model send a message, change a record, or reach an internal service.
- Retrieval to cross-tenant data exposure: over-broad retrieval or weak isolation returns another user's or tenant's data through the model.
- Output handling to downstream injection: unsafe rendering or use of model output injects into a browser, a database, or another system.
What We Actually Test in an AI System
Scope is set by the architecture, but a full AI penetration test typically covers the areas below, each tied to a concrete surface, with depth prioritized by where your system carries risk.
What an AI Penetration Test Can and Cannot Prove
Honesty about scope is part of the assurance. An AI penetration test is powerful for what it is, and it is important not to claim what it is not.
It can prove that specific, exploitable weaknesses exist in your deployed system: that an indirect injection reaches a tool, that retrieval exposes data across tenants, that a guardrail can be bypassed to a real consequence, that authorization around the model fails. Each of those is demonstrated with a reproducible exploit chain and evidence, and each comes with prioritized remediation.
It cannot prove that your model or system is safe in every possible context. It is a point-in-time test of a specific deployment, against a scoped set of objectives; it does not certify the model, guarantee that no adversarial input will ever succeed, or replace ongoing AI red teaming and governance. Model behaviour is probabilistic, and new attack techniques emerge continually. A penetration test tells you what an attacker can do to your system today and how to fix it, and it should be paired with continuous testing and a governance program, not treated as a one-time certificate of safety.
How Findings Become Business Risk
A finding is only useful when it is expressed as risk a leader can act on. The severity of an AI finding is decided by what the exploit reaches, the data exposed, the action performed, the tenant boundary crossed, not by how clever the prompt was. A jailbreak that produces edgy text is minor; the same injection that causes the system to email a customer list or delete a record is critical, and a good report draws that line clearly.
A professional report separates technical impact from business translation and maps findings to the frameworks your organization answers to. The NIST AI Risk Management Framework and MITRE ATLAS provide the AI-specific language; SOC 2 and general security frameworks still apply because most AI findings are, underneath, authorization or data-handling failures; and where personal information is reached, unauthorized access or disclosure may require a privacy-breach assessment and may trigger notification or reporting obligations under PIPEDA depending on the circumstances, addressed in our PIPEDA penetration testing guide. Building a governed platform before deploying AI is the subject of our enterprise knowledge platform guide.
What to Expect From an AI Penetration Testing Engagement
Buyers ask the same practical questions before commissioning a test. Here is what a CSPI AI engagement involves, so there are no surprises when you scope one.
| Who performs the testing | A named principal consultant (CREST CRPT, OSCP, OSEP), not an outsourced or rotating team. The person who tests writes the report. |
| Access required | The AI application or API, an architecture and data-flow description, test accounts for each role, a non-production or safe environment where possible, and any tool or integration credentials in scope. |
| Production impact | Testing is deliberately non-destructive. Anything with persistent state, such as memory or a vector store, is protected from pollution, model-call cost and rate limits are respected, and write or tool actions are agreed in advance. |
| Timeline | A focused single-system engagement typically takes one to two weeks of testing plus reporting; a complex agentic system with many tools and integrations extends it. The scoping call sets the schedule. |
| How findings are validated | Every finding is developed into a reproducible exploit chain with evidence, from attacker input to demonstrated impact. Automated results are triaged and confirmed before they appear in the report. |
| What you receive | A technical report with reproducible exploit chains, an executive summary, business-risk translation, prioritized remediation, framework mapping (OWASP LLM, MITRE ATLAS, NIST AI RMF), and a retest of fixed findings. |
| Included vs excluded | Scope is fixed in writing: the named AI system, components, and roles in scope; model-provider infrastructure, denial-of-service, and third-party systems out of scope unless explicitly agreed. |
How to Prepare for an AI Penetration Test
A few steps make the engagement faster and more valuable. Provide an architecture and data-flow description of the AI system, a list of the tools and integrations the model can use, and test accounts for each role, ideally two per role so cross-user and cross-tenant access can be tested. Point the test at a representative non-production environment where possible, and identify any persistent state that must be protected. Confirm expectations on model-call cost and rate limits, and confirm authorization and rules of engagement. Agree in advance how findings will be prioritized and retested. For where an AI test fits alongside application testing, see web application penetration testing and what penetration testing is; the agent-specific angle is covered in our agentic AI red teaming guide.
What You'll Receive in the AI Penetration Testing Playbook
The article explains the method. The Playbook is the actionable toolkit that lets you scope or govern an engagement, and it complements the guide rather than repeating it. It is a working document, not a marketing brochure, and it includes:
- An architecture and data-flow scoping worksheet to map the model interface, retrieval, tools, agents, and authorization before testing begins.
- A rules-of-engagement checklist covering authorization, persistent-state protection, cost and rate limits, and communication.
- A model-interface and prompt-injection test checklist for direct and indirect injection, system-prompt disclosure, and guardrail bypass.
- A retrieval and tool-use test matrix for RAG poisoning, over-broad retrieval, excessive agency, and unsafe tool actions.
- An authorization and data-exposure matrix for tenant isolation and access, plus an evidence template for reproducible exploit chains.
- A remediation prioritization matrix and an executive reporting checklist mapped to OWASP LLM, MITRE ATLAS, and NIST AI RMF, plus a fully sanitized worked example (indirect injection to tool action).
Frequently Asked Questions
What is AI penetration testing?
AI penetration testing is an authorized, human-led assessment of an AI-enabled system, the application, model interface, retrieval layer, tools, agents, data flows, authorization, and surrounding infrastructure. Rather than benchmarking a model, a tester finds weaknesses an attacker could exploit, chains them into a validated path such as data exposure or an unintended action, and documents evidence and prioritized remediation.
What is the difference between AI penetration testing and AI red teaming?
AI penetration testing is a scoped, point-in-time engagement that proves exploitable weaknesses in your deployed system with evidence. AI red teaming is broader and often continuous, probing model and system behaviour under adversarial pressure for safety and robustness. A penetration test provides the concrete exploit evidence; red teaming provides ongoing adversarial assurance. They complement each other.
Does AI penetration testing test the model or the whole system?
The whole system. The model is one component; the exploitable risk usually lives in how it is connected, the retrieval it trusts, the tools it can call, the data it can reach, and the authorization around it. Testing only the model in isolation misses the connections where AI systems actually break.
What is prompt injection, and what is indirect prompt injection?
Prompt injection is input that overrides the developer's instructions to the model, extracting the system prompt or bypassing a guardrail. Indirect prompt injection plants those instructions in content the system later retrieves, a document, a web page, a ticket, so an attacker with no direct access can still steer the model. Indirect injection is often the more serious, because it reaches retrieval and tools.
Does an AI penetration test prove our model is safe?
No. It proves whether specific, exploitable weaknesses exist in your deployed system, with evidence, and how to fix them. It is a point-in-time test of a specific deployment; it does not certify the model, guarantee that no adversarial input will ever succeed, or replace ongoing AI red teaming and governance. It tells you what an attacker can do today and should be paired with continuous testing.
How much does AI penetration testing cost?
Cost depends on scope: the components in the system, whether it includes retrieval and tool-using agents, the number of roles and tenants, and the size of the integration surface. Most enterprise AI penetration tests in Canada and the US are scoped as a fixed-price engagement after a scoping call, with pricing set before work begins.
Will an AI penetration test affect our production environment?
Testing is deliberately non-destructive and is preferably run against a non-production environment. Anything with persistent state, such as memory or a vector store, is protected from pollution, model-call cost and rate limits are respected, and any write or tool action against production is agreed in advance and scheduled, so the system is protected while the test still reflects real risk.
Does AI penetration testing support NIST AI RMF, SOC 2, or PIPEDA compliance?
Yes. Human-led AI penetration testing produces the exploitation evidence that maps to the NIST AI Risk Management Framework and MITRE ATLAS, and it supports SOC 2 and PIPEDA because most AI findings are authorization or data-handling failures. Where the system reaches personal information, the results also inform privacy-breach assessment obligations depending on the circumstances.
AI security is won or lost in how the model is connected to your data, tools, and users, not in the model alone. As an AI penetration testing company serving enterprises in Canada and the United States, we prove, with evidence, what an attacker could actually do to your AI-enabled system, and we are clear about what the test does and does not show. Start with the playbook above, then talk to us about a scoped AI penetration testing engagement.