Our methodology for securing an AI system

Why AI changes the game

A traditional application keeps code (the instructions) clearly separate from data. A large language model (LLM) receives everything as text: the developer’s instructions, the user’s question, and the content of an email, a document or a web page. It has no reliable way to tell a legitimate instruction from one an attacker slipped into that data.

As long as the model only answers questions, the risk stays limited. It becomes serious when the AI can reach internal data, reads content from outside and can act: send an email, call an API, update a record. That is exactly what Microsoft 365 assistants, agents and in-house applications connected to your systems do.

Our principles

• Test without causing harm. Active testing is preferably done in a staging environment, with test accounts and test data. In production, scope and time windows are agreed in writing.
• Minimal data. We use fictitious markers (“canaries”) rather than real sensitive data to demonstrate a leak.
• Reproducible evidence. Every finding comes with the request, the booby-trapped content, the response and the observed success rate.
• Recognized frameworks. We rely on the OWASP Top 10 for LLM Applications 2026, the OWASP Top 10 for Agentic Applications, MITRE ATLAS and the NIST AI Risk Management Framework.

1. Inventory and scoping

You cannot protect what you do not know about. We start by listing how AI is used, including uses IT never approved (“shadow AI”):

• SaaS assistants: Microsoft 365 Copilot, ChatGPT, Gemini, tools built into your business software.
• In-house applications: chatbots, retrieval-augmented generation (RAG) over your documents, calls to model APIs.
• Agents: AI that chains actions and uses tools (email, CRM, code, browser).

For each system we document a simple record: who uses it, what data it sees, what external content it reads and what it can do.

system: Customer service assistant (website)
model: cloud provider model API (hosted outside Québec)
users: public, unauthenticated
data_access:
  - public knowledge base
  - order history (through a tool, after customer verification)
external_content_read: visitor messages, cited web pages
possible_actions:
  - look up an order
  - open a support ticket
personal_information: yes  # PIA required (Law 25)
owner: Customer service director

2. Threat modelling

We follow the path of the text: where does it come from, where does it go and what can it trigger? Three questions guide the analysis:

• Does the system read untrusted content? Incoming email, shared documents, web pages, tickets, tool results.
• Does it have access to sensitive data? Personal information, internal documents, secrets, conversation history.
• Can it communicate outward or act? Send a message, call a URL, render a remote image, write to a system.

When all three conditions are met, a single indirect prompt injection can be enough to get data out, without the user clicking anything. That is the scenario we prioritize.

3. Assessing the controls

• Identity and permissions: does the agent use a dedicated least-privilege account, or an administrator token?
• Data access: does retrieval (RAG) respect the permissions of the user asking the question?
• High-impact actions: is human approval required before sending, paying or deleting?
• Output handling: is the model’s response treated as untrusted before it is rendered (images, links, HTML) or executed?
• Guardrails: input and output filters, injection detection, length and cost limits.
• Logging: are prompts, tool calls and decisions kept so an incident can be investigated?
• Supply chain: origin of models, plug-ins and MCP servers, and the provider’s terms on how your data is used.

4. Adversarial testing

A configuration review is not enough: we then try to hijack the system, in a controlled way. Direct and indirect prompt injection, guardrail bypass (jailbreaks), hidden context exposure, data exfiltration and tool abuse. Because a model does not always answer the same way, every test case is replayed several times.

The fourth article in this series covers how an AI penetration test runs.

5. Prioritization and deliverables

• Critical: data exfiltration or unauthorized action possible without any victim interaction. Example: a booby-trapped email that gets the assistant to send internal documents out.
• High: reliable guardrail bypass, or access to data beyond the user’s rights.
• Medium: hidden context exposure (system prompt, configuration) or misleading answers with limited consequences.
• Low: good practice or hygiene, such as missing consumption limits.

You receive an executive summary, a technical report with evidence, a prioritized action plan and a debrief session with your teams.