This is the first part of a multi-part (we’ll know when we’re done) series on creating Security Invariants for AI. Comments are available on the Google Doc version.
Security Invariants are an essential preventative control, only growing in importance as we adopt AI. A Security Invariant is
a system property that relates to the system’s ability to prevent security issues from happening. Security Invariants are statements that will always hold true for your business and applications.
A key feature of the invariant is that it always holds true. It cannot be bypassed or ignored. There is no “The Founder/CTO doesn’t want this to apply to them”. We’ve discussed Cloud Security Invariants in AWS before. You don’t write exceptions to invariants, you design invariants so anything that would need to be an “exception” works within the boundaries of the invariant.
In this paper/series of blog posts, we will address the implementation of security invariants in the context of Generative and Agentic AI.
Traditional governance assumes deterministic execution and stable control points. AI introduces three challenges:
While many tokens have been spent talking about the risks of models and their non-deterministic nature, that doesn’t mean we can’t still use invariants. People are non-deterministic tool-users and we manage to build invariants for them.
While the model is a dice-rolling word machine, the rest of the AI Stack consists of things we know and have been dealing with for decades: software, compute, networking, and authn/authz principals.
Before we get into the invariants, it helps to understand some AI fundamentals and make sure we are using the same terminology.
Artificial Intelligence (AI) is software that makes decisions or predictions by finding patterns in data, rather than following rules explicitly written by a programmer. Generative AI is the subset that produces new content instead of just classifying or predicting. It learned the shape of its training data well enough to make more of it (text, code, images, you name it). Think of it as a very sophisticated autocomplete that got way too big for its britches.
To figure out where invariants can live, we first need to understand what the stack actually is. So let’s take it apart.
The model is the core artifact of the whole GenAI stack. It is a neural network, trained on a large data set (like “the Internet”). The model is static once trained, and by the time you’re calling the API, that knowledge and how it will respond is frozen. You’re not talking to something that’s learning or reasoning in real time. You’re running inference against a static artifact that reflects every architectural and data decision the lab made during training.
The filtering decisions during training (what gets included, what gets screened out, and what never made it into the pipeline) are baked permanently into the model weights. Fine-tuning and reinforcement learning are used to shape the model’s personality and behavior. Those alignment choices are also frozen by the time you’re calling the API.
Inference is where the model executes. It takes your prompt and chops it into tokens. “Tokenization” sounds fancy, but it’s basically the model’s way of converting human text into numbers it can do math on. Once converted into numbers a lot of fancy vector math stuff happens to find relationships between the tokens.
Then comes the dice roll. The model doesn’t “know” the answer. It produces a probability distribution over its entire vocabulary for what token should come next. Temperature controls how spicy that distribution is: low temp, it almost always picks the most likely next token; high temp, it’ll take some weird swings. It samples from that distribution, appends the token, and repeats the whole process, one token at a time, until it hits a stop condition.
For our purposes, what matters is that the process is non-deterministic, and a non-deterministic process cannot be a control point since the behavior won’t be consistent.
Einstein said God doesn’t play dice with the universe, but your friend Claude is very much playing dice with the answer to your question.
The harness is everything around the model that isn’t the model itself. From a security standpoint, the harness just might be more important than the model. The harness is the software infrastructure that makes the model actually useful: it accepts the incoming request, formats the prompt, manages the system prompt and conversation history, calls the inference engine, handles tool use and MCP calls, applies any input/output classifiers, and returns the response to the caller. The Harness is not an us vs them thing. Major model providers include harnesses (e.g. Claude Code) but organizations can build their own. Including harnesses around other harnesses.
When calling a first-party lab’s model the harness is split. The provider’s side is doing a lot of heavy lifting: enforcing operator system prompts, running safety classifiers, managing context windows, routing to the right model version, handling retries, logging, and so on. How you invoke the inference, what system prompts you provide, how your code invokes tools is also part of the harness. (System prompts are prioritized instructions added as a preface to your actual prompt and are used by every AI provider to constrain activity and define responses. For example, “don’t tell people how to make bombs”. You can add your own system prompt, which runs after the provider’s system prompt).
Most importantly, the harness is where most of the preventative controls actually live. The model weights just generate tokens; the harness decides what’s allowed in and what’s allowed out.
One key point: the model using a system prompt is not an invariant, but the presence of the system prompt can be. “You must never delete production data” is a polite request to the dice-rolling machine, and the machine will probably honor it. Probably. You cannot use a system prompt to make the model’s compliance invariant.
What you can make invariant is the delivery: the harness can guarantee that every single inference call carries that instruction, enforced in software you control and verified in your pipeline. That’s a real control and a legitimate layer of defense-in-depth. An IAM policy with no Delete* actions is a constraint the model can’t violate, no matter what it tries. A guaranteed-present system prompt improves the odds the model will behave as expected. If your entire AI governance strategy is a system prompt, you don’t have governance, just vibes, or maybe, polite suggestions..
The model is like an engine, the harness is like a car. You need both of them to get somewhere, and you can define your own rules in the harness you control.
Where your model runs will further define potential control points. There are three ways to consume a frontier model:
Each step down that list trades convenience for control. There is no right answer, but there is a wrong move: picking a deployment model without asking which invariants you need to enforce and whether the enforcement points exist in that environment.
Agentic AI is what you get when you give a generative AI model tools, memory, and a feedback loop. Instead of answering a question and stopping (i.e., a chatbot), an agentic system can take actions by calling APIs, running code, reading files, browsing the web, and then feeding the results back into the next inference call to keep working toward a goal. It’s still the same dice-rolling word machine underneath, but now it’s got hands and a purpose. What is lacking is a conscience, a sense of consequences, and any semblance of common sense. It was trained on the internet after all.
Agents are built in code to create that feedback loop, memory, etc. You can write your own or run someone else’s.
Those hands come in a few distinct forms. The first one, Function/Tool Calling, is special; it’s implemented inside the harness itself, by the developer who built the agent. Every other hand on this list is external: the harness calls out to it, but the implementation lives somewhere else.
Function/Tool Calling: The foundational primitive. The developer defines a set of tools directly in the code and via API call usually via a JSON schema. The model emits a structured invocation, the harness executes it, and the result feeds back into the next inference. The action surface is whatever the developer explicitly wired up, nothing more. This is the only hand that lives inside the harness.
MCP (Model Context Protocol): A standardized protocol for tool discovery and invocation over a network boundary. Instead of tools defined inline in the harness, the model queries an external MCP server for available tools, then calls them by name. The tool manifest defines what actions the model can take. Connectors for email, calendar, Slack, GitHub, and similar services are, in practice, MCP servers. The network boundary is what makes HTTP MCP meaningfully different from the alternatives: it’s an interception point where governance controls can actually be applied. (MCP also supports a stdio transport, where the “server” is a local process communicating over stdin/stdout. stdio MCP has no network boundary and no interception point. It’s CLI with a structured handshake.)
Code/Command Execution: The model writes code or shell commands, and the harness runs them. Unlike Function/Tool Calling, the model is generating the tool at runtime rather than invoking a predefined one. This is the widest blast radius on the list: the action surface isn’t defined by a schema, it’s defined by whatever the execution environment can do and whatever credentials it can find, which is exactly why the invariant for this hand doesn’t live anywhere near the model. It lives in the sandbox: containerized execution, egress filtering, the harness (some restrict tool calls) and an environment scrubbed of credentials. If the code runs on the developer’s laptop with their SSO session, you’ve built a very enthusiastic insider threat.
Browser/Computer Use: The model drives a headless browser or desktop UI by issuing actions against screenshots. Instead of calling a defined API, it’s operating on visual state: click here, type there, read what comes back. The action surface is effectively unbounded: anything a human user could do in a browser or on a desktop, the agent can do too.
Memory Read/Write: Persistent state access: vector stores, key-value stores, knowledge graphs, or even simple markdown files. The model reads to inform its next inference and writes to store results for later. Unlike the other hands, this one operates on the agent’s own state rather than external systems, but write access to a shared memory store has real implications in multi-agent architectures.
RAG/Retrieval: Retrieval Augmented Generation. Data access that feeds back into the inference loop. Retrieval is frequently implemented as a tool call and is often assumed to be read-only, but that can still carry risks. First, the data the agent retrieves may be sensitive, including PII, financial records, and confidential documents. The agent’s retrieval scope is an access control problem. Second, retrieved content becomes part of the model’s context, making it a prompt injection vector; a malicious document in the retrieval corpus can instruct the agent to take actions the operator never intended. What the agent reads can directly influence what it does next, including mutations through other hands.
Agent-to-Agent (A2A): One agent invokes another. The calling agent issues a task; the subagent executes it, potentially using any of the above hands, and returns a result. The orchestrator doesn’t necessarily see or control how the subagent accomplishes the task. Trust and permission scope compound across the call chain.
Simon Willison gave us the cleanest framing of agentic risk: the lethal trifecta. An agent becomes dangerous when it has all three of: access to private data, exposure to untrusted content, and a channel to communicate externally. Any two are survivable. All three mean an attacker can plant instructions in something the agent reads, have it gather your sensitive data, and exfiltrate the results. The model will do it cheerfully because it cannot reliably distinguish instructions from data. This has bitten companies like Microsoft, Salesforce, and ServiceNow.
Look back at the list of capabilities. RAG is untrusted content. Memory and retrieval are private data. Browser use, MCP connectors, and code execution are all exfiltration channels. Most production agents ship with the full trifecta on day one because each hand was added for a legitimate business reason. As we discuss invariants, you’ll see we focus a lot on minimizing the risk of the trifecta.
There’s one layer conspicuously absent from most AI stack diagrams: who is the agent, and what is it allowed to do?
Today, the answer is almost always “the agent is whoever authorized it.” The agent runs with the human’s OAuth tokens, API keys, and session. An agent who is asked to summarize a spreadsheet has the same permissions as an agent who is asked to clean up old files, because there is no mechanism to scope credentials to a task. This is the new SSH-in-as-root, except the workload is non-deterministic and reads instructions off the internet.
Agent identity is a hot issue, with a lot of research, and it plays a big role in our invariants.
Now that we have a common understanding of how these systems work, we can start to define the ways we want to protect them. In Part II we’ll discuss where invariants can be inserted into the stack, and what threats we may want to address with invariants. In Part III we’ll define a number of invariants you might want to implement in your organization. After that, our focus will dive into the major first-party and cloud service provider environments and what options these providers offer for actual implementation.