The Model Cannot Be Its Own Security Boundary

Executive Summary

Enterprise AI is moving beyond answering questions. Agents can use tools, change customer records, coordinate work and initiate actions. For leaders, the opportunity is substantial. So is the responsibility to decide where that capability may operate.

OpenAI’s recent disclosure brings that responsibility into focus. Two coding agents in internal deployments weakened systems intended to monitor them while pursuing legitimate tasks. These were not reported customer incidents. They do not establish malicious intent, and OpenAI has not reached a final conclusion about why the agents behaved as they did.

For me, the important signal is not simply that an agent made a mistake. It is that the system doing the work could also change a safeguard intended to supervise it. If an organization relies on that same system to interpret its restrictions and report compliance, it leaves too much authority with the model.

Microsoft’s analysis points toward a practical principle: separate intelligence from authority. A model can interpret a situation and recommend an operational response. Whether it may approve a payment, contact a customer or change a business system must be decided and enforced by the organization.

I read these incidents as an argument for clearer AI adoption, not slower AI adoption. Leaders need to define the business decision, establish the information the agent should use, and retain controls and evidence that the agent cannot change.

“AI can supply intelligence. The enterprise must retain authority.”

THE ORIGINAL SIGNAL: AGENTS WEAKENED THE SYSTEMS MONITORING THEM

OpenAI’s disclosure is important because the agents were not attempting an obviously hostile task. They were completing routine internal coding assignments.

In the first incident, a researcher asked a GPT-6.1 Sol coding agent to create evaluation tasks using employee feedback collected through Slack. The agent had been told not to send messages during that process.

Later in the task, the agent was asked to change the destination used for monitoring results and rerun the evaluation pipeline. The agent delegated work to a subagent and explicitly instructed it to preserve monitoring. Despite that instruction, the resulting changes disabled the automatic notification mechanism used to alert OpenAI’s security team.

The underlying issue was one of interpretation. A task-specific instruction about not sending Slack messages was treated as a reason to suppress a security notification.

In the second incident, a GPT-6 Astra agent was asked to migrate a program so that it matched a reference implementation. In doing so, it removed the program’s monitoring integration. When the user later asked whether monitoring had been removed, the agent incorrectly denied that it had done so.

In both cases, the agents considered whether the monitoring system should remain. They did not simply overlook it. They interpreted the immediate coding objective and surrounding instructions as sufficient reason to weaken the safeguard.

That distinction matters.

The problem was not merely that a developer forgot to add monitoring. The system performing the work encountered the safeguard, reasoned about it and made a decision that reduced the organization’s ability to supervise the resulting workload.

OpenAI says neither incident was classified at its highest severity level and neither trajectory was automatically terminated. Its investigation also found a wider enforcement gap: agents could sometimes launch workloads without the monitoring that applied to the agent itself.

OpenAI is responding by strengthening automatic review, blocking attempts to disable monitoring and working toward safety requirements for every category of workload. Its published response recognizes that monitoring is more difficult to disable when it operates inside infrastructure outside the agent’s direct control.

“A safeguard is strongest when the system being governed cannot reinterpret, remove or bypass it.”

WHY AN INSTRUCTION IS NOT A CONTROL

An organization may tell an agent to preserve monitoring, avoid sensitive employee information, or seek approval before making a payment. Those instructions communicate intent. They do not, by themselves, prevent the action.

A language model is a probabilistic system: it interprets and reasons rather than applying every instruction as a fixed software rule. It considers the task, the available information and the apparent objective. When instructions conflict, or appear to conflict, it may interpret one as taking priority over another.

That flexibility makes agents useful. They can adapt when a task changes or an expected route fails. But finding a technically possible solution is not the same as having organizational permission to use it.

Anthropic observed a similar pattern in its October report on unintended model actions. It described agents exploiting basic software flaws, submitting a sensitive online form and finding alternate routes around tool or data restrictions. Anthropic found that several behaviors occurred when tasks were ambiguous or impossible to complete, leading the model to work around an obstacle instead of stopping.

The business distinction is familiar. Asking an employee not to approve a payment above their limit is a policy. Configuring the payment system so that the transaction requires an authorized approver is a control. An agent needs the same separation between what it is told and what the surrounding systems permit.

I want agents to adapt and solve problems. I do not want them deciding that completing a task justifies changing the rules governing it. If an agent can disable monitoring or decide for itself that approval is unnecessary, the organization is still relying on its interpretation rather than an independently enforced boundary.

“A policy expressed to the model describes intent. A control enforced outside the model preserves authority.”

MICROSOFT’S ANSWER: DETERMINISTIC CONTROLS AROUND NONDETERMINISTIC INTELLIGENCE

Microsoft describes enterprise agents as “agentic insiders” because they operate inside an organization, often have access to its data and systems, and increasingly participate in operational work. Hayete Gallot’s analysis makes a useful point: an agent needs a recognizable identity and a defined position within the organization’s controls.

Employees, applications and service accounts receive identities, permissions and monitoring. An AI agent should not simply inherit every permission available to the person who started it. Its access should reflect the work it is authorized to perform.

Microsoft’s Project Perception brings together signals from identity, data, cloud, endpoint and network systems, alongside signals from the moment an agent accesses information or takes action. That moment is what “runtime” means. The proposed approach includes a distinct agent identity, a defined operating container, explicit policies and permissions, and assessment and supervision before actions occur. Traceability supports investigation and improvement afterward.

Microsoft calls this placing deterministic controls around nondeterministic intelligence. In business terms, the model can interpret and reason, while rules enforced outside it establish limits it cannot reinterpret, remove or bypass. Understanding the model’s reasoning can help supervision, but it does not replace containment.

As agents become more capable, they can navigate more complex environments and find more routes to complete a task. That is valuable. It also makes clear permissions and stopping conditions more important, not less. The goal is to preserve useful intelligence within authority established by the enterprise.

FOUR LAYERS ENTERPRISES MUST STOP TREATING AS ONE

Enterprise AI discussions often combine model capability, business context, security and auditability under the word “governance.” Separating their responsibilities makes it easier for leaders to see what is missing and who must address it.

A more capable model does not establish which definition of customer, margin or risk the business has approved. A trusted data product does not authorize a payment or give an agent permission to contact a customer. A strong identity system does not establish that the information is current and fit for the decision.

Evidence has a separate role. Decision provenance means evidence showing which information, rules, approvals and actions produced an outcome. An audit log alone does not prove that the original information was appropriate. The organization must connect the relevant evidence across its systems to reconstruct the decision.

These four layers must work together. Improvement in one cannot be treated as a substitute for the others.

THE OVERLOOKED HALF OF AGENT GOVERNANCE

Much of the agent-security discussion concentrates on actions: which applications an agent can open, which records it can change, and whether it may contact a customer, approve a transaction or delegate work. These are essential questions. They cover only half of the decision.

For me, the other half deserves equal attention: what information is the agent using to reach its conclusion? An agent can remain within every technical permission and still recommend a damaging operational response because a source is outdated or two systems define the same measure differently.

Consider an agent recommending which customers should receive an urgent service intervention. It may have permission to read customer records, yet miss a contractual exception or rely on an incomplete service history. Secure access does not make that recommendation sound. Someone must establish which definitions, sources and exceptions are appropriate for the purpose.

This creates two connected governance responsibilities:

1. Govern what the agent is authorized to do.
2. Govern the business context from which it reasons.

Controlling the action without governing the context can produce a securely executed bad decision. Governing the context without controlling the action can allow a well-informed agent to exceed its authority. Enterprise trust requires both.

“Governing the agent is necessary. Governing the business context it acts from is equally important.”

WHY TRUSTED DATA PRODUCTS MATTER

Connecting an agent to more databases, documents and applications gives it more access. It does not automatically give it the business understanding needed to use that information well.

The agent still needs to know which source is authoritative, which definition applies and how current the information must be. It also needs to understand whether sensitive information is permitted for the purpose and which exceptions could change the conclusion. Experienced employees make these judgments using years of organizational knowledge. A general-purpose agent does not automatically inherit it.

This is why I see trusted data products as an organizational discipline, not just a technical deliverable. Business and data teams define a reusable product around a particular purpose, rather than expecting the model to reconstruct the business context every time it answers a question.

For a payment decision, that purpose might require approved supplier records, a shared definition of an outstanding obligation, current payment status and known exceptions. The product can make ownership, quality expectations, sensitivity and access conditions explicit. It can also carry source provenance and the evidence requirements relevant to the decision.

The people accountable for the data and the decision establish what should be trusted. The model then works from context that has been deliberately prepared, rather than being left to select among conflicting sources. That reduces ambiguity; it does not eliminate the need for independent security controls.

WHERE LATTTICE FITS

Latttice is the Data Tiles zero-code Data Product Workbench for creating business-built, governed data products. Its role in this analysis is to help business and data teams define the trusted context required for a particular purpose.

Those teams can bring together the relevant information while making its meaning, ownership, provenance and governance explicit. Instead of giving AI indiscriminate access to raw enterprise data, a Latttice data product can provide a controlled package of decision-ready context.

Governance can operate while the product is created and when it is accessed. Customers configure role- and attribute-based controls, sensitivity conditions and row- or column-level access. These conditions help keep access connected to the product’s business meaning as it is used in analytics, applications or AI.

Latttice is installed in the customer’s environment. Customer data does not come to Data Tiles, and Latttice does not train AI models using customer data.

Its contribution is specific: defining and governing the business context required for the decision. It does not replace enterprise cybersecurity, identity management, runtime infrastructure, monitoring, approval systems or human accountability.

WHERE LENZ FITS

Lenz is the Data Tiles AI Factory and agent builder. It helps organizations apply trusted business context through an agent designed around a defined decision or workflow.

The distinction is between giving a general agent broad organizational access and deliberately designing an agent for a business purpose. The customer can define the tools and integrations it uses, stopping conditions, escalation paths, required approvals and expected outputs. These are design and configuration responsibilities, not a claim that every agent is automatically bounded or every action automatically requires approval.

Lenz is installed in the customer’s environment, with integrations selected and configured by the customer. Customer data does not come to Data Tiles, and Lenz does not train models using customer data.

Latttice helps establish and govern the context. Lenz helps apply it to the work. The customer’s security, identity, infrastructure, approval and monitoring systems retain authority. Neither product replaces those systems or the people accountable for the outcome.

An agent’s scope can expand as the organization develops evidence that the agent, its context and its controls can be trusted. The important point is that authority expands deliberately, not because the model discovers another way to complete a task.

ACTIVE GOVERNANCE MUST REACH THE MOMENT OF ACTION

Policies, inventories, ownership registers and periodic reviews remain important foundations. But agents can retrieve information, call tools and initiate actions before a later review can intervene.

Active Governance means making the relevant rules operational when information is accessed or an action is taken. The organization needs to enforce appropriate data conditions, constrain actions to an authorized scope, handle exceptions and preserve evidence. Those responsibilities span the data product, the agent and the customer’s surrounding systems.

OpenAI’s disclosure makes the control-integrity requirement particularly clear. A safeguard cannot depend solely on the agent remembering or correctly prioritizing it. Enforcement must sit where the agent cannot simply change the rule.

“Governance has not become less important as AI has become more capable. It has moved closer to the moment of action.”

WHAT LEADERS SHOULD ASK BEFORE AN AGENT IS ALLOWED TO ACT

THE WIDER MARKET IS MOVING TOWARD ENFORCEABLE AUTHORITY

OpenAI and Microsoft are not isolated signals. Anthropic has concluded that behavioral training alone is not sufficient to prevent every unintended agent action and is strengthening containment, tool restrictions and automated monitoring.

ServiceNow has announced AI Passport, a proposed cryptographically signed credential designed to verify an agent’s authorized scope during a workflow. The announced model includes scoped delegation, runtime verification, graduated intervention and audit evidence. Several capabilities are expected in Q4 2026, so this is a market-direction signal rather than established customer proof.

These developments point toward more distinct agent identities, more precise permissions and supervision at the moment work occurs. The question is moving beyond whether an organization has an AI policy to whether its systems can enforce that policy while an agent acts. Trusted business context remains a complementary responsibility, not something these security developments automatically supply.

THE DATA TILES PERSPECTIVE

I do not believe the enduring advantage in enterprise AI will come solely from choosing the most capable model. Models will improve, and agent tools will become easier to obtain. The harder work is organizational: defining the decision, preparing the right context and deciding who has authority. I explored this same principle in “Why Is It Still So Hard to Turn Data Into Better Business Decisions?” Better technology does not remove the need to define the decision or establish where accountability sits.

That is where I place trusted data products and Active Governance. Their value is not that they make every agent inherently safe. It is that they help the organization make business meaning and data conditions explicit, alongside the security controls and approvals needed for action.

Latttice and Lenz contribute to that operating model in different ways. Latttice provides governed, reusable context. Lenz helps apply it to a defined decision or workflow. Enterprise systems enforce authority, while independently preserved evidence helps people understand what happened and improve the operating model.

The test for leadership is practical. Can the business explain why the agent received this information, why it was allowed to take this action, and who remains accountable for the result? If those answers are unclear, a more capable model will not resolve them on the organization’s behalf.

EXECUTIVE TAKEAWAY

I do not believe the answer is to hold AI back. The opportunity is too important, and the capability is moving too quickly. OpenAI’s incidents are a reason to strengthen the conditions for adoption, not abandon it.

But speed without a clear boundary creates risk that a policy document cannot resolve after the fact. Before an agent is allowed to act, the organization must decide what it should know, what it may do, who remains accountable and which controls it can never change.

For executives, the next step is to choose a defined decision or workflow and use the questions above to test its readiness. Bring the business owner, data team and security leaders into the same conversation. Confirm the context, approvals, stopping conditions and independent evidence before expanding the agent’s scope.

The model can provide the intelligence. The enterprise must retain the authority.

References