The Conventional Wisdom
Your compliance team wants to know if the AI model you're deploying is "secure." Your vendor shows you certifications. Your engineers run penetration tests against the model's API. Your auditors ask about the training data provenance.
Everyone's focused on the model itself, its accuracy, its biases, its training methodology. The assumption: if the model is sound, the AI system is sound.
Why It's Incomplete
The model is one part of the service. The agent harness is the rest, the scaffolding your application builds around the model. And here's what nobody wants to admit: the harness is where your AI system will actually fail in production.
I've watched teams spend months vetting a language model, then deploy it with a harness that lets it call any API endpoint it requests. I've seen organizations implement perfect model governance while their tool contracts accept any JSON blob without validation. The model performed exactly as designed. The harness let it execute 47 unintended database queries before anyone noticed.
Your AI agent doesn't operate in isolation. It takes inputs from users, calls tools in your environment, and returns outputs to systems that make decisions. Every one of those interactions is a control surface. Every one needs constraints. That's the harness.
The Evidence
What actually breaks when you deploy an AI agent?
Tool Contracts Fail. Your agent calls an API with malformed parameters because nothing enforced input schemas. Or it receives an error response your harness didn't anticipate, so it retries indefinitely. Tools need contracts that limit mistakes: input and output schemas, timeouts, and defined error states. If your harness doesn't validate tool calls before execution, you're running blind.
Permission Boundaries Leak. Your agent can read customer data to answer support questions. Can it also write to the billing system? Delete records? The model doesn't know. The harness must enforce permissions at every tool invocation. Consider integrating access control directly with your data layer, Oracle AI Database combines vector search with row-level security, ensuring the model never sees data the user couldn't access manually.
Observability Gaps Hide Failures. When your agent makes a bad decision, can you trace why? OpenTelemetry now provides semantic conventions for AI operations, letting you instrument model calls, tool invocations, and decision chains. Without this visibility, you're debugging production incidents by reading chat logs.
Output Validation Doesn't Exist. Your agent generates a SQL query. Does your harness validate it before execution? Or does it assume the model "knows better"? I've seen agents confidently return DROP TABLE statements. The model wasn't malicious. The harness just never said no.
What to Do Instead
Build your harness before you pick your model. Here's the checklist:
Define Tool Contracts First. For every tool your agent might call, specify:
- Input schema with type validation and range limits
- Output schema with expected error codes
- Maximum execution time
- Retry policy (or no retries for write operations)
- Required permissions
Write these contracts in code. Don't document them and hope the integration layer enforces them.
Implement Least-Privilege Tool Access. Your agent gets a service account. That account gets exactly the permissions needed for its documented function. If it's a support agent, it reads tickets and knowledge base articles. It doesn't write to the CRM. If it needs to escalate, it calls a tool that creates an escalation record, you control what that tool can do.
Instrument Everything. Every model call, every tool invocation, every decision point gets logged with context. Use structured logging that captures:
- Input that triggered the agent
- Model response
- Tools called and their results
- Final output returned
- Execution time for each step
OpenTelemetry's semantic conventions give you a standard vocabulary. Your observability platform can then correlate agent behavior across requests.
Validate Outputs Before They Leave the Harness. Your agent generates a response. Before you return it:
- Check for data leakage (did it include information outside the user's scope?)
- Validate format (is it parseable JSON if that's what you expect?)
- Scan for injection attempts (is it trying to manipulate downstream systems?)
- Rate-limit based on cost or risk
Think of it as egress filtering for AI outputs.
Test the Harness, Not Just the Model. Your test suite should cover:
- What happens when a tool times out?
- What happens when a tool returns an error the agent hasn't seen before?
- What happens when the agent requests a tool it doesn't have permission to use?
- What happens when the model returns malformed JSON?
These aren't model failures. They're harness failures. And they're completely predictable.
When the Conventional Wisdom Is Right
Model security does matter. If you're using a model trained on proprietary data, you need to verify that data handling. If you're accepting user inputs that get embedded in prompts, you need injection protection. If your model makes decisions that affect people, you need bias testing.
But here's the thing: those are harness concerns too. Prompt injection protection? That's input validation in your harness. Bias testing? That's output validation. Data handling? That's your tool contracts enforcing what the model can access.
The model gives you capabilities. The harness gives you control. You can't audit, secure, or comply with capabilities alone. You need control surfaces. You need boundaries. You need the harness.
Stop asking if the AI model is secure. Start asking if you can constrain what it does.
