1. What Happened — We Couldn't Reconstruct What an AI Agent Did from the Audit Log
It happened right after we started running an AI agent as a Pod on Kubernetes (K3s) in-house. There were signs that the agent had performed some operation on resources in the cluster, and we needed to confirm whether an unintended configuration change had occurred.
There's an obvious place to look first: the kube-apiserver's Audit Log. According to the official Kubernetes documentation, the Audit Log is supposed to be the primary source of truth for tracking who did what, when, and to what — recording every request to the cluster in chronological order.
But when we actually opened the log, all it recorded was the fact that "a certain Service Account called the API at a certain time." Nothing told us what the AI agent was trying to accomplish, or what chain of decisions led it to that operation. It was at this point we realized that running an AI agent on a platform like Captain.AI requires an operational design that accounts for this kind of traceability from the start.
2. Cause #1 — The Default Audit Policy Doesn't Record Request Content
Digging further, one cause turned out to be the Audit Policy configuration itself. The official Kubernetes documentation defines four audit levels:
- None: events matching this rule are not logged at all
- Metadata: logs metadata only — requesting user, timestamp, resource, verb, etc. — but not the request or response body
- Request: logs metadata plus the request body, but not the response body
- RequestResponse: logs metadata, request body, and response body in full
Many clusters default to something close to the Metadata level, to keep log volume and storage costs down. Under that setting, the fact that "the AI agent called an API to update a Pod spec" is recorded, but the content of the request — which field was changed to what value — is not. If a human operator were typing commands manually, you could cross-reference work logs to fill the gap. But an AI agent autonomously fires off API calls one after another, so this missing content becomes a dead end in any investigation.
The official Rancher procedure for managing K3s/RKE2 shows a configuration example that applies the Audit Policy only to control plane nodes via machineSelectorConfig, raising it to level: RequestResponse. In other words, it's not that "K3s can't record this" — it's that the default settings are almost always insufficient.
3. Cause #2 — A Shared ServiceAccount or Token Makes It Impossible to Tell Agents Apart
The second cause was in our authentication design. We were running multiple agent tasks under the same Service Account or a reused token, so the Audit Log recorded every one of them under the same "name." Traceability — the ability to reconstruct an action after the fact — never had a chance to exist in the first place.
This isn't a problem unique to our setup. An analysis from Qovery points out that when agents run under a shared kubeconfig or a reused CI token, the Audit Log retains only a fixed identifier — and "identity cannot be retro-fitted onto audit events that were already written." Six months later, if someone asks who made a given change and why, there's no way to answer.
That same article argues that an auditable agent action needs at least five recorded fields:
- Actor identity
- Timestamp
- The exact request and response content
- The human who approved it, or the policy that auto-approved it
- The resulting resource state
And it argues these should be stored in append-only storage outside the cluster, somewhere the agent's own workload can't touch. Fixing the Audit Log settings alone isn't enough — the authentication design itself needs to be rethought.
4. The Fix — Per-Agent Identity and a Staged Audit Policy Upgrade
We ended up making three concrete changes.
First, we split out a separate Service Account per agent. As the official Kubernetes documentation explains, since v1.22 Kubernetes defaults to short-lived, auto-rotating tokens via the TokenRequest API. Splitting these by agent function means the Audit Log can finally tell agents apart, giving traceability a real technical foundation.
Second, we scoped permissions down to namespace-scoped Roles. As the official RBAC documentation makes clear, a ClusterRole applies cluster-wide, so a single misbehaving agent can affect every namespace. Binding to a namespace-scoped Role via RoleBinding shrinks the blast radius considerably.
Third, we raised the Audit Policy level only for operations that matter. Logging every single request at RequestResponse level would blow up log volume, so we added rules that raise the level specifically for operations we don't want an AI agent performing unsupervised — like modifying Secrets or changing Deployments and StatefulSets.
Rather than building this kind of permission and logging design from scratch, there's also the option of using a platform where operations are visible by default. Kubo's Captain UI is built around the principle of not letting cluster operations become a black box, which cuts down on exactly this kind of Audit Policy trial-and-error.
5. The Gap That's Still Left — You Need Observability at Both the Infrastructure and Application Layers
Even after all that, a gap remains. The Audit Log is ultimately just an infrastructure-layer record of whether a request to kube-apiserver was permitted. The actual decision-making process — the agent querying an external LLM and deciding which tool to invoke based on the response — happens entirely outside Kubernetes.
The CNCF-backed kagent project is open source infrastructure for exactly this space — declaratively running AI agents on Kubernetes — and has been accepted as a CNCF Sandbox project. Solo.io's announcement about its commercial extension in this space describes an approach called "AgentOps," which traces interactions between users, agents, tools, and LLMs end to end. That reinforces the point: it's only by layering application-level tracing on top of infrastructure-level Audit Logs that you get the full picture of what an agent actually did.
Zooming out further, this isn't just one company's operational know-how — it's becoming an industry-wide concern. Even NIST's AI Risk Management Framework emphasizes, for autonomous AI systems, the importance of documenting in advance which tools an agent can invoke and which data it can access, and making clear where accountability sits. Audit Log configuration is just one starting point for backing that accountability with actual technical controls.
Summary
The Kubernetes Audit Log can tell you whether an API call was permitted — but it can't tell you what an AI agent was thinking, or what it actually did. That's less a misconfiguration than a fundamental limit of what an infrastructure-layer logging mechanism is built to guarantee.
If you're running AI agents on a production K3s/Kubernetes cluster, you need to combine four things before you can escape the "we don't know what it did" state: per-agent identity, namespace-scoped permission design, an Audit Policy upgraded specifically for sensitive operations, and application-layer tracing.
Running AI agents in production demands exactly this kind of robust infrastructure. Captain.AI is an AI agent execution platform that runs on top of Kubo, designed from the ground up around this kind of transparency. Rather than reinventing Audit Policy design from scratch, it's worth considering a platform where agent behavior is visible by default. Before you take AI agent operations into production, get in touch for a conversation.