[{"data":1,"prerenderedAt":346},["ShallowReactive",2],{"blog-en-k3s-ai-agent-audit-log-traceability-gap":3,"blog-related-en-k3s-ai-agent-audit-log-traceability-gap":296,"blog-en-k3s-ai-agent-audit-log-traceability-gap-alt":284},{"id":4,"title":5,"author":6,"body":7,"date":278,"description":279,"extension":280,"image":281,"locale":282,"meta":283,"navigation":284,"path":285,"seo":286,"stem":287,"tags":288,"__hash__":295},"blog\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap.md","We Couldn't Reconstruct What an AI Agent Did to Our Kubernetes API. The Gap Was in Our Audit Log Settings","Kubo Team",{"type":8,"value":9,"toc":268},"minimark",[10,15,23,26,37,46,50,56,63,92,95,113,117,123,126,135,138,156,159,163,169,172,183,194,200,209,213,219,222,236,245,249,252,255],[11,12,14],"h2",{"id":13},"_1-what-happened-we-couldnt-reconstruct-what-an-ai-agent-did-from-the-audit-log","1. What Happened — We Couldn't Reconstruct What an AI Agent Did from the Audit Log",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap\u002Fsection01.webp",[16,24,25],{},"It happened right after we started running an AI agent as a Pod on Kubernetes (K3s) in-house. There were signs that the agent had performed some operation on resources in the cluster, and we needed to confirm whether an unintended configuration change had occurred.",[16,27,28,29,36],{},"There's an obvious place to look first: the kube-apiserver's Audit Log. According to the ",[30,31,35],"a",{"href":32,"rel":33},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fdebug\u002Fdebug-cluster\u002Faudit\u002F",[34],"nofollow","official Kubernetes documentation",", the Audit Log is supposed to be the primary source of truth for tracking who did what, when, and to what — recording every request to the cluster in chronological order.",[16,38,39,40,45],{},"But when we actually opened the log, all it recorded was the fact that \"a certain Service Account called the API at a certain time.\" Nothing told us what the AI agent was trying to accomplish, or what chain of decisions led it to that operation. It was at this point we realized that running an AI agent on a platform like ",[30,41,44],{"href":42,"rel":43},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fcaptain-ai\u002F",[34],"Captain.AI"," requires an operational design that accounts for this kind of traceability from the start.",[11,47,49],{"id":48},"_2-cause-1-the-default-audit-policy-doesnt-record-request-content","2. Cause #1 — The Default Audit Policy Doesn't Record Request Content",[16,51,52],{},[19,53],{"alt":54,"src":55},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap\u002Fsection02.webp",[16,57,58,59,62],{},"Digging further, one cause turned out to be the Audit Policy configuration itself. The ",[30,60,35],{"href":32,"rel":61},[34]," defines four audit levels:",[64,65,66,74,80,86],"ul",{},[67,68,69,73],"li",{},[70,71,72],"strong",{},"None",": events matching this rule are not logged at all",[67,75,76,79],{},[70,77,78],{},"Metadata",": logs metadata only — requesting user, timestamp, resource, verb, etc. — but not the request or response body",[67,81,82,85],{},[70,83,84],{},"Request",": logs metadata plus the request body, but not the response body",[67,87,88,91],{},[70,89,90],{},"RequestResponse",": logs metadata, request body, and response body in full",[16,93,94],{},"Many clusters default to something close to the Metadata level, to keep log volume and storage costs down. Under that setting, the fact that \"the AI agent called an API to update a Pod spec\" is recorded, but the content of the request — which field was changed to what value — is not. If a human operator were typing commands manually, you could cross-reference work logs to fill the gap. But an AI agent autonomously fires off API calls one after another, so this missing content becomes a dead end in any investigation.",[16,96,97,98,103,104,108,109,112],{},"The ",[30,99,102],{"href":100,"rel":101},"https:\u002F\u002Franchermanager.docs.rancher.com\u002Fhow-to-guides\u002Fadvanced-user-guides\u002Fenable-api-audit-log-in-downstream-clusters",[34],"official Rancher procedure for managing K3s\u002FRKE2"," shows a configuration example that applies the Audit Policy only to control plane nodes via ",[105,106,107],"code",{},"machineSelectorConfig",", raising it to ",[105,110,111],{},"level: RequestResponse",". In other words, it's not that \"K3s can't record this\" — it's that the default settings are almost always insufficient.",[11,114,116],{"id":115},"_3-cause-2-a-shared-serviceaccount-or-token-makes-it-impossible-to-tell-agents-apart","3. Cause #2 — A Shared ServiceAccount or Token Makes It Impossible to Tell Agents Apart",[16,118,119],{},[19,120],{"alt":121,"src":122},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap\u002Fsection03.webp",[16,124,125],{},"The second cause was in our authentication design. We were running multiple agent tasks under the same Service Account or a reused token, so the Audit Log recorded every one of them under the same \"name.\" Traceability — the ability to reconstruct an action after the fact — never had a chance to exist in the first place.",[16,127,128,129,134],{},"This isn't a problem unique to our setup. ",[30,130,133],{"href":131,"rel":132},"https:\u002F\u002Fwww.qovery.com\u002Fblog\u002Fai-agents-kubernetes-audit-trails-policy-guardrails",[34],"An analysis from Qovery"," points out that when agents run under a shared kubeconfig or a reused CI token, the Audit Log retains only a fixed identifier — and \"identity cannot be retro-fitted onto audit events that were already written.\" Six months later, if someone asks who made a given change and why, there's no way to answer.",[16,136,137],{},"That same article argues that an auditable agent action needs at least five recorded fields:",[139,140,141,144,147,150,153],"ol",{},[67,142,143],{},"Actor identity",[67,145,146],{},"Timestamp",[67,148,149],{},"The exact request and response content",[67,151,152],{},"The human who approved it, or the policy that auto-approved it",[67,154,155],{},"The resulting resource state",[16,157,158],{},"And it argues these should be stored in append-only storage outside the cluster, somewhere the agent's own workload can't touch. Fixing the Audit Log settings alone isn't enough — the authentication design itself needs to be rethought.",[11,160,162],{"id":161},"_4-the-fix-per-agent-identity-and-a-staged-audit-policy-upgrade","4. The Fix — Per-Agent Identity and a Staged Audit Policy Upgrade",[16,164,165],{},[19,166],{"alt":167,"src":168},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap\u002Fsection04.webp",[16,170,171],{},"We ended up making three concrete changes.",[16,173,174,177,178,182],{},[70,175,176],{},"First, we split out a separate Service Account per agent."," As the ",[30,179,35],{"href":180,"rel":181},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fsecurity\u002Fservice-accounts\u002F",[34]," explains, since v1.22 Kubernetes defaults to short-lived, auto-rotating tokens via the TokenRequest API. Splitting these by agent function means the Audit Log can finally tell agents apart, giving traceability a real technical foundation.",[16,184,185,177,188,193],{},[70,186,187],{},"Second, we scoped permissions down to namespace-scoped Roles.",[30,189,192],{"href":190,"rel":191},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Freference\u002Faccess-authn-authz\u002Frbac\u002F",[34],"official RBAC documentation"," makes clear, a ClusterRole applies cluster-wide, so a single misbehaving agent can affect every namespace. Binding to a namespace-scoped Role via RoleBinding shrinks the blast radius considerably.",[16,195,196,199],{},[70,197,198],{},"Third, we raised the Audit Policy level only for operations that matter."," Logging every single request at RequestResponse level would blow up log volume, so we added rules that raise the level specifically for operations we don't want an AI agent performing unsupervised — like modifying Secrets or changing Deployments and StatefulSets.",[16,201,202,203,208],{},"Rather than building this kind of permission and logging design from scratch, there's also the option of using a platform where operations are visible by default. ",[30,204,207],{"href":205,"rel":206},"https:\u002F\u002Fkubo.hexabase.io\u002F",[34],"Kubo","'s Captain UI is built around the principle of not letting cluster operations become a black box, which cuts down on exactly this kind of Audit Policy trial-and-error.",[11,210,212],{"id":211},"_5-the-gap-thats-still-left-you-need-observability-at-both-the-infrastructure-and-application-layers","5. The Gap That's Still Left — You Need Observability at Both the Infrastructure and Application Layers",[16,214,215],{},[19,216],{"alt":217,"src":218},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap\u002Fsection05.webp",[16,220,221],{},"Even after all that, a gap remains. The Audit Log is ultimately just an infrastructure-layer record of whether a request to kube-apiserver was permitted. The actual decision-making process — the agent querying an external LLM and deciding which tool to invoke based on the response — happens entirely outside Kubernetes.",[16,223,224,229,230,235],{},[30,225,228],{"href":226,"rel":227},"https:\u002F\u002Fwww.cncf.io\u002Fblog\u002F2025\u002F08\u002F19\u002Fcelebrating-100-days-of-kagent\u002F",[34],"The CNCF-backed kagent project"," is open source infrastructure for exactly this space — declaratively running AI agents on Kubernetes — and has been accepted as a CNCF Sandbox project. ",[30,231,234],{"href":232,"rel":233},"https:\u002F\u002Fwww.solo.io\u002Fblog\u002Fkagent-enterprise",[34],"Solo.io's announcement"," about its commercial extension in this space describes an approach called \"AgentOps,\" which traces interactions between users, agents, tools, and LLMs end to end. That reinforces the point: it's only by layering application-level tracing on top of infrastructure-level Audit Logs that you get the full picture of what an agent actually did.",[16,237,238,239,244],{},"Zooming out further, this isn't just one company's operational know-how — it's becoming an industry-wide concern. Even ",[30,240,243],{"href":241,"rel":242},"https:\u002F\u002Fwww.nist.gov\u002Fitl\u002Fai-risk-management-framework",[34],"NIST's AI Risk Management Framework"," emphasizes, for autonomous AI systems, the importance of documenting in advance which tools an agent can invoke and which data it can access, and making clear where accountability sits. Audit Log configuration is just one starting point for backing that accountability with actual technical controls.",[11,246,248],{"id":247},"summary","Summary",[16,250,251],{},"The Kubernetes Audit Log can tell you whether an API call was permitted — but it can't tell you what an AI agent was thinking, or what it actually did. That's less a misconfiguration than a fundamental limit of what an infrastructure-layer logging mechanism is built to guarantee.",[16,253,254],{},"If you're running AI agents on a production K3s\u002FKubernetes cluster, you need to combine four things before you can escape the \"we don't know what it did\" state: per-agent identity, namespace-scoped permission design, an Audit Policy upgraded specifically for sensitive operations, and application-layer tracing.",[16,256,257,258,261,262,267],{},"Running AI agents in production demands exactly this kind of robust infrastructure. ",[30,259,44],{"href":42,"rel":260},[34]," is an AI agent execution platform that runs on top of Kubo, designed from the ground up around this kind of transparency. Rather than reinventing Audit Policy design from scratch, it's worth considering a platform where agent behavior is visible by default. Before you take AI agent operations into production, ",[30,263,266],{"href":264,"rel":265},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[34],"get in touch"," for a conversation.",{"title":269,"searchDepth":270,"depth":270,"links":271},"",2,[272,273,274,275,276,277],{"id":13,"depth":270,"text":14},{"id":48,"depth":270,"text":49},{"id":115,"depth":270,"text":116},{"id":161,"depth":270,"text":162},{"id":211,"depth":270,"text":212},{"id":247,"depth":270,"text":248},"2026-11-09","After running AI agents on Kubernetes, we realized that when something went wrong, the Audit Log alone couldn't tell us who did what, or why. Here's the Audit Policy blind spot and how we fixed agent traceability.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap",{"title":5,"description":279},"blog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap",[289,290,291,292,293,294],"k3s","kubernetes","ai-agent","audit-log","rbac","observability","-QcNH7ED4T8SOqAYFTj-YJ02LuV5RYJ3xamO1imGx_Q",[297,304,312,321,330,338],{"path":298,"title":299,"description":300,"date":301,"tags":302},"\u002Fblog\u002Fen\u002Fk3s-mcp-ai-agent-rbac-permission-bypass","We Asked an AI Agent to 'Check a Pod,' and It Took Over the Entire K3s Cluster: What an MCP Server's Permission Bypass Teaches Us About RBAC Design","A real-world MCP permission bypass shows why Kubernetes\u002FK3s clusters need disciplined RBAC design and audit-log separation before letting AI agents operate them.","2026-09-03",[289,290,303,293,291],"mcp",{"path":305,"title":306,"description":307,"date":308,"tags":309},"\u002Fblog\u002Fen\u002Fk3s-prometheus-agent-mode-stale-marker-edge-monitoring","We Switched K3s Edge Prometheus to Agent Mode. The Moment a Target Disappeared, So Did Its Metrics","Running Prometheus Agent mode on K3s edge clusters can leave stale values in your central dashboard when a target disappears, depending on how stale markers are handled. Here's how to design K3s edge monitoring correctly.","2026-11-03",[289,290,310,311,294],"prometheus","edge-computing",{"path":313,"title":314,"description":315,"date":316,"tags":317},"\u002Fblog\u002Fen\u002Fk3s-k8s-default-serviceaccount-automount-rbac-multi-tenant","K3s and Kubernetes Share the Same Default ServiceAccount Trap. What Happens When You Leave Automount Enabled in a Multi-Tenant Cluster","Comparing k3s vs k8s doesn't change the default ServiceAccount automount trap. We cover the privilege-escalation risk in multi-tenant clusters, a real CVE, and a 4-layer defense.","2026-10-23",[289,290,293,318,319,320],"serviceaccount","multi-tenancy","security",{"path":322,"title":323,"description":324,"date":325,"tags":326},"\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics","Question Your discoveryInterval: Why K3s HPA Keeps Scaling on Stale Metrics from Prometheus Adapter","K3s HPA won't scale even though CPU usage looks normal. The culprit is a mismatch between Prometheus Adapter's metrics-relist-interval\u002Fmetrics-max-age and the HPA sync-period. Here's the three-layer delay chain and how to diagnose it.","2026-10-12",[289,290,327,328,329,294],"hpa","prometheus-adapter","autoscaling",{"path":331,"title":332,"description":333,"date":334,"tags":335},"\u002Fblog\u002Fen\u002Fk3s-servicemonitor-prometheus-not-scraping-connection-points","Your ServiceMonitor YAML Is Correct. So Why Isn't Prometheus Scraping It? A K3s Diagnostic Guide to the 'Connection Points'","You configured a kubernetes service monitor on K3s, but nothing shows up in Prometheus Targets. 90% of the time, the cause is one of four connection points: label selectors, port names, RBAC, or the namespaceSelector. Here's a step-by-step diagnostic guide.","2026-08-30",[289,290,310,336,337,294],"service-monitor","monitoring",{"path":339,"title":340,"description":341,"date":342,"tags":343},"\u002Fblog\u002Fen\u002Fk3s-opentelemetry-observability-correlation","Stop Bouncing Between Three Dashboards: Correlating K3s Observability with OpenTelemetry","Are you burning incident-response time checking Prometheus, Loki, and Jaeger separately? Learn how to correlate metrics, logs, and traces with OpenTelemetry to cut investigation time on K3s clusters, with practical Collector deployment patterns.","2026-08-20",[289,290,344,294,345],"opentelemetry","distributed-tracing",1794305506444]