Audit and retention¶

Every request that goes through the proxy leaves a row, and every control-plane action leaves an event. Here's how to search them in Audit, how Operations sums them up, how to cross-reference them with the cluster's native audit log, and how long they're kept.

Search cluster requests¶

1. Open Audit. The Requests to clusters tab is selected.

2. The bar above the table keeps in view the four questions of an incident: who (Account), where (Cluster), when (Period) and how it went (Result). Account and Cluster are selects; Period and Result are chips, and pressing the chip that is on turns it off again. The rest is behind More filters:

   Filter	What it narrows
   Account	A person or a bot.
   Cluster	A cluster.
   Period	1 h, 24 h, 7 d, 30 d, or Custom. Only Custom shows From and To.
   Result	All; Denied (401 and 403); Failed (5xx, or an error after a success status, such as a stream the cluster cut); Writes (create, update, patch, delete and deletecollection, and also exec, attach and port-forward).
   Environment	All, Production (every environment marked as production) or one environment: the requests of the clusters it holds today. A cluster that changes environment takes its earlier requests with it, because the filter reads where it sits now, not where it sat when the request was made.
   Credential prefix	The klt_… prefix of a credential (you can see it in Credentials). Pasting a full token is fine: only its first 12 characters are kept.
   Namespace	An exact namespace.
   Verb	One at a time, in three families: Read (get, list), Write (create, update, patch, delete, deletecollection) and Stream (watch, proxy).
   Resource	For example pods or secrets.
   Tool, Agent session	One AI agent's requests: see AI agents.
   Command	The command that asked for the request, as kubectl declares it, without kubectl or arguments: get, apply, rollout restart. Matches only rows made with kubectl (see Where the command comes from).
   Include discovery	Also shows rows with no resource.

3. Click Search, or Apply inside More filters. Results come 50 at a time, most recent first.

Nothing is searched until you click Search, Apply inside More filters or Apply next to unapplied changes: choosing an account or pressing a chip only edits a draft. If you close More filters any other way, it keeps what you chose, and the sentence above the table adds unapplied changes until you apply it. Reset inside More filters empties its fields, not the four in the bar, and applies nothing until you click Apply. Once applied, the More filters button shows how many of its filters are in use.

The filters, the page and the tab live in the page's URL, so a link to an investigation reproduces it for whoever opens it; a sentence above the table says what is applied (for example "Denied requests by deploy-bot, in kind-local, last 24 h"), with a Clear button next to it. Paging through a relative period keeps the window's end fixed (the sentence adds until …, the pager says snapshot at …, and the URL keeps it as until), while Search starts again from page 1 with a window that ends now.

Above the bar, five figures count what the applied filters select: Requests, Denied, Failed, Writes and Agent calls (requests an AI agent made through an MCP tool). Denied, Failed and Writes count exactly what the Result chips of the same name keep; the figures are read-only, so to see those rows press the chip and Search. Under them, a sparkline draws the requests per hour of the period (per UTC day past 48 hours), with the denied share at the foot of each column in red; hover over a column to read its hour and its counts, and the line below gives the period and its peak. With no period chosen the table lists every request, but the figures count the last 24 hours, as that line says. In Free, the figures start where the 24 hours of audit do. GET /api/audit/summary gives the same counts: API.

A link can narrow the list to sensitive access with ?access= in the URL, as the sensitive-access figures of Home do: exec (the exec and attach subresources), secrets (get, list and watch on Secrets) or portforward. Allowed and denied requests count alike, so a figure and the list it opens show the same rows. No field in the bar sets it: the sentence above the table names it ("Only exec and attach") and its Clear filter button removes it and keeps the rest of the search.

Once an environment is marked as production, the sensitive-access line of Home counts the clusters of production alone and says so ("Sensitive access in production"). Its links add production=true to the address, which sets Production in Environment, so each figure is still the number of rows of the list it opens. ?environment= takes the id of one environment instead; asking for both, or for an environment that does not exist, is an error. Production shows nothing while no environment is marked as production.

By default, rows with no resource are hidden: discovery (/api, /apis, /version, /openapi), which kubectl does on every command, and also the requests kubelatch rejects before they reach the cluster. Those include authentication failures (no token, unknown, revoked, or expired token, the 401) and the 403s for a disabled account, a credential for another cluster, or no permissions. To see them, open More filters, check Include discovery and click Apply.

Rows for a 401 have no account, only the token's prefix: they don't show up if you filter by Account, and the person who owns the token doesn't see them in their My activity. Search by date, cluster, or the Credential prefix, which always includes discovery rows (there is no need to also check Include discovery).

A person without the admin role sees their own rows in My activity, which opens from See all my activity in the Recent activity card of Home: My activity.

What's in each row¶

Rows sit under a heading for their day (Today, or the date). The table shows the time to the second (hover for the full date), account, cluster, What, status, and duration. What is one line, verb resource/name (namespace): exec pods/web (default), delete pods/web-7f9c (shop), or discovery /api; for a request an AI agent made, a muted via apply follows it, with the tool. For a request made with kubectl, the command that asked for it follows, just as muted and without the tool's name: patch deployments/api (shop) followed by via rollout restart. When the command is the very word of the action (delete and via delete) it is not repeated. The status is a pill with the response code, tinted by its outcome, with the error in a line under it.

Identical requests one after another fold into one row: the same account, credential, cluster, action, object, status, error and source IP, at most 5 s apart on the same day. The row shows a × 12 button that unfolds the rest. A failure between two successes, or another credential or address, never folds under a success, and only the page you are looking at is folded. On a narrow screen the table is a list, one item per request or fold.

The requests of one kubectl run fold too, consecutive or not: a kubectl apply -f dir/ with thirty objects is one row that reads its main request like any other, patch configmaps/feature-flags (apps) followed by via apply, then +29 writes and a 62 requests button that unfolds them. The main request is the last write (the last request, if there was none); the status is the worst of them all, so a denial inside an apply never hides under a 200; the duration runs from the first to the last. Only requests of the same credential on the same cluster fold, and only within the page you are looking at. Clicking the row, or its What, opens the detail of the main request.

Click anywhere on a row, or its What, to open the Request detail in a panel on the right. The row's menu has the pivots of the panel's foot (Only this account, Only this cluster, Only this credential, Only this session) and Open. The panel opens on its title and one sentence that says the request, for example "deploy-bot changed configmaps/feature-flags in apps (kind-local) · 201 · 2 ms · via apply". Audit-ID and Request are copy fields, and the Account, Cluster and Agent session it names are links to their pages. The other fields:

At the foot of the panel, buttons pivot the investigation: Only this account, Only this cluster and Only this credential narrow the search you had to that one thing (its period and result stay) and take you back to the table; Only this credential also brings the discovery rows, so a rejected attempt doesn't disappear. Only this session (only on a request made with kubectl) narrows to the requests of that run, discovery rows included. Revoke this credential (shown only while the credential is still active) asks for confirmation and an optional reason, then revokes it right away, the same as from Credentials.

Statuses worth recognizing:

• 499 with client disconnected: the client hung up before receiving a response, for example a Ctrl-C.
• 403 with stream cut…: a revocation (of a credential, an account, or a permission) cut the stream before the cluster responded. If it was already open, the row keeps its 101 or 200 with that same error.
• A stream cut by a kubelatch restart ends with the error kubelatch restarting.

Authentication failures are logged at most once per IP every 10 s. Proxy policy 400s (paths with .. or %, Impersonate-* headers) don't leave a row, but the response still carries the Audit-ID header.

exec, attach, and port-forward¶

These are recognized by the subresource, not the verb. kubectl uses get over WebSocket and create over SPDY for the same action. What already translates this: exec, attach, port-forward, or logs.

To find them, press Writes in Result, which counts exec, attach and port-forward as writes, and filter by Resource pods in More filters. The Verb filter doesn't help here.

Where the command comes from¶

kubectl adds two headers to every request, Kubectl-Command and Kubectl-Session: the command without its arguments (kubectl rollout restart) and one identifier per run, the same on every request of one kubectl …. It sends them by default since version 1.22 and always since 1.35; with a version between 1.22 and 1.34, KUBECTL_COMMAND_HEADERS=false in the environment of whoever runs kubectl turns them off. kubelatch keeps them with the row, without the word kubectl on screen, and uses them to say via rollout restart and to fold the requests of one run.

helm, k9s, Lens, Argo CD, Flux, Terraform, kubectl-* plugins (krew included) and whatever goes through kubectl proxy send none: their rows carry no command and read as before, with their User-Agent. The header never carries arguments, resource names, files or flag values; the request itself gives those, as always.

The command is declared by the client: anyone can send another one, or none. kubelatch stores it as it came (dropping what does not have the shape of a command or of an identifier) and decides nothing by it: no permission, no approval, no security filter. It is a label for reading the audit, not proof of what ran; the proof is the request.

Cross-reference with the cluster's native audit log¶

Every proxy response, including errors, carries the Audit-ID header (kubectl -v=8 shows it). It's the row's id and the same auditID the API server writes to its own audit log.

1. Copy the Audit-ID from the row's detail.
2. Look it up in your provider's native audit log (CloudWatch on EKS, Cloud Logging on GKE, Azure Monitor on AKS) or in the API server's audit log file.

Keep the provider's native audit log turned on. Anything done with a ServiceAccount token obtained from within a pod doesn't go through kubelatch and is only recorded there.

AI agents¶

Every request an AI agent sends to a cluster is a row like any other, with two more fields: the tool and the agent session. In More filters, Tool and Agent session narrow the table to one tool or one agent, and What ends in the muted via and the tool. A delegated agent's rows have its person as subject: filter by Agent session to tell them from the person's own kubectl. Rows of a session's tool calls are also on the session's page in Agents, under Timeline, grouped by call, where its owner sees them too.

• read_secret leaves one row per call: each read of a Secret's values is on record.
• A write held for approval leaves a row for its preview and one for its dry run; the approved run is a third row. The approval's page says when it ran and what the cluster answered.
• whoami and request_access reach no cluster and leave no row here. Sessions and approvals are in the control plane: agent.session.start, agent.request.create and the rest, below.
• Closed approvals are kept as long as the rows here (KUBELATCH_AUDIT_RETENTION); a held write's body is deleted as soon as it's denied, runs or expires.

The control plane¶

The Control plane tab lists every administration action and every login attempt, one per row under a heading for its day, in fixed columns: Time, Account, What, Target and IP.

• What is a sentence built from the event's data, for example Credential "laptop" of "ux-dev" revoked: "lost" or "ana" added to group "ops", no expiry. An action kubelatch does not recognise shows its raw code alone.
• Account is the actor, by login when it is a known account. Actions with no actor show up as Server CLI (kubelatch-server) when they came from kubelatch-server user …, and as System or anonymous otherwise: automatic GitHub offboarding, failed attempts with unknown logins, a failed OIDC sign-in, a failed CI exchange.
• Target reads its kind in words (Account, Permission, Credential, Group…) and links to the page of the account, the account's permissions, the cluster, the group, the role, the credential, the agent session or the approval while it exists. One that was deleted stays as muted text followed by deleted, with no link; an approval reads that way once its cluster is gone. An AI agent's credential is plain text, since Credentials doesn't list it.
• Click the sentence, or anywhere on the row, to open the Event detail in a panel: the sentence, Date, Account, Action (in words, with its raw code beside it; a used account link reads by its purpose: invitation, reset, or linking with GitHub or the OIDC provider), Target and IP. View JSON unfolds the event's raw data (reason, via, …).
• A change made on several items at once (several permissions granted, revoked or extended, members removed or extended, approvals and access requests decided, or agent sessions cut) shows as one line while two or more of its rows sit together on the page: who did it, what it did and to how many (ana revoked 37 permissions), the derived changes it caused through groups (and 12 derived changes) and its reason in muted text. N rows opens one row per item, each with its own detail. Every row of the batch carries batch (the id the batch shares), batch_size and, when one was given, reason; an item that was already revoked or removed carries unchanged: true.
• A save of an environment, or of a cluster's environment and labels, also shows as one line when it cuts permissions, with the rows it cut under it: ana saved environment prod and cut 3 permissions or ana moved cluster eu-2 to prod and cut 2 permissions. The count is the permissions the save cut; the rows of a group's members follow their group permission. The sentence of each row cut ends with cut by the rule of prod. When the save also took away or gave permissions on a set, the sentence of its row adds how many.
• A permission on a set reads Permission viewer on staging granted to "alice", and the same when it's revoked or extended. Checking it again reads how many more clusters it reaches. What the set puts on each cluster has its own row, and its sentence ends with via staging.

The bar has Account (the actor), Action, Cluster and Period, and Search. Action opens a list grouped by family that you can search by its words or by the code, and All actions clears it. Cluster keeps the events of one cluster: its own and those of the permissions on it, the same that Changes shows on the cluster's page. Nothing is searched until you click Search (or Apply next to unapplied changes). The filters and the page are kept in the page's URL so a search can be shared or bookmarked, same as in Requests to clusters, and the sentence above the table says what is applied, with its Clear.

Action codes, by area:

An OIDC sign-in's login.success has the provider's issuer, the identity (subject_claim and subject), the email and, when the token carries them, amr and acr, which say how the person authenticated at the provider. A failed one's login.failure names the e-mail when known, and its reason is one of the codes in Errors signing in with OIDC: ambiguous_account lists the accounts in candidates, and oidc carries the provider's error in idp_error. oidc.link also has how many pending links it revoked (invites_revoked) and, when it replaced an identity of another provider, replaced_issuer or replaced_claim. A link.complete used through the OIDC provider carries the identity, and email_mismatch when the provider's e-mail isn't the account's. No event holds a token, a code or the client secret.

A break-glass account's password sign-in while GitHub or OIDC login is on has login.success with break_glass: true and method: password, and the page reads it as «{login} signed in with the break-glass password». It also reaches the administrators by e-mail and in the bell (Notifications).

Useful signals: several login.failure events in a row from the same IP, and oidc.link events with via email you didn't expect. Notices, in the top bar, counts those of the last seven days.

Operations¶

Operations, under Record beside Audit, sums up how your clusters are used: who works on them, what stands out and what nobody uses. Cluster operations, in the Trends of Home, opens it too, and each figure there leads to its part of the page.

The period is in the header: 7 days, 30 days or 90 days, and the page's address keeps it (?period=7). In Free the page reads the last 24 hours: its period is 24 h, with With Pro: 7, 30 and 90 days.

Four figures of the period come first, each with its chart:

• Short-lived credentials: the share of accounts that got in only with credentials that expire on their own (Short-lived and long-lived).
• Permissions that end on their own: the share of permissions in force with an end date, how it moved in the period and its line, with links to those without expiry and to the accounts with cluster-admin.
• Requests: the total and the share denied, which opens those requests in Audit. A column per day (per hour in Free) has the denied ones at its foot, and peak names the busiest day.
• Sensitive access: the exec (and attach), Secret reads and port-forward of the period, and how many accounts made them.

Under them come eight charts, and each mark in them leads to where you see the detail or fix what it shows:

• Anomalies: a bubble per account, with its requests across, its share of denied requests up and its sensitive access as its size, in the colour of its kind (People, Bots, Agents). The accounts that stand out most carry their name. A bubble opens the account's page; an AI agent's opens the account it acts for.
• Sensitive access: the exec, Secret reads and port-forward of the period. A slice opens Audit with those requests.
• Most active: the eight accounts with the most requests, allowed and denied. A bar opens the account's page.
• Requests by account type: each day's requests from people, bots and AI agents.
• Clusters: each cluster sized by its requests and coloured by its worst exposure: Serious when it gives privileges to everyone or is in error, To review for cluster-admin without expiry, Kubernetes out of support (more than three minor versions behind the one kubelatch is built against), the bootstrap v1 or permissions without expiry. A rectangle opens where that gets fixed: Needs attention or Details on the cluster's page, or Permissions narrowed to that cluster.
• Unused permissions: by cluster, the permissions in force without a single request in 30 days. A bar opens them in List for that cluster, and Review all N opens all of them (Revoke several permissions). Under the bars, and N unused credentials opens Credentials.
• How accounts get in: people, bots and agents by the credential they used, each row with how: 5 with the CLI · 1 unused, 2 with a CI token · 1 with a long-lived credential, 2 for people · 1 for bots. N active long-lived credentials opens them in Credentials, and N with permissions and no activity in 90 d opens those accounts in Users.
• When people work: the requests by weekday and hour, in your browser's time, and the share made outside working hours (Monday to Friday, 08:00 to 20:00).

An account stands out when it has at least 20 denied requests and a share of denied requests at least three times the median of the other accounts, and never under 5 %; or when it has at least 10 sensitive accesses, at least three times the median of the other accounts that have any. Requests with a token that is unknown, revoked or expired show as No account, and their bubble opens the refused requests in Audit. The page also counts the requests kubelatch refuses before they reach a cluster, so over the same days it can count more than Requests, 7 days on Home.

The period moves everything but Unused permissions, which always covers 30 days. In Free, Requests and Requests by account type go by the hour, Unused permissions says With Pro: permissions nobody uses in 30 days, and Permissions that end on their own and the accounts' figures, which don't need the audit, cover 30 days and say so. If your audit retention keeps less than 30 days, Unused permissions says it can't measure them.

Short-lived and long-lived¶

Short-lived credentials and How accounts get in count the enabled accounts that hold a permission in force, and the AI agents that work for them, by the credential they used in the period:

• Short-lived: a CLI session (12 hours by default, renewed by kubelatch login), a CI token (1 hour, exchanged on each run) or an agent's delegated session. They expire on their own.
• Long-lived: a standard credential, which lasts up to 30 days for a person and 90 for a bot by default. An agent that works as a bot gets in with its bot's credential, so it counts as long-lived.
• Unused: none in the period.

Each account goes into one group, long-lived first: someone who uses the CLI and also a standard credential still has a long-lived secret in use. A person who only worked through an agent counts as short-lived, and the agent counts under Agents too. There is one agent per account and client (alice's Claude Code, for example) with a session active now or used in the period. The figure is the share of short-lived among the accounts that got in, without Unused; when nobody got in, it says Nobody got in in 30 days.

The rows tell how the accounts got in during the period, even with a credential revoked since. N active long-lived credentials counts the standard credentials that work now, used or not: each one is a secret that keeps working until it expires or you revoke it.

Retention¶

KUBELATCH_AUDIT_RETENTION sets how long requests and control-plane events are kept. It defaults to 90 days (90d). It accepts days (Nd) or a Go duration, with a one-day minimum. 0 or 0d disables deletion, unless the instance never had a key, which caps it at 24 hours.

In the Free edition and with a lapsed key, Audit shows only the last 24 hours: the listings report the earliest moment served in floor, and an older event opened by its Audit-ID answers 404, like an unknown id. An instance that never had a key also keeps the audit at most 24 hours (a longer value is capped, and 0 becomes 24 hours too); one that has or had a key keeps its configured retention and only stops serving what is older than 24 hours until a key is installed. See Editions and license.

A periodic job deletes whatever is past retention every hour; the first pass runs one minute after startup. The log says how many rows it deleted (retention: rows deleted) or whether it failed (retention: pass failed).

Nobody else can modify or delete rows: see Immutable audit log.

Personal data¶

Rows store the IP, the login, the User-Agent, and the query string (the command= of each exec). The control plane stores the IP and login of each login attempt. The purpose is security and traceability of cluster access. The retention period is the answer to "how long do you keep this".

If you need to keep more than you want in the database, export to immutable storage before the retention job deletes it: pg_dump -t audit_events -t control_events, or GET /api/audit and GET /api/control-events (HTTP API).

Size¶

Each request is a row of around 1 KB, and kubectl makes several per command, not counting discovery. As a reference point, 50 active people generate a few hundred MB over 90 days.

To include the audit log in your backups, see Upgrades and backups.