Security model¶

What kubelatch guarantees, what it doesn't, and what each guarantee rests on. Read this before granting permissions or putting kubelatch into production. The mechanisms themselves are explained in Identity, System roles and The journey of a request; this page only states what they give you and where they stop.

What it protects and what it doesn't¶

kubelatch controls who reaches the API servers through it, with what identity, and what gets logged. Specifically:

• Every request carries its own credential, revocable instantly and with an expiry.
• The cluster sees the real person (user:<login>, or the name the cluster's identity template gives them) and only the groups from their active permissions.
• Every request is logged before it reaches the cluster, with an identifier that correlates it with the cluster's native audit log.
• The proxy's ServiceAccount can't impersonate anyone outside what kubelatch has granted.
• An environment's rules limit the permissions of people and groups, never those of bots: a bot that needs permanent access to prod gets it through a direct grant.
• A permission on a set of clusters never gives privileged access, because a set grows by itself: where the role or the name is privileged, the permission doesn't reach that cluster, and whoever needs it there gets it by hand, with its 8 hours.

kubelatch does not protect:

• What happens inside a namespace that's already been granted: the real boundary is the namespace (see System roles).
• Native access to clusters (cloud identities, admin kubeconfigs, ServiceAccount tokens pulled from a pod). That stays in the cluster's native audit log, not kubelatch's.
• The database against someone who holds its Postgres role, or the encryption key against someone who can read the Secrets in the management cluster.
• Workloads: kubelatch isn't in their path.
• What an AI agent does through another way into a cluster than kubelatch's MCP server, and what it does with the content it reads (see AI agents).

Credentials: klt_ tokens¶

A credential is an opaque token: klt_ followed by 32 random bytes in base64url.

• kubelatch stores only its SHA-256 and the first 12 visible characters (klt_ + 8), which are enough to recognize it in the inventory and in the audit log without being able to use it. The token has 256 bits of entropy, so a plain hash is enough: there's no dictionary to try.
• The token and the kubeconfig are shown only once, in a response the browser doesn't cache. The interface doesn't store them in browser storage or in the URL.
• There's no authorization cache: every request to the proxy checks the credential, so revoking it takes effect on the next request. Open streams are cut within 10 s at most.
• Expiry is mandatory: 30 days maximum for people and 90 for bots by default (KUBELATCH_MAX_TTL_USER, KUBELATCH_MAX_TTL_BOT). CI credentials last 1 h by default (KUBELATCH_CI_TOKEN_TTL).

Invitation and linking links (kli_…) follow the same scheme: single use, with an expiry, and only their hash is stored.

CLI sessions¶

kubelatch login ends in a credential of kind cli: a klt_ token like the others, with the same inventory, immediate revocation and audit. What sets it apart:

• It lasts 12 h (KUBELATCH_CLI_SESSION_TTL, never more than KUBELATCH_MAX_TTL_USER) and isn't renewed on its own.
• It's valid on every cluster where the person has permissions, and it's issued even with none: kubelatch status says so, and the proxy answers 403 as for any credential.
• It only exists for enabled people, never for bots, and only after the person approves the request in the web with their own session.
• Besides the proxy, the API accepts it as Authorization: Bearer, but only for self-service: your own profile, your own credentials and logout (the exact routes are in the HTTP API). Anything else answers 403 cli.scope. It never acts as an administrator, even if the person is one: it can't create permissions, clusters or users, and it can't approve another CLI sign-in.
• It never appears on screen: it goes from the API to a 0600 file (session.json) and from there to kubectl through the exec plugin. The kubeconfig carries no secret.

The approval in the web is the defence against phishing. A sign-in request lives 10 minutes; its codes are 32 random bytes, single use, and only their hash is stored. The browser flow (PKCE with S256) only ever redirects to http://127.0.0.1:<port> on the machine that started it, with the CLI's own state.

With --device, whoever holds a code and gets it approved gets the session. That's why the approval page warns you to approve only a kubelatch login you just ran, and why wrong codes are limited per person and per IP. Every approval, refusal and login leaves a cli.approve, cli.deny or cli.login event in Audit → Control plane.

The proxy and its ServiceAccount¶

• No cookies. The proxy only accepts Authorization: Bearer klt_… and ignores cookies, so a web page can't use your session against a cluster.
• No client Impersonate-*. Any incoming Impersonate-* header is rejected with 400. kubelatch already uses impersonation with the kubelatch-proxy ServiceAccount; letting a nested impersonation through would let anyone ask to be someone else. That's why kubectl --as isn't supported. To test "what can X do", use native admin access: kubectl auth can-i --as user:x --as-group kubelatch:ns:apps:viewer.
• Its own Audit-ID. The proxy generates its own and never honors the client's.
• Strict routes. It rejects ., .., empty segments and any escaping, so nothing gets decoded twice along the way.
• Scoped ServiceAccount. kubelatch-proxy can only impersonate the users and groups in the list the reconciler maintains: subjects with active permissions and the groups from those permissions. A failure in the proxy can't reach system:masters or anyone kubelatch hasn't granted something to.
• Prefixed names. user: and bot: avoid collisions with the cluster's native identities.
• One sentence on a 403. When the cluster refuses a request, the proxy adds one sentence to its message: the role that would allow it and where to ask for it (403 — no permission). It reveals nothing new: the namespace is the one in the request, each role's model is fixed, kubelatch doesn't ask the cluster, it names nobody else's permissions, and a custom role it names is one the person could ask for. Every other answer of the cluster, and every answer to a bot, passes as it came, and the audit log keeps the cluster's 403.

The reconciler and its ServiceAccount¶

The kubelatch-reconciler ServiceAccount is equivalent to cluster-admin by design: it has to be able to create the cluster-admin binding. It's never in the path of a client request. Its writes are bounded, which limits mistakes, not privilege.

Its ClusterRole may write ClusterRoles and bindings of any name: a custom role's ClusterRole, kubelatch-role-<id>, has a name nobody knows in advance, and RBAC can't limit a rule to a name prefix. The bound is the kubelatch-reconciler-jail admission policy, a ValidatingAdmissionPolicy (Kubernetes 1.30 or later) that the bootstrap manifest creates before that ClusterRole. It only looks at the reconciler's requests, and refuses:

• any ClusterRole, Role or binding whose name doesn't start with kubelatch-;
• the reconciler's own ClusterRole and ClusterRoleBinding, kubelatch-reconciler;
• the proxy's binding, kubelatch-proxy;
• a binding to anything but a ClusterRole (never a namespaced Role, whatever its name), or to any ClusterRole but kubelatch's tier and custom roles, admin and cluster-admin; never to kubelatch-reconciler or kubelatch-proxy.

The ServiceAccount can read the policy but not change or delete it. The policy bounds mistakes, and how far a compromised kubelatch reaches by name; it isn't a privilege boundary. It looks at names and at a binding's roleRef, not at subjects or rules: a stolen reconciler token can still bind cluster-admin to itself with a kubelatch-* ClusterRoleBinding, and a kubelatch-* ClusterRole carrying one of Kubernetes' aggregation labels (rbac.authorization.k8s.io/aggregate-to-admin, for instance) adds its rules to Kubernetes' own roles. The reconciler stays equivalent to cluster-admin, by design.

Each reconciliation checks that the policy and its binding are in place and still deny: a policy that lets through a request it can't evaluate, or a binding that only warns or narrows what the policy covers, doesn't count. If not, the reconciliation writes no custom role, deletes none, and leaves the cluster in Error until the manifest is applied again. Between someone deleting the policy and that pass, at most 10 minutes, the reconciler's writes are bounded only by kubelatch's own code. A cluster with the previous manifest has no policy: its reconciler can only write a fixed list of names, enough for the system tiers but not for custom roles. Update the bootstrap says how to move it to the current one.

The namespace boundary¶

Inside a namespace that's already granted, kubelatch can't separate anything: whoever creates pods reaches its Secrets and its ServiceAccounts. That's why developer, debugger and admin require Pod Security Admission, and why the system namespaces accept no permissions at all. System roles explains the boundary and its consequences.

Optional hardening: bindings to kubelatch-*¶

A namespace admin, whether granted by kubelatch or native, can create a RoleBinding to kubelatch-developer for whichever subject they want. That binding doesn't grant access through kubelatch, because the proxy only impersonates whoever has valid permissions. But a subject with a native credential would gain that role natively, and the binding confuses the reconciler, which logs bindings it doesn't own under names it manages.

The chart ships another ValidatingAdmissionPolicy (Kubernetes 1.30+), separate from the reconciler's jail, that only lets the kubelatch-reconciler ServiceAccount create or change bindings to a ClusterRole whose name starts with kubelatch-; namespaced Roles stay free. It's applied on each managed cluster, with that cluster's own kubeconfig. The policy also denies an administrator the bootstrap's own bindings, so moving the cluster to another instance or re-registering it requires removing it for a moment. How to install it: Harden the cluster.

Cluster tokens, encrypted¶

The tokens for each cluster's two ServiceAccounts are the keys to the kingdom. kubelatch stores them encrypted with AES-256-GCM under KUBELATCH_ENCRYPTION_KEY and never displays them. Each token is bound to its cluster and to its role (proxy or reconciler), so an encrypted token is useless on another cluster or in the other role.

The enrollment command's link (kle_…) is a bearer secret: whoever has it can register tokens on that cluster. Its life is short on purpose: it works once, expires after an hour, dies after 10 attempts that fail validation, is replaced by any new link for the same cluster, and only works while the administrator who created it is still an enabled administrator. It travels in the URL, so a reverse proxy's access log in front of kubelatch may record it; kubelatch stores only its hash and its own logs never record it. The cluster's page shows a pending link's expiry and the reason of the last failed attempt, never the command, which only the tab that generated it holds; the reason has both tokens cut out. The tokens go in the body of a POST sent through stdin, never in a URL or a process's arguments.

The same key also signs session cookies and the state cookies for GitHub and OIDC login. If it's lost or rotated, everyone has to sign in again and the tokens for each cluster have to be rotated. Passwords and klt_ credentials don't depend on it.

Enrolling with an administrator's identity¶

When an administrator enrolls a cluster with their identity, kubelatch uses the ID token their provider gives them only in the request that brings them back, only against that enrollment's URL and CA, over verified TLS and without following redirects, and only to apply the bootstrap; it never stores it. The identity must be the one linked to their account, and the enrollment is tied to their browser, their session and their admin role, with one attempt within 10 minutes.

kubelatch never asks for a kubeconfig or any administrator credential of a cluster: only the API server's URL and CA, which aren't secrets. The API has nowhere to receive a kubeconfig, and one pasted by mistake is wiped without being sent. A web page can't defend itself from a browser extension allowed on it. If one worries you, there are two defences outside the page: a private window, where extensions don't run unless you allow them, and, in a company, the ExtensionSettings policy of Chrome and Edge with kubelatch's host in runtime_blocked_hosts ({"*": {"runtime_blocked_hosts": ["*://kubelatch.example.com"]}}), which keeps every extension off kubelatch. That policy also protects the administrator's session, not only this screen.

The chart mounts the cluster's kube-root-ca.crt ConfigMap, which holds only its CA: it grants no permission, and the pod still runs without a ServiceAccount token.

Trusted proxies and the source IP¶

The audit log's source IP and the per-IP rate limiters (login, CLI sign-in, CI exchange and proxy authentication failures) use the TCP connection's address. Behind an Ingress or another L7 proxy, that address is the proxy's.

KUBELATCH_TRUSTED_PROXIES says which networks X-Forwarded-For is accepted from, read right to left up to the first untrusted hop. It only accepts CIDR: a bare IP prevents startup, and so does a prefix wider than /8 for IPv4 or /16 for IPv6.

• Don't put in ranges a client could reach: they could choose the IP that appears in the audit log and dodge the limits.
• Behind an Ingress, only trust the controller pods' IPs. A CIDR covering the whole cluster's pods turns every pod into a trusted proxy. That's why the chart, with exposure: ingress, ships a NetworkPolicy that only admits traffic from the controller's namespace, and needs a CNI that enforces NetworkPolicy.
• Behind an L4 LoadBalancer with externalTrafficPolicy: Local, leave it empty.

The rate limiters group IPv6 by /64, so rotating addresses within the prefix doesn't multiply the attempts. With two replicas, each one counts its per-IP limits in memory; the account lockout after 5 failures is global.

Passwords and sessions¶

• Passwords from 12 to 128 characters, different from the login, with no composition rules, hashed with argon2id.
• 5 consecutive failures lock the account for 15 minutes, and each IP gets 10 attempts per minute. The response is the same for a nonexistent login, a wrong password or a disabled account, and it takes the same time.
• Every login attempt, successful or not, is logged in Audit → Control plane with the IP. The exception is a login longer than 64 characters: it gets the same 401 without reaching the database or being logged.
• Changing your own password in My account has its own limit: 10 attempts per minute per account, and every failure is logged.
• 12 h sessions in a signed cookie, revocable all at once. API mutations are protected against cross-origin requests.
• The web UI and the API answer with a strict Content-Security-Policy (scripts only from kubelatch itself), X-Frame-Options: DENY, X-Content-Type-Options: nosniff and Referrer-Policy: same-origin, so no other page, not even one on a sibling subdomain, can frame a screen such as the CLI's device approval. With an https KUBELATCH_BASE_URL, Strict-Transport-Security is added too. Responses under /clusters/ keep the cluster's own headers.

With an https base URL, the session and state cookies carry the __Host- prefix (__Host-kubelatch_session, __Host-kubelatch_github_state, __Host-kubelatch_oidc_state): Secure, Path=/ and no Domain, so a sibling host under the same domain can't plant a session of its own in your browser, for example on the CLI's approval page. A cookie without the prefix is ignored.

After 5 failures, the 429 confirms that the login exists, and anyone can lock an account for 15 minutes. This is accepted for an internal tool; the per-IP limit makes it costlier.

Identity with GitHub¶

With the GitHub App configured, people's identity is GitHub's. The flow is described in Identity; here's what it means for security:

• Two-factor. kubelatch doesn't implement it: it relies on whatever the organization enforces. It can only check this if the App has Organization → Administration: read; without that permission, it warns and trusts. Turn on mandatory 2FA in GitHub before anything else.
• Passwords. With GitHub active, only break-glass accounts sign in with a password, and they have no two-factor. Keep them few, with long passwords in a manager, and use them only when GitHub isn't available. The hashes for other accounts stay in the database but can't be used to sign in.
• Offboarding. Whoever leaves the organization gets disabled automatically (signed webhook or hourly sync). The sync never disables on a GitHub error and never re-enables. The credentials' maximum TTL limits what a prolonged failure of both paths could leave open.
• The OAuth flow. A random state in a signed cookie valid for 10 minutes. The code is exchanged server-side with the client secret, and the person's GitHub token is discarded after login. PKCE isn't used: GitHub doesn't document it for GitHub Apps, and kubelatch is a confidential client, for which state plus client secret is what the OAuth 2.0 BCP requires. The App's installation tokens live only in memory.
• Webhook with no deduplication. A signed, repeated delivery can disable someone an admin re-enabled after they rejoined the organization. The account stays disabled until an admin re-enables it again, with new credentials because the old ones were revoked. Guard GITHUB_WEBHOOK_SECRET like any secret and rotate it if it leaks. If someone who rejoined shows up disabled, check Users and the user.disable events with via: github-webhook.

Identity with an OIDC provider¶

With an OIDC provider configured, people's identity is the provider's: its issuer, the name of the claim that identifies them and that claim's value (Identity). Here's what that means for security:

• What a sign-in checks. The authorization code flow, always with PKCE (S256), also for a confidential client, and a random state in a signed cookie valid for 10 minutes. The id_token is accepted only signed with RS256 by a key from the issuer's own JWKS: HS256, none and elliptic curves are refused before any key is looked up. Its iss must be the configured issuer (the one exception is Google's accounts.google.com, without the scheme), aud must hold the client id, azp must equal it whenever present and must be there when aud lists several audiences, exp is required, iat and nbf get 30 s of leeway for clock skew, and its nonce must be the one kubelatch sent.
• The e-mail counts only when the provider vouches for it. The verified claim (email_verified, or the one OIDC_EMAIL_VERIFIED_CLAIM names) must be true, and the domain must be exactly one of OIDC_ALLOWED_DOMAINS. With Google, the hd claim must also be there, be allowed and match the e-mail's domain: an account without it is a personal Gmail one. When the token lacks the e-mail or the verified claim (Okta's, in the authorization code flow, carries email without email_verified), kubelatch asks userinfo once, believes the answer only if its sub is the token's, and then takes the e-mail and its verification from that answer alone. A verified claim that says false is refused without asking.
• Linking by e-mail. Only on a first sign-in, only onto a single enabled account that has no identity of this provider, and only as OIDC_AUTOLINK allows. An identity of the same provider is never replaced by e-mail. Each link leaves an oidc.link event, revokes the account's pending links and shows up in Notices. The remaining risk, an account never linked whose e-mail address someone else gets in the provider, is covered by that notice, the invited policy and disabling people the day they leave.
• Passwords. As with GitHub, only break-glass accounts sign in with a password; they're also the way in when the provider is down.
• Offboarding. The provider gives no member list, so kubelatch disables no one on its own. Disabling a person in the provider stops their next sign-in, but what they already hold lives on until it expires: the browser session up to 12 h, the CLI session (KUBELATCH_CLI_SESSION_TTL, 12 h by default) and each credential (up to KUBELATCH_MAX_TTL_USER, 30 days by default). Disable in Users cuts all of it at once: do it the same day.
• Two-factor. Whatever the provider enforces. kubelatch has none of its own, and records the provider's amr and acr, when it sends them, in each login.success.
• The provider's tokens. The id_token, the access_token, the authorization code, the PKCE verifier and the client secret never reach a log line, an event, a cookie or a response: the tokens only live while the callback request lasts. That matters with a public client whose client id a cluster also trusts: there, that id_token would open the API server too.
• Calls to the provider. Only to the configured issuer and the https endpoints its discovery document names, without following redirects, with a 10 s timeout and a size cap on every answer. Each replica keeps the discovery document for 24 h and the signing keys for 1 h. While the provider doesn't answer, sign-ins through it fail with provider_unavailable; break-glass accounts still get in.

Without an external login¶

Without the GITHUB_* and OIDC_* variables, local login has neither two-factor nor automatic offboarding. A stolen password gives access to clusters with that person's permissions. Active mitigations: 12 h sessions, lockout on attempts and per-IP limit, credentials with an expiry, instant revocation and audit of every login. This is acceptable for a test or an environment without an identity provider. In production, configure the GitHub App or your OIDC provider and, meanwhile, disable the accounts of people who leave on the same day.

Secret-free CI credentials¶

A GitHub Actions workflow doesn't store any kubelatch secret: it exchanges an OIDC id_token that GitHub signs for that run (CI: trust rules). Verification is strict:

• Signature. RS256 only, with the public keys from <issuer>/.well-known/jwks (KUBELATCH_GITHUB_ACTIONS_ISSUER). kubelatch caches them for 1 h; an unknown key forces a refresh, at most once per minute. If GitHub doesn't respond, it keeps using the cached ones for up to 24 h after the last successful refresh. Past that window, all exchanges fail with 401.
• Claims. Exact iss, aud equal to KUBELATCH_BASE_URL, and exp, nbf and iat with a 30 s margin for clock skew.
• Single use. The jti is stored on exchange; a reused token is rejected with 401.
• Ids, not names. Trust rules compare repository_owner_id, repository_id, ref and environment. kubelatch doesn't interpret sub: it accepts both the classic repo:owner/name:… format and the immutable-id one.

Any token failure responds with the same 401 invalid token; the reason only appears in the log and in the ci.exchange.failure event.

The license key¶

A Pro key is a signed token (Ed25519) that kubelatch verifies against the public keys built into its binary, at startup and every hour. The check calls no service and sends nothing out of your instance, so air-gapped installations work. The only call kubelatch makes to Picaporte Labs is the daily version check, which fetches a small JSON document and sends nothing, and shows administrators only a notice when a newer version is out (Know when a new version is out); KUBELATCH_UPDATE_CHECK turns it off. A key pasted in the interface is stored encrypted with KUBELATCH_ENCRYPTION_KEY; if that key is lost or changed, the pasted key cannot be opened (If the encryption key is lost or changed). A key in KUBELATCH_LICENSE is read from the environment. A key that does not verify, or cannot be opened, leaves kubelatch in Free with a warning; it never stops it. A limit refuses a new registration and nothing else: the proxy, permissions and credentials never degrade because of a key.

AI agents¶

The MCP server adds no second way to the clusters: every tool call goes through the proxy, with the same impersonation, the same fail-closed audit row and the cluster's RBAC deciding. On top of that:

• /mcp accepts only a bot's own credentials and the access tokens of delegated agents; those tokens are valid nowhere else. A person's credential can't be handed to an agent at /mcp.
• A delegated agent never gets more than its person: each of its permissions derives from one of theirs, as the same role or viewer, never cluster-admin, and ends with it.
• The consent and every approval need a person's web session and a click; no link answers them.
• An approved write runs the stored request once, with a precondition; its body is encrypted at rest while it waits.
• Secret values reach an agent only through read_secret, and each read is an audit row.

The limits (content that carries instructions, a value read that can't be taken back, other ways in on the agent's machine) are in AI agents.

Immutable audit log¶

kubelatch keeps two records in Postgres: one row per request to the proxy, written before forwarding it and completed when it finishes, and one event per control-plane action (creations, permissions, credentials, revocations, account changes, trust rules, every login attempt), written in the same transaction as the action.

The database itself prevents tampering: a request row can only be completed once, control-plane events are never modified, and neither record can be truncated. Only the periodic retention job deletes, and only rows older than KUBELATCH_AUDIT_RETENTION (90 days by default, 24 hours at most if the instance never had a key; see Audit and retention). Deleting a cluster doesn't delete its history, and accounts are never deleted, only disabled.

The cluster's native audit log, a second source¶

Keep each cluster's native audit log active. It records what doesn't pass through kubelatch (native access, ServiceAccount tokens used from pods) and is an independent second source. The API server uses kubelatch's Audit-ID as its own auditID, so a kubelatch row and its native event can be found through the same identifier.

Personal data¶

The audit log stores, per request, the IP, the login, the User-Agent and the query string; in an exec, the query contains the command the person ran. Control-plane events store the actor and IP of every administrative action and the login and IP of every login attempt. Purpose: security and traceability of access to clusters. Retention: KUBELATCH_AUDIT_RETENTION, 90 days by default (24 hours at most if the instance never had a key). Audit and retention covers how to look things up and delete them.

What kubelatch doesn't cover¶

Session recording, ingesting a cluster's native audit log, a relay for private clusters, namespace and quota management, kubectl --as, SAML, more than one OIDC provider at a time, the identity provider's groups, provisioning or offboarding from it (SCIM), per-credential rate limiting on the proxy, and an agent's exec, port-forward, watch or log follow. It also doesn't replace native emergency access to each cluster, which must exist and be tested (see Accounts and recovery).