Skip to content

Why a proxy

kubelatch puts itself in the path of every request to the API servers. This page explains what other ways of handing out credentials were studied, why they were discarded, and what the chosen approach costs.

The problem

What kubelatch has to solve at the same time:

  • Per-person and per-pipeline credentials on mixed clusters: managed EKS, GKE and AKS, and self-managed kubeadm, k3s, RKE2 or Talos.
  • Tiered permissions with standard RBAC.
  • An inventory of credentials: whose each one is, when it expires, when it was last used.
  • Per-request audit (verb, resource, namespace, outcome) the same across every cluster.
  • Short-lived credentials for CI with no static secrets.

The condition that narrows the options the most is the first one. On a managed cluster you don't control the API server's flags, so any mechanism that needs them doesn't work for the whole fleet.

What was discarded

Alternative Why not
X.509 certificates (kube-apiserver-client CSR) Kubernetes doesn't revoke certificates: one that's been issued is valid until it expires. Also, EKS approves the CSR but doesn't issue the certificate.
OIDC on the API server Requires API server flags or configuration: only viable on self-managed clusters. GKE doesn't support a custom OIDC provider and it's in preview on AKS. And GitHub isn't an OIDC provider for people (its only OIDC issuer is the Actions one), so another component like Dex would have to be added.
Authentication or authorization webhooks Only configurable on self-managed clusters.
Per-cloud exec plugins (IAM, gcloud, Entra) Give each provider's cloud identity, not a credential that kubelatch issues, inventories and revokes. Three providers, three inventories.
Native tokens without a proxy (one ServiceAccount per person and TokenRequest tokens) Work in every environment and revoke in seconds. But per-request audit would end up in each cloud's native audit log: three formats, minutes of delay, cost per GB and, on GKE, no reads by default. And a token's last use isn't known without reading those logs.
Existing tools (Teleport, Paralus, Pinniped, Rancher, Tailscale's proxy…) None of the free ones combine a live credential inventory, per-request audit and first-class CI credentials. The ones that cover it are heavy platforms, with shifting licenses or pricing of a different order. Teleport Community Edition was the reasonable alternative to building this.

What was chosen: impersonation

Only two mechanisms work the same way across all four environments without touching the API server: TokenRequest tokens and impersonation. Impersonation is acting on behalf of another identity. It's pure API server logic plus RBAC: a ServiceAccount with the impersonate verb sends Impersonate-User and Impersonate-Group, and the API server applies the RBAC of the impersonated identity.

With a proxy that uses impersonation:

  • One single path for every cluster. It only needs standard RBAC and one ServiceAccount per cluster.
  • Its own credentials. kubelatch issues and validates the token. Revoking it takes effect on the next request, and the inventory knows the real last-used time because every request passes through it.
  • Unified audit, in real time and with no per-GB cost. The proxy logs every request with the same schema on every cluster.
  • Correlation with the cluster's native audit log. The API server reuses the proxy's Audit-ID as its own auditID, and records the person as impersonatedUser.
  • Native RBAC. The cluster still decides every verb; kubelatch only says who the person is and which groups they're in.

It's the same pattern used by Pinniped (on managed clusters), Tailscale, NetBird, OpenUnison and StrongDM. It's not an exotic design: it's the only portable primitive that puts the application in the path of every request.

Proof that the pattern holds up under real use was done before kubelatch was written: exec, cp, port-forward (over WebSocket and over SPDY), logs -f, a watch lasting more than two minutes, and an object over 1 MB, all over TLS and through the proxy.

What it costs

Putting a proxy in the path has a price, and the design accepts it:

  • The proxy is on the critical path. If kubelatch goes down, people and CI lose access through it (workloads don't). Mitigation: two replicas, zero-downtime rollouts and native emergency access to each cluster.
  • Long-lived streams need care. A dedicated HTTP/1.1 transport is needed for exec and port-forward, unbuffered forwarding for watch and logs -f, and no read or write timeouts. Intermediaries (load balancers, Ingress) need high timeouts and no body-size limit.
  • There are traps in impersonation. The Kubernetes client will pass through a request that already carries Impersonate-User; that's why the proxy rejects any incoming Impersonate-* and kubectl --as isn't supported. And a credential with no permissions would already be an authenticated user to the API server, so the proxy blocks it earlier.
  • The namespace boundary is still weak. The proxy doesn't change what RBAC allows inside a namespace (see Tiers).
  • Private clusters have to be reachable from kubelatch. There's no built-in relay: tunnels are documented instead (see Private clusters).

What was deliberately left out

kubelatch's simplicity comes down to three boundaries: identity stays with GitHub, authorization stays with native RBAC, and deep audit stays with the API server. That's why kubelatch has no CA of its own, no identity provider of its own, no relay, and no session recording. That's exactly what has made the alternatives heavy.

Native TokenRequest tokens remain a possible future escape hatch for whatever shouldn't go through the proxy.

To learn more

The full research is in the repository, outside this site: docs/research/informe-credenciales-kubernetes-rbac-auditoria.md (the report, with its sources), the notes in docs/research/notas/ (native mechanisms, existing tools, per-environment audit, impersonation proxy, RBAC design, identity and CI), and the results of the proxy spike in docs/research/spike-proxy-resultados.md.