Install¶

This guide installs kubelatch on a management cluster with its Helm chart, with Postgres, TLS, the encryption key and the first administrator.

Every release is tested by repeating these steps on a disposable kind cluster, with the database in CloudNativePG.

Before you start¶

Every command in this guide uses two variables: the management cluster's kubeconfig and the version to install. Set them before you start:

export MGMT_KUBECONFIG=<path-to-the-management-cluster-kubeconfig>
export KUBELATCH_VERSION=<version>   # without the v: 1.2.3 (https://github.com/picaportelabs/kubelatch/releases)

What the chart installs¶

The chart is oci://ghcr.io/picaportelabs/charts/kubelatch. Its version is kubelatch's version, and it installs the image ghcr.io/picaportelabs/kubelatch of that same version. From the release after 0.43.0, both are signed: Verify a release. In the release's namespace it creates:

• A ServiceAccount with no permissions and no token mounted: kubelatch doesn't talk to the management cluster's API server with it.
• A Deployment with one replica: maxSurge: 1, maxUnavailable: 0, terminationGracePeriodSeconds: 60, non-root user, read-only filesystem and no capabilities.
• The ConfigMap kubelatch-config, from the config values. The pods carry its hash: changing it redeploys.
• A LoadBalancer Service on port 443 with externalTrafficPolicy: Local.
• A NetworkPolicy that only lets traffic into kubelatch's port.

The pod mounts, read-only, the kube-root-ca.crt ConfigMap that Kubernetes publishes in every namespace: the CA of the cluster kubelatch runs in, so enrolling that cluster with your identity needs nothing pasted. It grants no permission, and the pod still has no ServiceAccount token. clusterCA.mount: false turns it off.

Depending on the values it also creates an Ingress, a cert-manager Certificate, a PodDisruptionBudget, the generated encryption key, a CloudNativePG database and the hardening policy. All the values are in Helm values, and helm show values oci://ghcr.io/picaportelabs/charts/kubelatch --version "$KUBELATCH_VERSION" prints them with their comments.

By default it doesn't create the namespace, and it never creates the Secrets you bring: the encryption key, DATABASE_URL and the certificate.

Install step by step¶

1. Create the namespace with the restricted Pod Security level: kubelatch runs under it, and nothing else belongs in this namespace.

   kubectl --kubeconfig "$MGMT_KUBECONFIG" create namespace kubelatch
   kubectl --kubeconfig "$MGMT_KUBECONFIG" label namespace kubelatch pod-security.kubernetes.io/enforce=restricted
   

   With Helm 3.17 or later, the chart can create it instead: namespace.create: true in your values and --create-namespace --take-ownership in step 5. The namespace then carries the label and stays after helm uninstall. Older versions can't install that option: Helm needs the namespace to store the release before creating anything, and without --take-ownership it refuses to adopt the one --create-namespace made.

2. Create the secrets. The chart loads the kubelatch-secrets Secret whole (secrets.existingSecret):

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create secret generic kubelatch-secrets \
     --from-literal=DATABASE_URL='postgres://kubelatch:<password>@<host>:5432/kubelatch?sslmode=require' \
     --from-literal=KUBELATCH_ENCRYPTION_KEY="$(head -c32 /dev/urandom | base64)"
   

   sslmode=require encrypts the connection but doesn't check the server's certificate. Use sslmode=verify-full, which also checks it and the host name. If the certificate comes from a public CA, that's all. With a private CA (RDS, Cloud SQL…), put the CA in a Secret, mount it with extraVolumes and extraVolumeMounts in your values (step 4), and name it with sslrootcert in DATABASE_URL:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create secret generic kubelatch-pg-ca --from-file=ca.crt=<ca-bundle>.pem
   

   extraVolumes:
     - name: pg-ca
       secret:
         secretName: kubelatch-pg-ca
   extraVolumeMounts:
     - name: pg-ca
       mountPath: /etc/kubelatch/pg-ca
       readOnly: true
   

   DATABASE_URL=postgres://kubelatch:<password>@<host>:5432/kubelatch?sslmode=verify-full&sslrootcert=/etc/kubelatch/pg-ca/ca.crt
   

   kubelatch re-reads that file when it changes, so rotating the CA only means updating the Secret, with no restart. That works because the Secret is mounted as a directory: with subPath the kubelet never updates the file.

   With the database in CloudNativePG, leave DATABASE_URL out: it comes from the operator's Secret, and one in kubelatch-secrets is ignored. Save KUBELATCH_ENCRYPTION_KEY in your secret manager, apart from the Postgres backups: see The encryption key.

3. Put the certificate in place. With your own certificate:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create secret tls kubelatch-tls --cert=tls.crt --key=tls.key
   

   With cert-manager, the chart creates a Certificate that fills kubelatch-tls (next step). If you must go through an Ingress, read Exposing kubelatch first.

4. Write your values in a values.yaml. At a minimum config.baseURL: the exact public URL, with no trailing slash. Every value, with its default, is in Helm values, and what each kubelatch variable does is in Configuration. Don't turn on GitHub or OIDC login yet: that comes later, following GitHub login or OIDC login (SSO). The public repository has two complete examples, with the L4 LoadBalancer and with an Ingress, in examples/helm.

   config:
     baseURL: https://kubelatch.example.com
   # With cert-manager instead of your own kubelatch-tls Secret:
   # tls:
   #   certManager:
   #     enabled: true
   #     issuerRef: {name: letsencrypt, kind: ClusterIssuer}
   # With the database in CloudNativePG (install the operator first):
   # postgres:
   #   mode: cnpg
   

5. Install the chart. Keep the release name kubelatch: the resources are then called kubelatch, as in this documentation (with another name they are <release>-kubelatch).

   helm --kubeconfig "$MGMT_KUBECONFIG" install kubelatch oci://ghcr.io/picaportelabs/charts/kubelatch \
     --version "$KUBELATCH_VERSION" -n kubelatch -f values.yaml
   

   If a value is missing or doesn't fit (no config.baseURL, an Ingress without a host, postgres.mode: cnpg without the operator…), helm install fails with a message that names it and installs nothing.

6. Check that it starts up.

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch logs deploy/kubelatch | head
   curl https://kubelatch.example.com/readyz
   

   The log shows a listening line with version, tls and audit_retention. /readyz responds ok. With CloudNativePG, the pod waits for the operator's Secrets and restarts until the database is ready: wait for the database first with kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch wait --for=condition=Ready clusters.postgresql.cnpg.io/kubelatch-db --timeout=5m.

The probes work like this: the startupProbe gives migrations up to two minutes. /readyz pings Postgres. /healthz only checks the process: a Postgres outage doesn't restart the pod or cut open sessions, and the proxy responds 503 in the meantime.

The encryption key¶

KUBELATCH_ENCRYPTION_KEY encrypts each cluster's ServiceAccount tokens and signs sessions. It isn't in Postgres: if you lose it, a restored database can't use the clusters' tokens (If the encryption key is lost or changed). The chart takes it from one of two places:

To copy the generated key:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get secret kubelatch-encryption-key -o jsonpath='{.data.encryption-key}' | base64 -d

Once kubelatch runs with a key, keep that key: turning on secrets.generateEncryptionKey later would give the pod a new one, and every cluster's tokens would have to be rotated. If kubelatch-secrets already has KUBELATCH_ENCRYPTION_KEY, the chart refuses to generate one (Troubleshooting). A key supplied through extraEnvFrom is not detected: the generated key would silently take its place, so never combine the two.

Create the first administrator¶

The image has no shell: kubectl exec runs the binary directly, with the same variables as the server.

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch exec -it deploy/kubelatch -- /kubelatch-server user create admin --admin --display-name "Admin"

It asks for the password twice, with no echo (12 characters minimum). With no terminal, pipe it through standard input:

printf '%s' '<long-password>' | kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch exec -i deploy/kubelatch -- /kubelatch-server user create admin --admin --password-stdin

If you're going to turn on GitHub or OIDC login, mark this account as a break-glass account now (--break-glass). Accounts and recovery explains why.

Check that it works¶

1. Open https://kubelatch.example.com and sign in as admin.
2. Register a cluster in Clusters: Register a cluster.
3. Invite people in Users (Users and bots) and grant them tiers in Permissions (Grant permissions).
4. Each person signs in to Home and issues their credential: Get a credential.

Exposing kubelatch: L4 LoadBalancer or Ingress¶

kubelatch carries long, unusual connections: watch and logs -f running for hours with almost no traffic, exec/attach/port-forward as SPDY or WebSocket upgrades over HTTP/1.1, and multi-megabyte applys. Any intermediary with timeouts or application buffers breaks them. That's why the default option (exposure: loadBalancer) is L4.

L4 LoadBalancer (default)¶

The load balancer passes the TLS bytes through as-is and kubelatch terminates TLS on port 8443 with the kubelatch-tls certificate. The source IP arrives intact (audit logging and per-IP limits with no X-Forwarded-For needed), with no application timeouts or size limits. Only the load balancer's TCP idle timeout can cut a connection. kubelatch sends TCP keepalives every 15 s, which all three clouds count as activity. Each cloud's settings go in service.annotations:

With externalTrafficPolicy: Local only nodes with a kubelatch pod pass the health check. That's what you want. Leave config.trustedProxies empty.

Ingress (alternative)¶

If your organization requires going through an Ingress, use exposure: ingress. It's built for ingress-nginx and isn't part of the release tests: validate it on your cluster. TLS terminates at the controller (the kubelatch-tls Secret, or cert-manager with tls.certManager.enabled) and kubelatch listens on plain HTTP on :8080, behind a ClusterIP Service.

exposure: ingress
config:
  baseURL: https://kubelatch.example.com
  trustedProxies: 10.244.0.0/16   # the controller pods' IPs: narrow it for your cluster
ingress:
  host: kubelatch.example.com
  className: nginx
networkPolicy:
  ingressControllerNamespace: ingress-nginx

The chart puts the three settings the proxy needs on the Ingress:

1. proxy-read-timeout and proxy-send-timeout set to 3600. The 60 s default cuts watch, logs -f and an idle exec.
2. proxy-body-size: "0". With the 1 MiB default, a large kubectl apply gets a 413 before it reaches kubelatch.
3. HTTP/1.1 backend (backend-protocol: HTTP). Never HTTP/2 or gRPC toward the backend: SPDY/WebSocket upgrades travel over HTTP/1.1.

It also turns off response buffering (proxy-buffering: "off"). ingress.annotations adds your own (for example cert-manager.io/cluster-issuer).

You have to adjust these:

• ingress.host: the host of config.baseURL, exactly. Otherwise the chart stops: kubelatch only knows one public host.
• The certificate: either tls.certManager.enabled (the chart's Certificate) or a cert-manager.io/cluster-issuer (or cert-manager.io/issuer) annotation in ingress.annotations, never both: with both, the chart stops.
• config.trustedProxies: the controller pods' IPs and nothing else (the narrowest range that covers them, or the nodes' CIDR if it runs with hostNetwork). The chart refuses to install without it. The example value, 10.244.0.0/16, is the pod CIDR for an entire kind cluster: it's not right for production. Details in Trusted proxies.
• networkPolicy.ingressControllerNamespace: the chart's NetworkPolicy only lets traffic reach :8080 from that namespace (ingress-nginx by default). Egress isn't restricted: kubelatch needs Postgres, the clusters, and GitHub or your identity provider. Keep networkPolicy.enabled on: with exposure: ingress and no NetworkPolicy, the chart stops (Troubleshooting).

Other controllers (Traefik, HAProxy, AWS ALB) have equivalent settings: ingress.className and ingress.annotations. Whichever you use, ingress-nginx included, run three tests before signing off on it: kubectl get pods -w for more than 2 minutes, a kubectl exec left idle for 5 minutes, and a 2 MB kubectl create -f.

kubelatch treats any Authorization header on /api as a CLI token, so a proxy in front that injects one (Ingress basic-auth, oauth2-proxy with pass-authorization-header) breaks the web UI: don't add it on kubelatch's host.

Trusted proxies and the source IP¶

The audit log, the control-plane log and the per-IP rate limiters (login, CLI sign-in, CI exchange and proxy authentication failures) use the TCP connection's IP. Behind an L7 proxy that IP is the proxy's. config.trustedProxies (KUBELATCH_TRUSTED_PROXIES) says which networks X-Forwarded-For is trusted from:

• It's read right to left. The first hop that isn't trusted is the client; anything a client spoofs ends up to its left and is ignored.
• Only CIDRs are accepted (10.244.0.0/16, 10.0.0.1/32). A bare IP prevents startup.
• A prefix wider than /8 in IPv4 or /16 in IPv6 also prevents startup.
• IPv4 addresses mapped into IPv6 (::ffff:10.0.0.1) are compared unmapped.

Never set a range that reaches a client (for example 0.0.0.0/0): it could pick its own IP in the audit log and dodge the limits. If it's missing behind an Ingress, everything shows up as coming from the controller and the login limit applies to everyone at once.

Certificates and a private CA¶

kubelatch reads the certificate from /etc/kubelatch/tls and reloads the pair when it changes on disk: it checks the dates at most every 10 s. A cert-manager renewal doesn't need a restart. If the new pair isn't valid (a key from another certificate, a half-written file), it keeps serving the previous one and logs it (tls: reload failed). Every hour it warns if fewer than 30 days remain.

With tls.certManager.enabled the chart's Certificate reproduces the recommended one: ECDSA P-256, 90 days, renewed 30 days before, for the host of config.baseURL (tls.certManager.dnsNames, duration and renewBefore change them).

With a public CA (Let's Encrypt), kubeconfigs don't carry a CA: kubectl trusts the system store. With a private CA, point config.kubeconfigCA (KUBELATCH_KUBECONFIG_CA) at the PEM bundle and every issued kubeconfig will carry certificate-authority-data. With a private cert-manager issuer, the Secret already has ca.crt: use /etc/kubelatch/tls/ca.crt (with exposure: loadBalancer only: behind an Ingress, kubelatch doesn't mount the certificate).

The file must be a regular file, at most 1 MiB, and contain only CERTIFICATE blocks. A key pasted in by mistake or another PEM block prevents startup.

kubelatch never puts a managed cluster's CA into kubeconfigs. kubectl verifies kubelatch, and kubelatch verifies each API server with the CA it was enrolled with.

Several replicas¶

Two replicas keep kubelatch serving while a node is drained or a pod restarts. Spread them over two nodes with affinity, since the chart doesn't do it on its own. Use the preferred form: with the required form on a cluster of two nodes, the extra pod of a rollout (maxSurge: 1) has nowhere to go.

replicas: 2
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: kubelatch
              app.kubernetes.io/instance: kubelatch

With replicas above 1 the chart adds a PodDisruptionBudget with pdb.minAvailable (1). The load balancer or the Ingress needs no session affinity: any replica answers any request.

What two replicas share, because it lives in Postgres or is signed with the encryption key both of them read:

• Sign-ins. A web session, a kubelatch login in the browser or with a device code, an agent's OAuth sign-in, a GitHub login or an OIDC login can start on one replica and finish on the other. The account lockout after 5 failed logins counts across both.
• Agent approvals. An agent's call can wait for the decision on one replica while an administrator approves it through the other. The call sees the decision within a second.
• Revocation. Revoking a credential or disabling an account cuts its open exec, port-forward and watch within 10 s, on whichever replica holds them.
• Background jobs. Each replica runs the reconciler, the audit retention and the GitHub sync, and a Postgres lock keeps two passes from overlapping. Reconcile answers that a pass is already running while the other replica's pass holds the cluster. Migrations run on one replica while the other waits.

What each replica keeps to itself:

What isn't supported:

• Two installations on the same tables. Two releases need their own database, or at least their own schema. Replicas of one release are supported.
• Moving a stream to the other replica. A rollout or a restarted pod cuts the exec, port-forward and watch it holds. kubectl reopens a watch on its own, on the replica still serving, but an exec or a port-forward has to be run again: see What happens during a rollout.

Next steps¶

• GitHub login, if your organization uses GitHub.
• OIDC login (SSO), if your company signs in with Google Workspace, Microsoft Entra ID, Okta or another identity provider.
• Register a cluster and, optionally, harden it.
• Database with CloudNativePG, if the chart runs your Postgres.
• Upgrades and backups, before going to production.