Install¶

This guide gets kubelatch running on a management cluster, with Postgres, TLS and the first administrator.

There's a test, make e2e-install (e2e/install.sh), that repeats these steps on a disposable kind cluster: use it as a working example.

Before you start¶

All the commands in this guide point at the management cluster with this variable. Set it before you start:

export MGMT_KUBECONFIG=<path-to-the-management-cluster-kubeconfig>

What deploy/k8s brings¶

The manifests are kustomize. The base (deploy/k8s/base) creates:

• The Namespace kubelatch, with restricted Pod Security Admission.
• A ServiceAccount with no token mounted: kubelatch doesn't talk to the management cluster's API server with it.
• A Deployment with one replica: maxSurge: 1, maxUnavailable: 0, terminationGracePeriodSeconds: 60, non-root user, read-only filesystem and no capabilities.
• A LoadBalancer Service with externalTrafficPolicy: Local and per-cloud annotations commented out.
• The ConfigMap kubelatch-config, generated from config.env with a hash in its name: changing it redeploys.

You create the secrets and the certificate yourself. On top of the base there are three overlays:

Install step by step¶

1. Edit deploy/k8s/base/config.env. At a minimum KUBELATCH_BASE_URL: the exact public URL, with no trailing slash. The rest of the variables are in Configuration. Don't add the GITHUB_* variables yet: GitHub login is turned on later, following GitHub login.

2. Create the namespace and the secrets.

   kubectl --kubeconfig "$MGMT_KUBECONFIG" apply -f deploy/k8s/base/namespace.yaml
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create secret generic kubelatch-secrets \
     --from-literal=DATABASE_URL='postgres://kubelatch:<password>@<host>:5432/kubelatch?sslmode=require' \
     --from-literal=KUBELATCH_ENCRYPTION_KEY="$(head -c32 /dev/urandom | base64)"
   

   Save KUBELATCH_ENCRYPTION_KEY in your secret manager, apart from the Postgres backups. It encrypts each cluster's ServiceAccount tokens and signs sessions. If you lose it, see Upgrades and backups.

3. Put the certificate in place and apply the manifests. Do you have to go through an Ingress? Read Exposing kubelatch first and apply overlays/ingress. With cert-manager, edit deploy/k8s/overlays/cert-manager/certificate.yaml (DNS name and issuerRef) and apply the overlay:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" apply -k deploy/k8s/overlays/cert-manager
   

   Without cert-manager, create the Secret by hand and apply the base:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create secret tls kubelatch-tls --cert=tls.crt --key=tls.key
   kubectl --kubeconfig "$MGMT_KUBECONFIG" apply -k deploy/k8s/base
   

4. Check that it starts up.

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch logs deploy/kubelatch | head
   curl https://kubelatch.example.com/readyz
   

   The log shows listening with version, tls=true and audit_retention. /readyz responds ok.

The probes work like this: the startupProbe gives migrations up to two minutes. /readyz pings Postgres. /healthz only checks the process: a Postgres outage doesn't restart the pod or cut open sessions, and the proxy responds 503 in the meantime.

Create the first administrator¶

The image has no shell: kubectl exec runs the binary directly, with the same variables as the server.

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch exec -it deploy/kubelatch -- /kubelatch user create admin --admin --display-name "Admin"

It asks for the password twice, with no echo (12 characters minimum). With no terminal, pipe it through standard input:

printf '%s' '<long-password>' | kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch exec -i deploy/kubelatch -- /kubelatch user create admin --admin --password-stdin

If you're going to turn on GitHub login, mark this account as a break-glass account now (--break-glass). Accounts and recovery explains why.

Check that it works¶

1. Open https://kubelatch.example.com and sign in as admin.
2. Register a cluster in Clusters: Register a cluster.
3. Invite people in Usuarios (Users) (Users and bots) and grant them tiers in Permisos (Permissions) (Grant permissions).
4. Each person signs in to Inicio (Home) and issues their credential: Get a credential.

Exposing kubelatch: L4 LoadBalancer or Ingress¶

kubelatch carries long, unusual connections: watch and logs -f running for hours with almost no traffic, exec/attach/port-forward as SPDY or WebSocket upgrades over HTTP/1.1, and multi-megabyte applys. Any intermediary with timeouts or application buffers breaks them. That's why the default option is L4.

L4 LoadBalancer (default)¶

The load balancer passes the TLS bytes through as-is and kubelatch terminates TLS. The source IP arrives intact (audit logging and per-IP limits with no X-Forwarded-For needed), with no application timeouts or size limits. Only the load balancer's TCP idle timeout can cut a connection. Go sends TCP keepalives every 15 s, which all three clouds count as activity.

With externalTrafficPolicy: Local only nodes with a kubelatch pod pass the health check. That's what you want. Leave KUBELATCH_TRUSTED_PROXIES empty.

Ingress (alternative)¶

If your organization requires going through an Ingress, use deploy/k8s/overlays/ingress. It's built for ingress-nginx and the install test doesn't cover it: validate it on your cluster. TLS terminates at the controller (Secret kubelatch-tls in the kubelatch namespace) and kubelatch listens on plain HTTP on :8080.

ingress.yaml brings the three required settings:

1. proxy-read-timeout and proxy-send-timeout set to 3600. The 60 s default cuts watch, logs -f and an idle exec.
2. proxy-body-size: "0". With the 1 MiB default, a large kubectl apply gets a 413 before it reaches kubelatch.
3. HTTP/1.1 backend (backend-protocol: HTTP). Never HTTP/2 or gRPC toward the backend: SPDY/WebSocket upgrades travel over HTTP/1.1.

You also have to adjust three things in the overlay:

• Its own config.env. overlays/ingress/config.env entirely replaces the base one: put KUBELATCH_BASE_URL and the rest of the variables you use there.
• KUBELATCH_TRUSTED_PROXIES in overlays/ingress/config.env: the controller pods' IPs and nothing else (the narrowest range that covers them, or the nodes' CIDR if it runs with hostNetwork). The example value, 10.244.0.0/16, is the pod CIDR for an entire kind cluster: it's not right for production. Details in Trusted proxies.
• The NetworkPolicy (networkpolicy.yaml) only lets traffic reach :8080 from the ingress-nginx namespace. Change kubernetes.io/metadata.name if your controller lives elsewhere. Egress isn't restricted: kubelatch needs Postgres, the clusters and GitHub.

Other controllers (Traefik, HAProxy, AWS ALB) have equivalent settings. Whichever you use, ingress-nginx included, run three tests before signing off on it: kubectl get pods -w for more than 2 minutes, a kubectl exec left idle for 5 minutes, and a 2 MB kubectl create -f.

Trusted proxies and the source IP¶

The audit log, the control-plane log and the three per-IP rate limiters (login, CI exchange and proxy authentication failures) use the TCP connection's IP. Behind an L7 proxy that IP is the proxy's. KUBELATCH_TRUSTED_PROXIES says which networks X-Forwarded-For is trusted from:

• It's read right to left. The first hop that isn't trusted is the client; anything a client spoofs ends up to its left and is ignored.
• Only CIDRs are accepted (10.244.0.0/16, 10.0.0.1/32). A bare IP prevents startup.
• A prefix wider than /8 in IPv4 or /16 in IPv6 also prevents startup.
• IPv4 addresses mapped into IPv6 (::ffff:10.0.0.1) are compared unmapped.

Never set a range that reaches a client (for example 0.0.0.0/0): it could pick its own IP in the audit log and dodge the limits. If it's missing behind an Ingress, everything shows up as coming from the controller and the login limit applies to everyone at once.

Certificates and a private CA¶

kubelatch reads the certificate from /etc/kubelatch/tls and reloads the pair when it changes on disk: it checks the dates at most every 10 s. A cert-manager renewal doesn't need a restart. If the new pair isn't valid (a key from another certificate, a half-written file), it keeps serving the previous one and logs it (tls: reload failed). Every hour it warns if fewer than 30 days remain.

With a public CA (Let's Encrypt), kubeconfigs don't carry a CA: kubectl trusts the system store. With a private CA, point KUBELATCH_KUBECONFIG_CA at the PEM bundle and every issued kubeconfig will carry certificate-authority-data. With a private cert-manager issuer, the Secret already has ca.crt: use /etc/kubelatch/tls/ca.crt.

The file must be a regular file, at most 1 MiB, and contain only CERTIFICATE blocks. A key pasted in by mistake or another PEM block prevents startup.

kubelatch never puts a managed cluster's CA into kubeconfigs. kubectl verifies kubelatch, and kubelatch verifies each API server with the CA that was pasted in when it was registered.

Next steps¶

• GitHub login, if your organization uses GitHub.
• Register a cluster and, optionally, harden it.
• Upgrades and backups, before going to production.