Upgrades and backups¶

How to roll out a new version without surprises, when to use two replicas, and what to back up so you can restore kubelatch. Also covers the encryption key and how to uninstall.

Upgrade kubelatch¶

The commands use $MGMT_KUBECONFIG, the path to the management cluster's kubeconfig (see Install).

A change to the image or to config.env (the ConfigMap's name carries a hash) triggers a rollout when applied. A change to a Secret doesn't: restart it yourself with kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout restart deploy/kubelatch.

1. Change the image under images: in deploy/k8s/base/kustomization.yaml, or whatever configuration needs it.
2. Give teams a heads-up if it's working hours: the rollout cuts open exec and port-forward sessions.

3. Apply with the same overlay you used to install:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" apply -k deploy/k8s/overlays/cert-manager
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch
   

4. Check the version in the log (listening with version=…) or with /kubelatch version:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch exec deploy/kubelatch -- /kubelatch version
   

What happens during a rollout¶

With maxSurge: 1 and maxUnavailable: 0, the new pod is Ready (migrations applied, Postgres reachable) before the old one receives SIGTERM. New requests never find nobody listening.

But a rollout cuts in-flight long-lived connections. On receiving SIGTERM, kubelatch:

1. Stops accepting connections.
2. Gives normal requests 3 s to finish.
3. Closes every open exec, attach, port-forward, watch, and logs -f, and completes their audit rows with kubelatch reiniciando (kubelatch restarting).

terminationGracePeriodSeconds: 60 leaves margin for all of that. kubectl only retries watches; an exec or a port-forward has to be relaunched.

Migrations¶

Migrations are applied on startup, with a Postgres lock: with two replicas, one migrates and the other waits. They're forward-only. To go back to an earlier version of the binary, restore the database from the backup taken before the upgrade.

Two replicas¶

kubectl --kubeconfig "$MGMT_KUBECONFIG" apply -k deploy/k8s/overlays/ha

The ha overlay sets two replicas and a PodDisruptionBudget with minAvailable: 1: a node drain always leaves one pod serving. The overlay builds on the base; if you use cert-manager or Ingress, combine their resources in your own kustomization.

All the state is in Postgres, so any replica can serve any client. The reconciler, the retention job, and the GitHub sync take turns using Postgres advisory locks.

What is not shared are the in-memory per-IP limiters (login, CI exchange, and proxy authentication failures). Each replica counts separately, so the effective limit doubles. The account lockout after 5 failures is global, though: it lives in the database.

Backups¶

You need two things, kept separately:

1. Take a daily pg_dump, or use your managed service's snapshots.
2. Keep the key in the secrets manager, separate from the backups.
3. Test a restore before you need one: restore into another Postgres and start a kubelatch pointing at it with the same key.

Postgres settings¶

• idle_in_transaction_session_timeout: the reconciler keeps a transaction open for an entire pass (up to 2 minutes of calls to the cluster, plus the write). If your Postgres has this timeout set, it must be above 3 minutes. Otherwise, long passes fail with terminating connection and the cluster stays in Error until the next one.
• Roles: today a single role does both migrations and traffic. The triggers that protect the audit log protect you from kubelatch bugs, not from whoever holds that role. Splitting a migrations role (owner of the tables) from a runtime role is a pending improvement. In the meantime, don't reuse kubelatch's role for anything else.
• Size: see Audit and retention.

If the encryption key is lost or changed¶

KUBELATCH_ENCRYPTION_KEY signs sessions and encrypts each cluster's ServiceAccount tokens. kubelatch only supports one key at a time: there's no rotation with two keys. Changing it (or losing it) has these effects:

• Everyone has to sign in again: previous cookies stop working.
• No cluster can be used until its tokens are pasted back in. In Clusters they show Error with vuelve a pegar los tokens del cluster (paste the cluster's tokens again), Namespaces and granting a permission respond 409 asking to paste the JSON again, and Reconciliar (Reconcile) responds 502 with the same reason shown under the status. kubectl gets a 503 with kubelatch no puede consultar su base de datos (kubelatch can't query its database); the log says proxy: load cluster.
• Unchanged: passwords (argon2id), klt_ credentials (SHA-256), permissions, and the audit log.

If you only lost it from the secrets manager, recover it from the Secret the pod loads and there's no need to rotate it:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get secret kubelatch-secrets -o jsonpath='{.data.KUBELATCH_ENCRYPTION_KEY}' | base64 -d

To change it on purpose, or if it's gone from everywhere:

1. Generate a new key: head -c32 /dev/urandom | base64.

2. Update KUBELATCH_ENCRYPTION_KEY in the kubelatch-secrets Secret, save it in the secrets manager, and restart with rollout restart: a Secret change doesn't redeploy on its own. Wait for it to finish before continuing: during the rollout, pods with the old key and the new one coexist, and tokens pasted midway might not decrypt on the other pod.

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout restart deploy/kubelatch
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch
   

3. For each cluster, in Clusters → Volver a pegar tokens (Paste tokens again): run the one-liner with that cluster's kubeconfig and paste the JSON. There's no need to re-apply the bootstrap. See Register a cluster.

4. Check that each cluster goes back to Listo (Ready).

Uninstall¶

1. Delete each cluster from Clusters, which cleans up its RBAC, then remove its bootstrap (Delete a cluster). If kubelatch can no longer reach it, clean everything up on the cluster itself:

   kubectl --kubeconfig <cluster-kubeconfig> delete -l app.kubernetes.io/managed-by=kubelatch clusterroles,clusterrolebindings,rolebindings -A
   kubectl --kubeconfig <cluster-kubeconfig> delete namespace kubelatch-system
   

   In both cases, if you applied the hardening policy, delete it too: it doesn't carry kubelatch's labels.

   kubectl --kubeconfig <cluster-kubeconfig> delete validatingadmissionpolicybinding,validatingadmissionpolicy kubelatch-bindings
   

2. Delete kubelatch from the management cluster (with -k pointing at the overlay you used, if it wasn't the base):

   kubectl --kubeconfig "$MGMT_KUBECONFIG" delete -k deploy/k8s/base
   

3. Delete the Postgres database, after exporting the audit log if you need it.