Upgrades and backups¶

How to roll out a new version without surprises, when to use two replicas, and what to back up so you can restore kubelatch. Also covers the encryption key and how to uninstall.

Know when a new version is out¶

Within 30 seconds of starting and then once a day, kubelatch reads https://docs.kubelatch.com/releases/latest.json, which names the latest published version; a failed attempt is retried an hour later. The request carries no identifier, no version and nothing about your instance (What we commit to). When that version is newer than yours, administrators see:

a notice in the bell, A new version of kubelatch is out: 0.31.0., with the link How to upgrade;
on Home, one more installation recommendation in the line under the title, which leads to the health list of Edition;
the Version row in Edition: 0.30.0 · 0.31.0 is out, with What's new, which opens that version's notes in Releases, and the same How to upgrade.

How to upgrade opens this page. The notice has no counter and can't be dismissed: it goes away when you upgrade. A build without a version (dev) never checks.

To check without waiting for the day, press Check now in the Version row. With two replicas, only the one that answers checks at that moment; the other keeps its daily check.

KUBELATCH_UPDATE_CHECK=false turns the check off, and Version then says update check off. In an air-gapped installation, KUBELATCH_UPDATE_CHECK_URL points it at an internal mirror of the document (Configuration).

Upgrade kubelatch¶

The commands use $MGMT_KUBECONFIG and $KUBELATCH_VERSION, as in Install, and the values.yaml you installed with.

Audit older than 24 hours

Since 0.27, an instance that never had a license key runs as Free and keeps its audit at most 24 hours (0.26 kept 7 days). About a minute after the first start of the new version, the retention job deletes the older audit. To keep it, have a key in place by then: set KUBELATCH_LICENSE before upgrading, or install the key in Edition right after. A trial key counts. See Editions and license.

Read the release notes of the new version and set KUBELATCH_VERSION to it. The chart's version is kubelatch's: the chart brings its image.
Give teams a heads-up if it's working hours: the rollout cuts open exec and port-forward sessions.

Upgrade with the same values:

helm --kubeconfig "$MGMT_KUBECONFIG" upgrade kubelatch oci://ghcr.io/picaportelabs/charts/kubelatch \
  --version "$KUBELATCH_VERSION" -n kubelatch -f values.yaml
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch

Check the version in the log (the listening line carries version) or with /kubelatch-server version:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch exec deploy/kubelatch -- /kubelatch-server version

If the release notes say the bootstrap manifest changed, update it on each cluster: Update the bootstrap. Until you do, a cluster keeps working with what its manifest allows.

A configuration change is the same helm upgrade with the edited values.yaml. What restarts the pods and what doesn't:

Change	Rollout
New version (image)	Yes.
Any value under `config`	Yes: the pods carry the hash of the `ConfigMap`.
`secrets.env`, with `secrets.create`	Yes: the pods carry the hash of that Secret.
A Secret of yours (`kubelatch-secrets`, `kubelatch-tls`…)	No. Restart with `kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout restart deploy/kubelatch`. A renewed certificate needs no restart: kubelatch reloads it.
`replicas`, `hardening`, `networkPolicy`, the Ingress	No: they don't touch the pods.

Never rename after installing

Never change the release name, nameOverride or fullnameOverride after installing. The chart derives the generated key Secret (<fullname>-encryption-key) and the database Cluster (<fullname>-db) from them. A new name means a new key and a new, empty database, and kubelatch starts on them without any error. The old objects are kept, but nothing uses them.

What happens during a rollout¶

With maxSurge: 1 and maxUnavailable: 0, the new pod is Ready (migrations applied, Postgres reachable) before the old one starts terminating. The old pod keeps serving for shutdownDelaySeconds (5 s by default, a preStop sleep) before it receives SIGTERM: kube-proxy and the load balancer need that long to stop sending it new connections. New requests never find nobody listening. If your load balancer takes longer to stop sending traffic to a target, raise shutdownDelaySeconds (at most 25).

But a rollout cuts in-flight long-lived connections. On receiving SIGTERM, kubelatch:

Stops accepting connections.
Gives normal requests 3 s to finish.
Closes every open exec, attach, port-forward, watch, and logs -f, and completes their audit rows with kubelatch restarting.

terminationGracePeriodSeconds: 60 leaves margin for the delay plus all of that. kubectl only retries watches; an exec or a port-forward has to be relaunched.

Signing in again after 0.19.0¶

Upgrading to 0.19.0 or later from an earlier version signs everyone out of the web interface once: the session cookie changed its name (it now carries the __Host- prefix). Credentials, CLI sessions and kubeconfigs are unaffected.

The hardening policy and CloudNativePG¶

Up to 0.59.0, the hardening policy also refuses bindings to namespaced Roles whose name starts with kubelatch-, and CloudNativePG calls the Role of the chart's database kubelatch-db. With hardening.enabled: true and postgres.mode: cnpg, the database keeps serving but the operator stops managing it: no changes to the Cluster, no updates, no failover.

The current policy reserves only ClusterRoles. A helm upgrade installs it on the management cluster. On a cluster where you applied it by hand, apply the file from the new chart again.

The operator picks up the Cluster again on its next retry, up to about 17 minutes later, and applies every change that was pending. A new imageName, for instance, restarts the database pods; with one instance, kubelatch is unavailable until the pod is back. To see whether an image is pending before you upgrade, compare these two:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get cluster kubelatch-db -o jsonpath='{.spec.imageName}{"\n"}'
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get pods -l cnpg.io/cluster=kubelatch-db -o jsonpath='{.items[*].spec.containers[0].image}{"\n"}'

To choose the moment instead of waiting for the retry, annotate the Cluster once the policy is in place, and the operator picks it up at once:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch annotate cluster kubelatch-db kubelatch.io/reconcile="$(date +%s)" --overwrite

Migrations¶

Migrations are applied on startup, with a Postgres lock: with two replicas, one migrates and the other waits. They're forward-only. To go back to an earlier version of the binary, restore the database from the backup taken before the upgrade (helm rollback only brings back the manifests).

Going back to a release without custom roles

The migrations that brought custom roles keep the previous release working on the migrated database, so a helm rollback alone brings it back. That release doesn't know custom roles: it would send a custom role's group without its role- prefix, and its reconciler would take those groups out of what kubelatch-proxy may impersonate. The API server refuses a request when any of its groups is refused, so everyone holding a custom role would get 403 on everything on that cluster, their system roles included. Before going back to a release older than custom roles, revoke every permission of a custom role: in Roles, Delete each custom role and choose Revoke all and delete, or revoke them one by one in Permissions.

Two replicas¶

replicas: 2

With two or more replicas the chart adds a PodDisruptionBudget with minAvailable: 1: a node drain always leaves one pod serving. How to spread the pods, what the replicas share and what each one keeps to itself (the rate limits, a key pasted in Edition, the last sync shown, the OIDC provider's discovery document) is in Several replicas.

With the database in CloudNativePG, its high availability is separate: From one instance to three.

Backups¶

You need two things, kept separately:

What	Why
The Postgres database	Contains everything: accounts and password hashes, clusters with their encrypted tokens, permissions, credentials (only their SHA-256), and the audit log.
`KUBELATCH_ENCRYPTION_KEY`	Not stored in Postgres. Without it, the ServiceAccount tokens in the backup can't be recovered.

Take a daily pg_dump, or use your managed service's snapshots. With CloudNativePG, turn on its backups: Backups.
Keep the key in the secrets manager, separate from the backups.
Test a restore before you need one: restore into another Postgres and start a kubelatch pointing at it with the same key.

Without the key, the backup is incomplete

A Postgres backup restored with a different key still starts, but no cluster works until its tokens are rotated, and everyone has to sign in again. Always keep the key alongside your restore procedure.

Postgres settings¶

idle_in_transaction_session_timeout: the reconciler keeps a transaction open for an entire pass (up to 2 minutes of calls to the cluster, plus the write). If your Postgres has this timeout set, it must be above 3 minutes. Otherwise, long passes fail with terminating connection and the cluster stays in Error until the next one.
Roles: a single role runs both migrations and traffic, and it owns the tables. Don't reuse kubelatch's role for anything else.
Size: see Audit and retention.

If the encryption key is lost or changed¶

KUBELATCH_ENCRYPTION_KEY signs sessions and encrypts each cluster's ServiceAccount tokens. kubelatch only supports one key at a time: there's no rotation with two keys. Changing it (or losing it) has these effects:

Everyone has to sign in again: previous cookies stop working.
No cluster can be used until its tokens are renewed. In Clusters they show Error with could not prepare the reconciler client: rotate the cluster's tokens, the Namespaces card and granting a permission respond 409 asking to rotate the tokens, and Reconcile responds 502 with the same reason, shown on the cluster's page. kubectl gets a 503 with kubelatch cannot query its database; the log says proxy: load cluster.
A license key pasted in Edition can't be opened: kubelatch runs as Free, with a warning in Edition, until you install the key again. The audit already collected stays kept, with the retention you configured; only what is older than 24 hours stops being served until a key verifies again. A key in KUBELATCH_LICENSE is not affected.
Unchanged: passwords (argon2id), klt_ credentials (SHA-256), permissions, and the audit log.

If you only lost it from the secrets manager, recover it from the Secret the pod loads and there's no need to rotate it. From your Secret, or from the one the chart generated (secrets.generateEncryptionKey):

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get secret kubelatch-secrets -o jsonpath='{.data.KUBELATCH_ENCRYPTION_KEY}' | base64 -d
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get secret kubelatch-encryption-key -o jsonpath='{.data.encryption-key}' | base64 -d

To change it on purpose, or if it's gone from everywhere:

Generate a new key: head -c32 /dev/urandom | base64.
Put the new key in place and save it in the secrets manager:
- With your own Secret, update KUBELATCH_ENCRYPTION_KEY in kubelatch-secrets.
- With the generated key, delete the kubelatch-encryption-key Secret and run helm upgrade with the same values: the chart generates a new one, which you copy to the secrets manager.
In both cases, restart the pods: a Secret change doesn't redeploy on its own. Wait for the rollout to finish before continuing. During it, pods with the old key and the new one coexist, and tokens saved midway might not decrypt on the other pod.
```
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout restart deploy/kubelatch
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch
```
For each cluster, open its page in Clusters and choose Re-enroll in its menu: generate the enrollment command and run it with that cluster's kubeconfig. Applying the bootstrap again is part of it. See Register a cluster.
Check that each cluster goes back to Ready.

Uninstall¶

Delete each cluster from Clusters, which cleans up its RBAC, then remove its bootstrap (Delete a cluster). If kubelatch can no longer reach it, clean everything up on the cluster itself:

kubectl --kubeconfig <cluster-kubeconfig> delete -l app.kubernetes.io/managed-by=kubelatch clusterroles,clusterrolebindings,rolebindings -A
kubectl --kubeconfig <cluster-kubeconfig> delete namespace kubelatch-system
kubectl --kubeconfig <cluster-kubeconfig> delete validatingadmissionpolicybinding,validatingadmissionpolicy kubelatch-reconciler-jail

In both cases, if you applied the hardening policy by hand, delete it too: it doesn't carry kubelatch's labels. In the management cluster, hardening.enabled removes it with the release.

kubectl --kubeconfig <cluster-kubeconfig> delete validatingadmissionpolicybinding,validatingadmissionpolicy kubelatch-bindings

Uninstall the release:

helm --kubeconfig "$MGMT_KUBECONFIG" uninstall kubelatch -n kubelatch

It deletes everything the chart created except what holds data or can't be recreated, which stays in place:

Stays	Delete it with
The generated encryption key (`kubelatch-encryption-key`)	`kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch delete secret kubelatch-encryption-key`
The CloudNativePG database (`Cluster kubelatch-db`, its volumes and, with backups on, its `ObjectStore` and `ScheduledBackup`)	Uninstall and the data
Your Secrets and the namespace	`kubectl --kubeconfig "$MGMT_KUBECONFIG" delete namespace kubelatch` (this also deletes everything above: the `Cluster`, its volumes and the generated key, so export what you need first)

Delete your own Postgres database, if it was external, after exporting the audit log if you need it.