Database with CloudNativePG¶

With postgres.mode: cnpg the chart creates, next to kubelatch, a Postgres cluster run by the CloudNativePG operator. Nobody writes or copies a password: kubelatch reads the connection from the Secret the operator generates.

The commands use $MGMT_KUBECONFIG and $KUBELATCH_VERSION, as in Install. Every postgres.cnpg value, with its default, is in Helm values.

What the chart creates¶

Install the operator¶

The chart doesn't install operators: CloudNativePG goes in first, once per cluster. kubelatch is tested with CloudNativePG 1.30.1; backups need 1.26 or later.

kubectl --kubeconfig "$MGMT_KUBECONFIG" apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.1.yaml
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n cnpg-system rollout status deploy/cnpg-controller-manager

Its admission webhook answers a few seconds after the rollout: give it a moment before helm install.

Without the operator, helm install stops with postgres.mode=cnpg needs the CloudNativePG operator and creates nothing. If yours lives in another namespace, set it in postgres.cnpg.operatorNamespace so the NetworkPolicy lets it in. Its pods must carry the app.kubernetes.io/name: cloudnative-pg label: both the operator's manifest and its Helm chart set it.

Install kubelatch with the database¶

Follow Install with two changes:

1. In kubelatch-secrets, only the encryption key (or no Secret at all, with secrets.existingSecret: "" and secrets.generateEncryptionKey: true). The chart always takes DATABASE_URL from kubelatch-db-app: one in kubelatch-secrets is ignored, and one in secrets.env (with secrets.create) stops the install.

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create secret generic kubelatch-secrets \
     --from-literal=KUBELATCH_ENCRYPTION_KEY="$(head -c32 /dev/urandom | base64)"
   

2. In values.yaml:

   postgres:
     mode: cnpg
     cnpg:
       storage:
         size: 10Gi
   

The operator creates the database while kubelatch starts; kubelatch's pod waits for the operator's Secrets and restarts until it can connect. Wait for both:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch wait --for=condition=Ready clusters.postgresql.cnpg.io/kubelatch-db --timeout=5m
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch --timeout=5m

TLS to the database¶

kubelatch always connects with TLS and checks the server's certificate and host name (PGSSLMODE=verify-full) against the operator's CA. From the kubelatch-db-ca Secret the chart mounts only ca.crt, never the CA's private key, at /etc/kubelatch/db-ca/ca.crt (PGSSLROOTCERT).

CloudNativePG renews its CA on its own (by default every 90 days). kubelatch re-reads the file whenever it changes, so new connections check against the renewed CA with no restart. If the file can't be read, new connections fail: kubelatch never falls back to an unchecked connection.

Sizing¶

idle_in_transaction_session_timeout stays at 0 (the Postgres default), as kubelatch's reconciler needs (Postgres settings).

Backups¶

Backups are off by default, and the install notes remind you. With them on, the chart configures continuous WAL archiving and a daily base backup to object storage (S3, GCS or Azure Blob) through the Barman Cloud Plugin, CloudNativePG's supported way. The plugin needs cert-manager and goes in the operator's namespace. The install test doesn't cover backups: validate them on your cluster.

kubectl --kubeconfig "$MGMT_KUBECONFIG" apply -f \
  https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.15.0/manifest.yaml
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n cnpg-system rollout status deploy/barman-cloud

Put the storage credentials in a Secret (S3 in this example) and turn backups on:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create secret generic kubelatch-backup \
  --from-literal=ACCESS_KEY_ID=<key-id> --from-literal=ACCESS_SECRET_KEY=<secret>

postgres:
  mode: cnpg
  cnpg:
    backup:
      enabled: true
      destinationPath: s3://<bucket>/kubelatch
      credentials:
        s3Credentials:
          accessKeyId: {name: kubelatch-backup, key: ACCESS_KEY_ID}
          secretAccessKey: {name: kubelatch-backup, key: ACCESS_SECRET_KEY}
      retentionPolicy: 30d

For GCS and Azure, credentials takes googleCredentials or azureCredentials, as the plugin documents them. schedule (daily at 03:00 by default) uses CloudNativePG's cron syntax, with seconds.

Check that WAL is being archived and that backups complete:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get clusters.postgresql.cnpg.io kubelatch-db -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")].status}'
kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get backups.postgresql.cnpg.io

The ObjectStore kubelatch-db and the ScheduledBackup kubelatch-db-daily carry helm.sh/resource-policy: keep, like the Cluster. They stay after helm uninstall, after leaving postgres.mode: cnpg and after turning backups off. A Cluster left behind goes on archiving WAL to that object store: without it, archiving would fail and pg_wal would fill the volume. To stop backups for good, follow Uninstall and the data.

Restore from a backup¶

A restore creates a new Cluster from the object store. The chart's Cluster is always named kubelatch-db, so the damaged one goes first.

1. Check what you are going to restore from. At least one backup must be completed, and WAL archiving must say True:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get backups.postgresql.cnpg.io
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get clusters.postgresql.cnpg.io kubelatch-db -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")].status}'
   

   If the database can still be read, take a dump as well: the next step deletes its volumes for good.

   primary=$(kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get clusters.postgresql.cnpg.io kubelatch-db -o jsonpath='{.status.currentPrimary}')
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch exec "$primary" -c postgres -- pg_dump -Fc kubelatch > kubelatch.dump
   

2. Stop kubelatch and delete the damaged Cluster (its volumes go with it):

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch scale deploy/kubelatch --replicas=0
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch delete clusters.postgresql.cnpg.io kubelatch-db
   

3. Add the recovery to your values. The new Cluster reads the old backups (serverName: kubelatch-db) and archives into a new folder (backup.serverName). Every restore gets a fresh one (kubelatch-db-2, then kubelatch-db-3…), because CloudNativePG refuses to archive into a folder that already holds an archive:

   postgres:
     mode: cnpg
     cnpg:
       bootstrap:
         recovery:
           source: origin
           # To a point in time: recoveryTarget: {targetTime: "2026-09-26 10:00:00+00"}
       externalClusters:
         - name: origin
           plugin:
             name: barman-cloud.cloudnative-pg.io
             parameters:
               barmanObjectName: kubelatch-db
               serverName: kubelatch-db
       backup:
         enabled: true
         serverName: kubelatch-db-2
         # destinationPath and credentials as before
   

   postgres.cnpg.bootstrap takes exactly one method: initdb, recovery or pg_basebackup. Leave database, owner and secret out: the chart sets the first two to kubelatch and refuses any other value (Troubleshooting).

   initdb with import also moves an existing Postgres into CloudNativePG: point postgres.cnpg.externalClusters at your current database and follow CloudNativePG's import documentation.

4. Apply it and wait:

   helm --kubeconfig "$MGMT_KUBECONFIG" upgrade kubelatch oci://ghcr.io/picaportelabs/charts/kubelatch \
     --version "$KUBELATCH_VERSION" -n kubelatch -f values.yaml
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch wait --for=condition=Ready clusters.postgresql.cnpg.io/kubelatch-db --timeout=30m
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout status deploy/kubelatch
   

   The upgrade brings replicas back to the chart's value. The operator writes a new password into kubelatch-db-app: if the pods started before it, restart them with kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch rollout restart deploy/kubelatch.

5. Take a base backup right away. Until then the new folder only holds WAL and nothing can be restored from it, and the daily ScheduledBackup only took its immediate backup when it was created:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch create -f - <<'EOF'
   apiVersion: postgresql.cnpg.io/v1
   kind: Backup
   metadata:
     generateName: kubelatch-db-after-restore-
   spec:
     cluster:
       name: kubelatch-db
     method: plugin
     pluginConfiguration:
       name: barman-cloud.cloudnative-pg.io
   EOF
   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch get backups.postgresql.cnpg.io
   

6. Leave bootstrap and externalClusters in your values: CloudNativePG only reads them when it creates the Cluster. Next time, restore from kubelatch-db-2 and archive into kubelatch-db-3.

Test the procedure before you need it, in another namespace or another cluster, with the same encryption key and a throwaway backup.serverName (or another bucket or path). A test restore must never archive into a folder production uses or will use next: CloudNativePG refuses to archive into a folder that already holds an archive, so whichever Cluster comes second fails to archive WAL and its pg_wal fills the volume.

Upgrade Postgres¶

• Minor version (17.5 → 17.6): change postgres.cnpg.imageName to the new tag and run helm upgrade. The operator restarts the instances one by one; with three, it moves the primary first (a few seconds without writes). The default rolling tag (17-standard-trixie) doesn't restart anything on its own: pin a minor version (17.6-standard-trixie) to decide when.
• Major version (17 → 18): change imageName to an 18 image and run helm upgrade. CloudNativePG (1.26 or later) upgrades in place with pg_upgrade: it stops the instances, so kubelatch answers 503 until it finishes. Take a backup first; you can't go back to 17 without restoring it.

From one instance to three¶

postgres:
  mode: cnpg
  cnpg:
    instances: 3

helm upgrade adds two replicas, cloned from the primary. If the primary fails, the operator promotes a replica and moves the kubelatch-db-rw service to it; kubelatch reconnects on its own (requests during the switch get 503). The operator also creates the PodDisruptionBudget of the instances.

Uninstall and the data¶

helm uninstall leaves the Cluster kubelatch-db, its volumes and its Secrets in place, and also the generated encryption key. With backups on, the ObjectStore and the ScheduledBackup stay too, and go on archiving WAL and taking the daily backup. Installing again with the same release name and namespace takes them back, with all the data.

Helm's keep only protects against helm uninstall. GitOps tools apply the manifests themselves, so the chart also sets Argo CD's argocd.argoproj.io/sync-options: Delete=false,Prune=false on these objects. With another tool, check that it never prunes the Cluster or the key Secret.

The NetworkPolicy kubelatch-db is not kept: after helm uninstall, any pod can reach the database's port (it still asks for the password) until you install again or delete the Cluster.

To delete the database for good, after exporting whatever you need:

kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch delete clusters.postgresql.cnpg.io kubelatch-db

With backups on, stop them in this order, so no new base backup starts and WAL archiving ends before the object store's definition goes:

1. Delete the daily backup:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch delete scheduledbackups.postgresql.cnpg.io kubelatch-db-daily
   

2. If you keep the Cluster, set postgres.cnpg.backup.enabled: false and run helm upgrade with the same values: the chart removes the plugin from the Cluster and WAL archiving stops. If not, delete the Cluster as above.

3. Delete the object store's definition:

   kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch delete objectstores.barmancloud.cnpg.io kubelatch-db
   

The backups already in the object store stay there until you delete them.