Register a cluster¶

Registering a cluster gives kubelatch two ServiceAccounts on it: one to act as the proxy and one to keep RBAC in sync. When it's done, the cluster shows up as Listo (Ready) in Clusters and you can start granting permissions on it.

Before you start¶

• A kubelatch administrator (Install).
• An administrator kubeconfig for the cluster you're registering.
• Network: kubelatch's pods must reach the API server over https. If it's private, see Private clusters.
• Kubernetes 1.28 or later on the cluster: validation uses SelfSubjectReview.

flowchart LR
  A["Registrar" in Clusters] --> B[Apply bootstrap.yaml]
  B --> C[Run the one-liner]
  C --> D[Paste the JSON]
  D --> E[kubelatch validates and encrypts]
  E --> F[First reconciliation]

Register the cluster¶

1. In Clusters, fill in Identificador (Identifier) and Nombre (Name) and click Registrar (Register). The identifier is a slug (lowercase letters, digits and dashes, up to 63 characters) and goes in the proxy's URL: https://<kubelatch>/clusters/<id>. The cluster is left in Sin tokens (No tokens) and the bootstrap dialog opens.

2. Download the manifest (descargar bootstrap.yaml (download bootstrap.yaml)) and apply it with that cluster's administrator kubeconfig:

   kubectl --kubeconfig <cluster-kubeconfig> apply -f kubelatch-bootstrap-<id>.yaml
   

   It creates the kubelatch-system namespace, the kubelatch-proxy and kubelatch-reconciler ServiceAccounts each with a token Secret, the reconciler's static ClusterRole and the kubelatch-proxy ClusterRole.

3. Copy the one-liner from the dialog and run it with the same kubeconfig. It uses kubectl's current context, so pin it in a subshell:

   (export KUBECONFIG=<cluster-kubeconfig>; <one-liner>)
   

   It waits up to a minute for Kubernetes to fill in the tokens and prints a JSON with server, ca, proxyToken and reconcilerToken.

4. Paste the JSON into JSON de los tokens (Tokens JSON) and click Guardar tokens (Save tokens).

kubelatch validates the tokens against the cluster, stores them encrypted and kicks off the first reconciliation. The tokens are never shown again.

What kubelatch validates¶

• The API server URL: https only, no user, query or fragment, and never a literal link-local, metadata or multicast IP (names aren't resolved to check this).
• The CA: the base64 PEM (certificate-authority-data) or the raw PEM.
• Each token's identity with SelfSubjectReview: proxyToken must be kubelatch-proxy and reconcilerToken, kubelatch-reconciler.
• Each ServiceAccount's permissions with SelfSubjectAccessReview: the proxy must be able to impersonate uids; the reconciler, read RBAC objects, write its own fixed names, bind each tier's roles, list namespaces and create SubjectAccessReviews.

If something fails, the 400 says what: proxyToken: el token pertenece a … (the token belongs to …), la ServiceAccount kubelatch-reconciler no tiene el permiso …: aplica el manifiesto bootstrap de nuevo (the kubelatch-reconciler ServiceAccount doesn't have permission …: apply the bootstrap manifest again). More cases in Troubleshooting.

If the cluster is the management cluster itself¶

Your kubeconfig's server may not be reachable from inside the pod. Use https://kubernetes.default.svc with the same CA:

kubectl --kubeconfig "$MGMT_KUBECONFIG" apply -f kubelatch-bootstrap-<id>.yaml
printf '{"server":"%s","ca":"%s","proxyToken":"%s","reconcilerToken":"%s"}\n' \
  https://kubernetes.default.svc \
  "$(kubectl --kubeconfig "$MGMT_KUBECONFIG" config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')" \
  "$(kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch-system get secret kubelatch-proxy-token -o jsonpath='{.data.token}' | base64 -d)" \
  "$(kubectl --kubeconfig "$MGMT_KUBECONFIG" -n kubelatch-system get secret kubelatch-reconciler-token -o jsonpath='{.data.token}' | base64 -d)"

If the kubeconfig uses a CA file¶

The one-liner reads certificate-authority-data. If your kubeconfig uses certificate-authority: /path/ca.crt (common with kubeadm or minikube), it prints "ca":"" and kubelatch responds 400 falta la CA del cluster (ca) (missing cluster CA (ca)). Put the base64 file into ca: base64 -w0 < /path/ca.crt (on macOS, base64 -i /path/ca.crt).

Check that it works¶

• In Clusters, the status is Listo, with the Kubernetes version under the URL and a date in Última reconciliación (Last reconciliation).
• Namespaces lists the namespaces with their PSA level (PSA restricted, sin PSA (no PSA)…).

• The cluster shows the ClusterRoles for each tier:

  kubectl --kubeconfig <cluster-kubeconfig> get clusterroles -l app.kubernetes.io/managed-by=kubelatch
  

Reconciliation¶

The reconciler keeps each tier's ClusterRoles, one binding per tier and scope with active permissions, and the exact list of users and groups the proxy can impersonate, in sync on the cluster. kubelatch-proxy starts out only allowed to impersonate uids; the reconciler adds users and groups to it as permissions exist. It runs on each cluster:

• On any change: granting or revoking a permission, pasting tokens, disabling or re-enabling an account. Usually takes under a second.
• Every 10 minutes. This pass removes bindings for permissions that have expired (the proxy stops accepting them the instant they expire).
• On demand: the Reconciliar (Reconcile) button in Clusters, or POST /api/clusters/<id>/reconcile.

The manual call responds 200 with the cluster, 502 with the reason if the cluster fails, and 409 if another pass is already running.

If a pass fails, the cluster moves to Error with the reason, and kubelatch retries with growing backoff: 1, 2, 4… minutes, up to 10. A namespace deleted while permissions are still active leaves the cluster in Error until they're revoked, but the rest of the bindings keep being applied. The mechanism's details are in Architecture.

Everything kubelatch creates carries the labels app.kubernetes.io/managed-by=kubelatch and kubelatch.io/instance=<KUBELATCH_INSTANCE_ID>. It only deletes bindings with those labels and its own names, and never deletes anything after a read error. To add CRDs to a tier, see Grant permissions.

Rotate the ServiceAccount tokens¶

The tokens live in the cluster's kubelatch-proxy-token and kubelatch-reconciler-token Secrets. To change them:

1. Delete both Secrets. The old tokens stop working and the proxy fails on that cluster until step 4.

   kubectl --kubeconfig <cluster-kubeconfig> -n kubelatch-system delete secret kubelatch-proxy-token kubelatch-reconciler-token
   

2. Apply the bootstrap again (Volver a pegar tokens (Paste tokens again) → descargar bootstrap.yaml). Kubernetes creates new tokens.

3. Run the one-liner.
4. Paste the JSON with Volver a pegar tokens and click Guardar tokens.

Applying the bootstrap again returns kubelatch-proxy to the uids-only rule: the proxy can't impersonate anyone until the next reconciliation. Saving the tokens already kicks off a reconciliation that restores the list; Reconciliar is only there to confirm the cluster goes back to Listo.

Harden the cluster (optional)¶

A namespace admin, whether granted by kubelatch or native, can create a RoleBinding to kubelatch-developer for anyone. That doesn't grant access through kubelatch, but it does grant a native credential for that subject, and it confuses the reconciler.

deploy/k8s/hardening/vap-kubelatch-bindings.yaml is a ValidatingAdmissionPolicy (Kubernetes 1.30 or later). It only lets kubelatch-system/kubelatch-reconciler create or change bindings whose roleRef starts with kubelatch-. Apply it on each managed cluster, with its kubeconfig:

kubectl --kubeconfig <cluster-kubeconfig> apply -f deploy/k8s/hardening/vap-kubelatch-bindings.yaml

Apply it after the bootstrap. make e2e-install checks that it denies a manual binding and that the reconciler keeps working.

The policy also blocks the bootstrap itself. Its kubelatch-proxy and kubelatch-reconciler ClusterRoleBindings point at kubelatch-* roles, and whoever applies the bootstrap is an administrator, not the reconciler. Rotating tokens works because those bindings don't change. But a bootstrap that has to create or change them fails with Forbidden:

• moving the cluster to another instance (it changes its kubelatch.io/instance label);
• registering it again after removing the bootstrap.

In those cases, remove the policy's binding first, apply the bootstrap, then reapply the policy:

kubectl --kubeconfig <cluster-kubeconfig> delete validatingadmissionpolicybinding kubelatch-bindings
kubectl --kubeconfig <cluster-kubeconfig> apply -f kubelatch-bootstrap-<id>.yaml
kubectl --kubeconfig <cluster-kubeconfig> apply -f deploy/k8s/hardening/vap-kubelatch-bindings.yaml

Delete a cluster¶

Click Borrar (Delete) in Clusters (or DELETE /api/clusters/<id>). kubelatch:

1. Tries to clean up the cluster's RBAC with the reconciler's token, for up to 15 s. It deletes this instance's bindings and tier ClusterRoles and leaves kubelatch-proxy with the uids-only rule.
2. Deletes the cluster, its permissions and the credentials scoped to it. Audit rows are kept.

The final dialog says how the cleanup went: how many bindings and ClusterRoles it deleted, that it couldn't be attempted (no tokens, or tokens that don't decrypt), or why it failed. If it warns about a reconciliation in progress, check that nothing is left.

If the cleanup didn't happen, remove what's left with the cluster's kubeconfig. This doesn't touch the bootstrap:

kubectl --kubeconfig <cluster-kubeconfig> delete clusterroles,clusterrolebindings -l 'app.kubernetes.io/managed-by=kubelatch,kubelatch.io/kind in (binding,tier)'
kubectl --kubeconfig <cluster-kubeconfig> delete rolebindings -A -l 'app.kubernetes.io/managed-by=kubelatch,kubelatch.io/kind in (binding,tier)'

With several kubelatch instances, add ,kubelatch.io/instance=<id> to the selector.

kubectl --kubeconfig <cluster-kubeconfig> delete namespace kubelatch-system
kubectl --kubeconfig <cluster-kubeconfig> delete clusterroles,clusterrolebindings kubelatch-proxy kubelatch-reconciler

kubectl --kubeconfig <cluster-kubeconfig> delete validatingadmissionpolicybinding,validatingadmissionpolicy kubelatch-bindings

Move a cluster to another instance¶

A cluster is registered on a single kubelatch instance. The KUBELATCH_INSTANCE_ID label makes each instance only touch its own objects. To move it:

1. If you applied the hardening policy, remove its binding: the new bootstrap changes its ClusterRoleBindings' label.
2. Register it on the new instance.
3. Delete it on the old one. The cleanup only removes bindings and ClusterRoles labeled with the old instance, but it does return kubelatch-proxy to the uids-only rule.
4. Click Reconciliar on the new instance right after.

This doesn't work for sharing a cluster between two instances at once, because the bootstrap is a single set per cluster.