Skip to content

Troubleshooting

Common symptoms when operating kubelatch, their cause, and how to fix them. Problems from someone using a credential are in If something fails, and every error message is in Errors.

To locate a specific request, use the Audit-ID header from the response (kubectl -v=8 shows it). It's the row's id in Auditoría (Audit), the one that appears in kubelatch's log, and the one the cluster receives in its own audit log.

Installation and startup

Symptom Cause Fix
The pod won't start and the log lists configuration errors kubelatch validates every variable on startup and shows all the errors together. Fix them in config.env or in the Secrets. Details for each variable are in Configuration.
KUBELATCH_KUBECONFIG_CA: … contains a … block The file has more than certificates in it, for example a private key. Point only at ca.crt (or tls.crt if it's self-signed).
KUBELATCH_TRUSTED_PROXIES: … is not a CIDR or … is broader than the minimum A bare IP, or a prefix wider than /8 (IPv4) or /16 (IPv6). Use CIDR (10.0.0.1/32) and the narrowest range possible.
GITHUB_APP_PRIVATE_KEY: not a PEM private key The variable doesn't have the full PEM, headers included, or the line breaks were lost. Create the Secret with --from-file=GITHUB_APP_PRIVATE_KEY=<file>.pem.
… is required when GitHub login is configured Some GITHUB_* variable is set but others are missing. Set all five required ones, or remove all of them. See GitHub login.
/readyz responds 503 database unavailable Postgres isn't responding. Check DATABASE_URL and network reachability to Postgres. In the meantime, the proxy responds 503; open sessions aren't cut.
tls: reload failed after renewing the certificate The new pair is inconsistent, or cert-manager hasn't written both files yet. kubelatch keeps using the previous one. Wait for the next check (10 s). If it persists, check the kubelatch-tls Secret.
tls: the served certificate expires soon Fewer than 30 days remain. Renew the certificate; kubelatch reloads it without restarting.
GitHub login is enabled but no admin can log in You enabled GitHub without a break-glass account or a linked admin. kubelatch user set-break-glass <login>: see Accounts and recovery.

Ingress and network

Symptom Cause Fix
kubectl get pods -w, logs -f, or exec cut off after 60 s behind an Ingress The controller's timeouts aren't set. Apply the settings in deploy/k8s/overlays/ingress/ingress.yaml: Ingress. Doesn't happen with an L4 LoadBalancer.
kubectl apply of a large manifest responds 413 The controller's body-size limit (1 MiB by default). proxy-body-size: "0".
All the audit log shows the same IP, or login responds 429 to everyone KUBELATCH_TRUSTED_PROXIES is missing the controller's CIDR. See Trusted proxies.
The API responds 403 petición de otro origen rechazada (cross-origin request rejected) The browser sends an Origin different from KUBELATCH_BASE_URL. Serve the UI from that same URL. Check that KUBELATCH_BASE_URL matches the public URL.
A rollout cut an exec or a port-forward This is expected: on shutdown, kubelatch closes streams after 3 s. Relaunch the command. kubectl only retries watches. See Upgrades.

Clusters and reconciliation

Symptom Cause Fix
The one-liner says no aparece el token de …: ¿aplicaste el manifiesto en este contexto? (the token for … doesn't show up: did you apply the manifest in this context?) The bootstrap isn't applied to the current context's cluster. Apply the bootstrap and run the one-liner with the same KUBECONFIG.
400 falta la CA del cluster (ca) (missing cluster CA) The kubeconfig uses certificate-authority: <file> and the one-liner only reads certificate-authority-data. Put the base64-encoded file in ca: base64 -w0 < /path/ca.crt (on macOS, base64 -i).
400 … el token pertenece a … (the token belongs to …) The tokens in the JSON are mixed up or belong to another cluster. Run the one-liner again with the correct kubeconfig.
400 la ServiceAccount … no tiene el permiso … (the ServiceAccount … doesn't have the … permission) when pasting the JSON The bootstrap isn't fully applied, or it's from another version. The tokens weren't saved. Download the bootstrap again, apply it, and paste the JSON again with Guardar tokens (Save tokens).
Cluster in Error with … deniega la operación a la ServiceAccount kubelatch-reconciler: vuelve a aplicar el manifiesto bootstrap (… denies the operation to the kubelatch-reconciler ServiceAccount: re-apply the bootstrap manifest) The reconciler is missing some permission from the bootstrap. Apply the bootstrap again and click Reconciliar (Reconcile).
The cluster is in Error The last reconciliation failed; the reason shows under the status (API server inalcanzable (API server unreachable), … vuelve a aplicar el manifiesto bootstrap (… re-apply the bootstrap manifest), a namespace that no longer exists…). Fix the cause. kubelatch retries with increasing backoff; Reconciliar forces a pass. A namespace deleted while permissions are still active keeps the Error until they're revoked.
Reconciliar responds 502 The cluster is still failing. The message carries the reason.
Reconciliar responds 409 Another pass is already running, or the cluster has no tokens. Wait a few seconds, or paste the tokens.
reconciler: pass failed … terminating connection due to idle-in-transaction timeout Postgres's idle_in_transaction_session_timeout is lower than a pass takes. Raise it above 3 minutes or turn it off for kubelatch's role.
409 … ¿cambió KUBELATCH_ENCRYPTION_KEY? (did KUBELATCH_ENCRYPTION_KEY change?) or vuelve a pegar los tokens del cluster (paste the cluster's tokens again) The encryption key changed or was lost. Paste each cluster's tokens again: Upgrades.

Permissions and proxy

Symptom Cause Fix
kubectl responds Forbidden even though the permission exists Reconciliation hasn't run yet. If the cluster says cannot impersonate, kubelatch-proxy doesn't yet include that person. Check Última reconciliación (Last reconciliation) and the status in Clusters, or click Reconciliar.
403 … no tiene permisos activos en el cluster (… has no active permissions on the cluster) No active permission on that cluster (revoked or expired). Grant the permission in Permisos (Permissions).
403 esta credencial está restringida al cluster … (this credential is restricted to cluster …) The credential was issued for another cluster. Issue another one without a restriction, or for that cluster.
403 la cuenta está deshabilitada (the account is disabled) The subject is disabled. Habilitar (Enable) in Usuarios (Users) and issue a new credential.
401 credencial revocada (credential revoked) or credencial caducada (credential expired) Credentials aren't reactivated. Issue another one.
400 kubelatch no admite cabeceras Impersonate-* (kubelatch doesn't support Impersonate-* headers) kubectl --as was used. Not supported: see Check what someone can do.
400 ruta no válida: segmentos vacíos, '.' o '..' o caracteres escapados… (invalid path: empty segments, '.' or '..' or escaped characters…) The path has //, ., .., or a % (a name encoded by the client). kubectl doesn't encode valid names; with curl or other tools, put the name unencoded.
502 no se pudo hablar con el API server del cluster: … (couldn't talk to the cluster's API server: …) kubelatch can't reach the registered URL. Check the network or the tunnel (Private clusters). The full error is in the log.
503 kubelatch no puede consultar su base de datos (kubelatch can't query its database) Postgres isn't responding, or, if it only happens for one cluster, its tokens can't be decrypted (log proxy: load cluster). Check Postgres, or paste the tokens again.
GET /api/audit responds 400 A filter is invalid; the message says which. subject and credential are uuids, cluster is a slug, from/to are RFC 3339 with from earlier, page is between 1 and 10000.

Audit rows with status 499 or 403 with no obvious reason: see Audit.

Accounts

Symptom Cause Fix
Login responds 429 demasiados intentos (too many attempts) 5 consecutive failures (15 min lockout) or more than 10 attempts per minute from that IP. Wait, or an Enlace de reset (Reset link), or kubelatch user set-password.
Nobody can sign in Forgotten password, locked account, or the only admin is disabled. The CLI via kubectl exec: When nobody can sign in.
Nobody can sign in after configuring GITHUB_* Only break-glass accounts keep their password. kubelatch user set-break-glass <login> and sign in from Cuenta de emergencia (contraseña) (Break-glass account (password)).
409 when removing the role from, or disabling, an admin It's the last administrator who can sign in. Name another admin first.

GitHub login

Symptom Cause Fix
GitHub shows redirect_uri is not associated with this application The App's Callback URL doesn't match <KUBELATCH_BASE_URL>/api/auth/github/callback exactly. Fix the scheme, host, port, and trailing slash in the App.
app not installed in the organization in the sync log The App isn't installed, or GITHUB_ORG isn't the organization's exact login. Install it and check GITHUB_ORG.
Everyone sees no es miembro activo de la organización (not an active member of the organization; not_member) The App doesn't have Organization → Members: read, or the organization isn't the one in GITHUB_ORG. Check the App's permissions and GITHUB_ORG.
no_2fa when signing in The organization doesn't enforce 2FA. Turn it on in GitHub and wait up to 10 minutes (cache). If you really don't want to enforce it, GITHUB_REQUIRE_ORG_2FA=false.
no está vinculada a ninguna cuenta de kubelatch (not linked to any kubelatch account; unknown_account) There's no automatic sign-up. Create the account or generate an Enlace de vinculación (Linking link).
github_taken when linking That GitHub account is already linked to someone else. Desvincular GitHub (Unlink GitHub) on the other account first.
membership sync: pass failed; nobody was disabled GitHub error, App uninstalled, or zero members. Check the message. The next pass retries it.
The webhook responds 401 firma no válida (invalid signature) The App's secret and GITHUB_WEBHOOK_SECRET don't match. Fix it. GitHub retries deliveries from the App's Advanced tab.
The webhook responds 415 The webhook isn't using Content type: application/json. Change it in the App's webhook configuration.
The webhook responds 413 The delivery is over 1 MiB. Shouldn't happen with normal GitHub deliveries.
The webhook responds 404 GITHUB_WEBHOOK_SECRET is missing. Configure it, or turn off the App's webhook.

CI with GitHub Actions

Symptom Cause Fix
The exchange responds 401 token no válido (invalid token) The id_token fails verification: signature, iss, aud different from KUBELATCH_BASE_URL, expired, or jti already redeemed. The reason is in the log (ci: token rejected reason=…, or ci: token replayed if the jti was already redeemed) and in the control plane (ci.exchange.failure). Common causes: the workflow's KUBELATCH_URL doesn't match letter for letter, the token was redeemed twice, or the server's clock is off by more than 30 s.
403 ninguna trust rule coincide … (no trust rule matches …) There's no rule for that repository, ref, and environment. The message carries repository_owner_id, repository_id, ref, and environment: create the rule with those values, or drop the ref or the environment.
403 naming the bot The bot is disabled or has no active permissions. Enable it or grant it permissions.
429 More than 60 exchanges per minute from the same IP. Reduce the frequency.
curl: (7) or (35) in the workflow The runner can't reach kubelatch over https, or doesn't trust its certificate. You need a name reachable from the runners and a trusted certificate (or KUBELATCH_KUBECONFIG_CA).

What to watch

kubelatch doesn't expose /metrics yet. With what's there:

  • /readyz in your external monitoring: 503 means Postgres is unreachable.
  • The JSON log (kubectl logs): tls: the served certificate expires soon, tls: reload failed, reconciler: pass failed (with the cluster and the reason), retention: pass failed, membership sync: pass failed, ci: token rejected.
  • GET /api/clusters: status and last_error for each cluster (the same as Clusters).
  • GET /api/control-events: several login.failure events in a row from the same IP.