Troubleshooting¶

Common symptoms when operating kubelatch, their cause, and how to fix them. Problems from someone using a credential are in If something fails, and every error message is in Errors.

To locate a specific request, use the Audit-ID header from the response (kubectl -v=8 shows it). It's the row's id in Audit, the audit_id of every line kubelatch logs about that request, and the id the cluster receives in its own audit log. For the API and the web UI, the header is X-Request-Id: see The server log.

Installation and startup¶

Symptom	Cause	Fix
The pod won't start and the log lists configuration errors	kubelatch validates every variable on startup and shows all the errors together.	Fix them in `values.yaml` (`config`) or in the Secrets, and run `helm upgrade`. Details for each variable are in Configuration.
`helm install` fails with `postgres.mode=cnpg needs the CloudNativePG operator` (or `needs cert-manager`, `needs the Barman Cloud Plugin`)	The chart checks that the operator's API exists before creating anything.	Install what the message names: CloudNativePG (Install the operator), cert-manager (its installation guide; or use your own certificate, see Install), or the Barman Cloud Plugin (Backups). With `helm template`, pass `--api-versions` with the API the message names.
`helm install` or `helm upgrade` fails with `postgres.cnpg.bootstrap.<method>.database must be kubelatch (or left out)` (or `.owner`), or with `postgres.cnpg.bootstrap.<method>.secret is not supported`	The restore or import names another database, another owner or its own Secret. kubelatch connects through the `kubelatch-db-app` Secret, which CloudNativePG only writes for the bootstrap's database and owner, and not at all when the bootstrap names its own secret.	Leave `database`, `owner` and `secret` out: the chart sets `kubelatch`. See Restore from a backup.
`helm install` or `helm upgrade` fails with `secrets.existingSecret … already has KUBELATCH_ENCRYPTION_KEY: secrets.generateEncryptionKey is only for new installs; keep using the key in secrets.existingSecret (removing it would make the chart generate a different key and seal every cluster token)`	`secrets.generateEncryptionKey` is on, but kubelatch already has a key in its Secret. A generated key would replace it, and the clusters' tokens sealed with it would stop working.	Turn `secrets.generateEncryptionKey` off and keep the key in `secrets.existingSecret`. Don't remove it from the Secret. See The encryption key.
`postgres.cnpg.bootstrap takes exactly one method` or `the method must be one of initdb, recovery or pg_basebackup`	`postgres.cnpg.bootstrap` has more than one method, or an unknown one.	Keep a single `initdb`, `recovery` or `pg_basebackup`.
The database's `Cluster` stays in the phase `Unable to create required cluster objects` and ignores changes to its spec, while the CloudNativePG operator logs `cannot create Cluster auxiliary objects` and `ValidatingAdmissionPolicy 'kubelatch-bindings' … denied request`	The hardening policy of 0.59.0 or earlier refuses the operator's RoleBinding to its Role `kubelatch-db`. The database keeps serving.	Install the current policy: The hardening policy and CloudNativePG.
`KUBELATCH_KUBECONFIG_CA: … contains a … block`	The file has more than certificates in it, for example a private key.	Point only at `ca.crt` (or `tls.crt` if it's self-signed).
`KUBELATCH_TRUSTED_PROXIES: … is not a CIDR` or `… is broader than the minimum`	A bare IP, or a prefix wider than `/8` (IPv4) or `/16` (IPv6).	Use CIDR (`10.0.0.1/32`) and the narrowest range possible.
`GITHUB_APP_PRIVATE_KEY: not a PEM private key`	The variable doesn't have the full PEM, headers included, or the line breaks were lost.	Create the Secret with `--from-file=GITHUB_APP_PRIVATE_KEY=<file>.pem`.
`… is required when GitHub login is configured`	Some `GITHUB_*` variable is set but others are missing.	Set all five required ones, or remove all of them. See GitHub login.
The pod won't start and the configuration errors name `OIDC_*` variables	Some of `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET` and `OIDC_ALLOWED_DOMAINS` are set and a required one is missing, or a value breaks its rule: an `http` issuer outside loopback, a Microsoft Entra ID issuer without the tenant id (`common`, `organizations`, `consumers` or a domain name), a domain with a wildcard, scopes without `openid` or `email`, a subject claim that can change, such as `email` or `upn`.	Set `OIDC_ISSUER`, `OIDC_CLIENT_ID` and `OIDC_ALLOWED_DOMAINS` (and `OIDC_CLIENT_SECRET` for a confidential client), or remove all four. Each rule is in Configuration.
`/readyz` responds `503 database unavailable`	Postgres isn't responding.	Check `DATABASE_URL` and network reachability to Postgres. In the meantime, the proxy responds `503`; open sessions aren't cut.
`tls: reload failed` after renewing the certificate	The new pair is inconsistent, or cert-manager hasn't written both files yet. kubelatch keeps using the previous one.	Wait for the next check (10 s). If it persists, check the `kubelatch-tls` Secret.
`tls: the served certificate expires soon`	Fewer than 30 days remain.	Renew the certificate; kubelatch reloads it without restarting.
`… but no admin can log in`	You turned on GitHub or OIDC login without a break-glass account or an admin who can sign in through it. With OIDC, it also shows until an admin signs in for the first time and gets linked.	`kubelatch-server user set-break-glass <login>`: see Accounts and recovery.

Ingress and network¶

Symptom	Cause	Fix
`kubectl get pods -w`, `logs -f`, or `exec` cut off after 60 s behind an Ingress	The controller's timeouts aren't set: another controller, or the chart's annotations overridden in `ingress.annotations`.	Use `exposure: ingress` with ingress-nginx, or your controller's equivalent settings: Ingress. Doesn't happen with an L4 `LoadBalancer`.
`helm install` or `helm upgrade` fails with `exposure=ingress needs networkPolicy.enabled: kubelatch serves plain HTTP on 8080 and trusts X-Forwarded-For from config.trustedProxies, so only the Ingress controller may reach it`	`exposure: ingress` with `networkPolicy.enabled: false`. Behind the Ingress, kubelatch listens on plain HTTP and believes the `X-Forwarded-For` of `config.trustedProxies`: without the `NetworkPolicy`, any pod in that range could reach it and spoof the audit log's IP.	Set `networkPolicy.enabled: true` and `networkPolicy.ingressControllerNamespace` to your controller's namespace. See Ingress.
`helm install` or `helm upgrade` fails with `ingress.host (…) must match the host of config.baseURL (…): kubelatch only knows one public host`	`ingress.host` and the host of `config.baseURL` differ.	Make `ingress.host` exactly the host of `config.baseURL`. See Ingress.
`helm install` or `helm upgrade` fails with `tls.certManager.enabled and an ingress cert-manager annotation both request a Certificate for the same secretName: keep only one source of the TLS secret`	Both the chart's `Certificate` and a `cert-manager.io/…issuer` annotation in `ingress.annotations` would fill `kubelatch-tls`.	Keep one: turn off `tls.certManager.enabled` or remove the annotation. See Ingress.
`kubectl apply` of a large manifest responds `413`	The controller's body-size limit (1 MiB by default).	`proxy-body-size: "0"`.
All the audit log shows the same IP, or login responds `429` to everyone	`KUBELATCH_TRUSTED_PROXIES` is missing the controller's CIDR.	See Trusted proxies.
The API responds `403 cross-origin request rejected`	The browser sends an `Origin` different from `KUBELATCH_BASE_URL`.	Serve the UI from that same URL. Check that `KUBELATCH_BASE_URL` matches the public URL.
A rollout cut an `exec` or a `port-forward`	This is expected: on shutdown, kubelatch closes streams after 3 s.	Relaunch the command. kubectl only retries `watch`es. See Upgrades.

Clusters and reconciliation¶

Symptom	Cause	Fix
The enrollment command prints `kubelatch: waiting for the token of … (up to a minute)...` and, after a minute, `kubelatch: the token of … does not show up in namespace kubelatch-system`	Kubernetes didn't fill in the token Secrets in that cluster. The script printed the context it used (`enrolling cluster "…" with kubectl context "…"`): an empty `KUBECONFIG` falls back to `~/.kube/config`, so it may be the wrong cluster.	Rerun the command, same link, with the right kubeconfig or `--context <name>`.
The one-liner (Do it by hand) prints `waiting for the token of … in kubectl context "…"` and, after a minute, `the token of … does not show up: did you apply the manifest in this context?`	The bootstrap isn't applied to that context's cluster.	Apply the bootstrap and run the one-liner with the same `KUBECONFIG`.
The command prints `kubelatch: this enrollment link is invalid or expired; generate a new one in Clusters`, or `GET /api/enroll/…` and the API respond `404` with `enrollment.invalid`	The link was already used, expired (it lasts one hour), was invalidated, or a newer command replaced it. It also stops working when the administrator who generated it is disabled or loses the admin role, and kubelatch revokes it after 10 refused attempts (tokens it couldn't validate), so that a leaked link can't keep making it call other addresses.	Open the cluster's page in Clusters and generate another command in its Enroll card (Generate a new command); on an enrolled cluster, choose Re-enroll in its menu first, which opens the re-enrollment card.
The command prints `kubelatch: no terminal to confirm on: rerun with --yes`	It asks for confirmation on `/dev/tty` and there's no terminal (CI, a remote session without tty).	Run it with `--yes`: `curl … \\| sh -s -- --yes`.
The command prints `curl: (60) SSL certificate problem` or `unable to get local issuer certificate`	kubelatch's certificate comes from a CA that curl doesn't trust (a private CA).	Point curl at the CA: `CURL_CA_BUNDLE=/path/ca.crt`, together with `KUBECONFIG` in the subshell.
The command prints `kubelatch: kubelatch refused the tokens: … API server unreachable …`	kubelatch can't reach the API server address the script sent (your kubeconfig's, if you set none).	Rerun it, same link, with `--server <url kubelatch can reach>`, or generate the command again with that address in API server URL for kubelatch (under Advanced options). See Private clusters.
The command prints `kubelatch: kubectl has no current context: pass --context <name>`	The kubeconfig has no current context.	Add `--context <name>` after `sh -s --`.
`400 … the token belongs to …`	The tokens in the JSON are mixed up or belong to another cluster.	Run the enrollment command (or the one-liner) again with the correct kubeconfig.
`400 the ServiceAccount … lacks the permission …` when enrolling or pasting the JSON	The bootstrap isn't fully applied, or it's from another version. The tokens weren't saved.	Run the enrollment command again (it applies the manifest), or, by hand, download the bootstrap again, apply it and paste the JSON with Save tokens.
Applying the bootstrap fails with `no matches for kind "ValidatingAdmissionPolicy" in version "admissionregistration.k8s.io/v1"`	The cluster runs Kubernetes older than 1.30. The manifest's admission policy, which bounds the reconciler, needs 1.30.	Upgrade the cluster to Kubernetes 1.30 or later.
Cluster in Error with `bootstrap incomplete: the cluster has the reconciler's new rights but not the admission policy kubelatch-reconciler-jail: apply the bootstrap manifest again`	Someone deleted the `kubelatch-reconciler-jail` policy or its binding. kubelatch writes no custom role on that cluster, and deletes none, until it's back.	Apply the bootstrap again (Update the bootstrap) and click Reconcile.
Cluster in Error with `custom roles are granted on this cluster but it has the previous bootstrap: apply the bootstrap manifest again`	The cluster has the previous manifest, which doesn't allow custom roles. The system tiers keep working.	Update the bootstrap and click Reconcile.
Cluster in Error with `custom role <id> no longer passes validation and was not applied: edit or delete it`	The role holds something kubelatch now refuses (a later release tightened a rule). Its permissions take no effect on any cluster.	Open the role in Roles with Edit: the builder shows what to fix. Or delete it. See Roles.
Cluster in Error with `… denies the operation to the kubelatch-reconciler ServiceAccount: apply the bootstrap manifest again`	The reconciler is missing some permission from the bootstrap.	Apply the bootstrap again and click Reconcile.
The cluster is in Error	The last reconciliation failed; the reason shows on the cluster's page, with View the full error for the rest (`API server unreachable`, `… apply the bootstrap manifest again`, a namespace that no longer exists…).	Fix the cause. kubelatch retries with increasing backoff; Reconcile forces a pass. A namespace deleted while permissions are still active keeps the Error until they're revoked.
Reconcile responds `502`	The cluster is still failing.	The message carries the reason.
Reconcile responds `409`	Another pass is already running, or the cluster is Not enrolled.	Wait a few seconds, or enroll the cluster (Register a cluster).
`reconciler: pass failed … terminating connection due to idle-in-transaction timeout`	Postgres's `idle_in_transaction_session_timeout` is lower than a pass takes.	Raise it above 3 minutes or turn it off for kubelatch's role.
A namespace's Pod Security label changed, or the namespace was deleted, and its permissions took minutes to stop	The watch on that cluster's namespaces is down, so only the 10-minute pass noticed (Reconciliation). Each replica's log says `namespace watch down` with the cause: the reconciler's ServiceAccount lacks the `watch` right on namespaces (`cannot watch resource "namespaces"`: someone edited the bootstrap), or a proxy or load balancer in front of the API server cuts or holds back long requests (`not opened within 1m0s`). `namespace watch: cannot build the cluster's client` means the cluster's tokens can't be read.	Apply the bootstrap again (Update the bootstrap), or let long requests through that proxy: kubelatch's watch, like `kubectl get -w`, stays open for up to 10 minutes. If it's the tokens, rotate them (Re-enroll). `namespace watch up` says it's back; meanwhile, Reconcile applies a change at once.
`409 … (did KUBELATCH_ENCRYPTION_KEY change?)`, or `rotate the cluster's tokens` in the cluster's error	The encryption key changed or was lost.	Rotate each cluster's tokens (Re-enroll): Upgrades.

The enrollment command prints the server's error text in English (kubelatch: kubelatch refused the tokens: …), whatever the interface language; the web UI shows the translated message.

Enrolling a cluster with your identity¶

The panel shows these messages in the interface's language, under The enrollment with your identity did not finish or as the error of Continue to … (Enroll with your identity); the failure's What this failure means leads here. The sign-in's own messages are those of OIDC login.

Symptom	Cause	Fix
With my identity (…) is disabled, with Your account is not linked to …. Link it in My account., or `your account is not linked to the identity provider: link it in My account, or use the script`	Your account has no identity of the configured provider linked: a break-glass account, or one that has never signed in with the provider.	Link it in My account, or use the script.
`kubelatch does not have this cluster's CA: paste it`	kubelatch runs inside this cluster is chosen and kubelatch has no CA mounted: an older chart, `clusterCA.mount: false`, or kubelatch outside Kubernetes.	Paste the CA, or leave `clusterCA.mount` at `true` and run `helm upgrade` (What the chart installs).
`this cluster has no stored address and CA yet: choose where it is`	Where it already is was asked of a cluster that has no stored API server URL and CA: it was never enrolled, or its record changed in between.	Choose kubelatch runs inside this cluster or Another cluster and give the CA.
`this cluster's address changed after the page showed it: check where it is and try again`	Where it already is named an address the cluster no longer stores: another administrator changed it (new tokens pasted, an enrollment elsewhere) after your page loaded. Nothing was created and the token didn't leave.	Check the address the panel now shows under Where it already is, then continue.
`with "where it already is" kubelatch uses the stored URL and CA: send neither`	An API client sent `in_place` with a URL, a CA or `in_cluster`. The panel never does.	Send `in_place` alone, or the URL and the CA without it.
`an enrollment with an administrator's identity is running on this cluster; wait for it to finish`	kubelatch is applying the manifest with an administrator's token, and an enrollment under way doesn't stop halfway. Cancel, Invalidate, Generate command and a new enrollment get this `409`.	Wait: it takes two minutes at most, and the panel says how it ended.
`the API server does not accept the … token for the client …: make it trust that client and check that the token carries the claims it maps (such as a verified email), or use the script`	The API server answered `401` to your ID token: it doesn't trust that issuer, or not with kubelatch's client ID as audience, or the token lacks a claim the cluster maps (the e-mail, or its `email_verified` is `false`). Nothing was written.	Make the cluster trust kubelatch's client, or map a claim the ID token carries (Configure the cluster), or use the script.
`the API server at … presented a certificate the CA does not sign`	The CA isn't the one that signs that API server's certificate: it's another cluster's, or that of a proxy in front of it. The token wasn't sent.	Paste that API server's CA: the read-only command prints the one your kubeconfig uses.
`the cluster runs Kubernetes 1.29.4; kubelatch needs 1.30 or later`	The bootstrap's admission policy needs Kubernetes 1.30. Nothing was written.	Upgrade the cluster.
`the API server sees you as oidc:bruno@acme.com (groups oidc:developers) and that identity cannot create clusterrolebindings`	The name the API server gives you lacks a right the manifest needs. Nothing was written.	Give that user, or one of those groups, `cluster-admin` with a ClusterRoleBinding in the cluster, or use the script.
`could not apply …/…: the API server answered …`	The API server refused an object of the manifest: an admission webhook, or the hardening policy, which answers `403` to a bootstrap that creates or changes its ClusterRoleBindings. What came before it stays applied.	Fix the cause and click Try again: applying again is idempotent. With the hardening policy, remove its binding first (Harden the cluster).
`the ServiceAccount tokens did not show up within a minute`	Kubernetes didn't fill in the two token Secrets in `kubelatch-system`.	Try again. If it happens again, check the cluster's controller manager, which fills them in.
A reason from What kubelatch validates, such as `proxyToken: the token belongs to …`	The ServiceAccount tokens didn't validate.	Try again: it applies the manifest again.
You signed in to … with an identity other than the one linked to your account (…). Try again with that one.	You signed in at the provider with an account other than the one linked to your kubelatch account.	Try again with the linked identity; if the provider keeps another session open, sign out of it there first.
The enrollment expired, or another one replaced it, before you came back from …. Start it again.	More than 10 minutes went by at the provider, or meanwhile someone started another enrollment of the cluster or cancelled this one.	Try again.

Permissions and proxy¶

Symptom	Cause	Fix
kubectl responds `Forbidden` even though the permission exists	Reconciliation hasn't run yet. If the cluster says `cannot impersonate`, `kubelatch-proxy` doesn't yet include that person.	Check the cluster's status and its `reconciled …` line in Clusters, or click Reconcile.
`403 … has no active permissions on cluster`	No active permission on that cluster (revoked or expired).	Grant the permission in Permissions.
`403 this credential is restricted to cluster …`	The credential was issued for another cluster.	Issue another one without a restriction, or for that cluster.
`403 the account is disabled`	The subject is disabled.	Enable in Users and issue a new credential.
`401 credential revoked` or `credential expired`	Credentials aren't reactivated.	Issue another one.
`400 kubelatch does not accept Impersonate-* headers`	`kubectl --as` was used.	Not supported: see Check what someone can do.
`400 invalid path: empty segments, '.' or '..' or escaped characters…`	The path has `//`, `.`, `..`, or a `%` (a name encoded by the client).	kubectl doesn't encode valid names; with curl or other tools, put the name unencoded.
`502 could not talk to the cluster's API server: …`	kubelatch can't reach the registered URL.	Check the network or the tunnel (Private clusters). The full error is in the log.
`503 kubelatch cannot query its database`	Postgres isn't responding, or, if it only happens for one cluster, its tokens can't be decrypted (log `proxy: load cluster`).	Check Postgres, or rotate the tokens.
`GET /api/audit` responds `400`	A filter is invalid; the message says which.	`subject` and `credential` are uuids, `cluster` is a slug, `from`/`to` are RFC 3339 with `from` earlier, `page` is between 1 and 10000.

Audit rows with status 499 or 403 with no obvious reason: see Audit.

Accounts¶

Symptom	Cause	Fix
Login responds `429 too many attempts`	5 consecutive failures (15 min lockout) or more than 10 attempts per minute from that IP. The log says which: `login: account locked` (and `login: refused, account locked` while it lasts) or `login: rate limited`, with the login and `client_ip`.	Wait, or Reset access on the account in Users, or `kubelatch-server user set-password`.
Nobody can sign in	Forgotten password, locked account, or the only admin is disabled.	The CLI via `kubectl exec`: When nobody can sign in.
Nobody can sign in after configuring `GITHUB_` or `OIDC_`	Only break-glass accounts keep their password.	`kubelatch-server user set-break-glass <login>` and sign in from Sign in with a password.
`409` when removing the role from, or disabling, an admin	It's the last administrator who can sign in.	Name another admin first.

Symptom	Cause	Fix
GitHub shows `redirect_uri is not associated with this application`	The App's Callback URL doesn't match `<KUBELATCH_BASE_URL>/api/auth/github/callback` exactly.	Fix the scheme, host, port, and trailing slash in the App.
`app not installed in the organization` in the sync log	The App isn't installed, or `GITHUB_ORG` isn't the organization's exact login.	Install it and check `GITHUB_ORG`.
Everyone sees "Your GitHub account is not an active member of the organization." (`not_member`)	The App doesn't have Organization → Members: read, or the organization isn't the one in `GITHUB_ORG`.	Check the App's permissions and `GITHUB_ORG`.
`no_2fa` when signing in	The organization doesn't enforce 2FA.	Turn it on in GitHub and wait up to 10 minutes (cache). If you really don't want to enforce it, `GITHUB_REQUIRE_ORG_2FA=false`.
"That GitHub account isn't linked to any kubelatch account. Ask an administrator for a linking link." (`unknown_account`)	There's no automatic sign-up.	Create the account or generate a Linking link.
`github_taken` when linking	That GitHub account is already linked to someone else.	Unlink GitHub on the other account first.
`membership sync: pass failed; nobody was disabled`	GitHub error, App uninstalled, or zero members.	Check the message. The next pass retries it.
The webhook responds `401 invalid signature`	The App's secret and `GITHUB_WEBHOOK_SECRET` don't match.	Fix it. GitHub retries deliveries from the App's Advanced tab.
The webhook responds `415`	The webhook isn't using Content type: `application/json`.	Change it in the App's webhook configuration.
The webhook responds `413`	The delivery is over 1 MiB.	Shouldn't happen with normal GitHub deliveries.
The webhook responds `404`	`GITHUB_WEBHOOK_SECRET` is missing.	Configure it, or turn off the App's webhook.

{name} is the provider's name, OIDC_DISPLAY_NAME. Every message a person can read on the login page is in When a sign-in fails, with what to do.

Symptom	Cause	Fix
The provider refuses the redirect URI (`redirect_uri_mismatch`, `invalid redirect_uri` or similar)	The redirect URI registered in the provider doesn't match `<KUBELATCH_BASE_URL>/api/auth/oidc/callback` exactly.	Fix the scheme, host, port and trailing slash in the provider.
Everyone sees "kubelatch can't reach {name} right now. Try again in a few minutes; if it persists, tell an administrator." (`provider_unavailable`), and the log says `oidc login failed` with `code` `provider_unavailable` (at start-up, `OIDC discovery failed; sign-ins will retry it`)	kubelatch can't read the provider's discovery document (`<OIDC_ISSUER>/.well-known/openid-configuration`), its keys or its token endpoint: a wrong `OIDC_ISSUER`, a firewall between the pod and the provider, the provider is down, or its certificate comes from a private CA. Users and Notices show the last error.	Check `OIDC_ISSUER`, written exactly as the provider publishes it (trailing slash included), and that the pod reaches the provider over https. For a private CA, mount its certificate and point `SSL_CERT_FILE` at it, as the Keycloak section explains. kubelatch tries again on the next sign-in, at most once a minute.
Everyone sees "kubelatch couldn't verify the answer from {name}. Try again; if it persists, tell an administrator." (`token`)	The `id_token` fails verification: it isn't signed with RS256, its `iss` isn't exactly `OIDC_ISSUER`, its `aud` lacks `OIDC_CLIENT_ID`, it lacks the claim of `OIDC_SUBJECT_CLAIM`, or the server's clock is off by more than 30 s.	The log line `oidc login failed` (with `code` `token`) carries the cause in `err`. Fix the variable, have the provider sign ID tokens with RS256, or fix the clock.
"{name} didn't share your e-mail address with kubelatch. Ask an administrator to check the configuration." (`no_email`)	Neither the `id_token` nor the provider's `userinfo` carries an e-mail address.	Make the provider release `email` to this client (in Entra, the optional claim `email`): see the provider's section in Providers.
"{name} doesn't vouch for your e-mail address, so kubelatch can't match it to an account." (`email_unverified`)	The claim named in `OIDC_EMAIL_VERIFIED_CLAIM` isn't `true` for that person.	Entra needs `OIDC_EMAIL_VERIFIED_CLAIM=xms_edov` and that optional claim; Keycloak, Trust Email for the users that come from LDAP, if administrators alone write their `mail`; authentik 2025.10 or later, an `email` mapping that passes on a real verification, not a hard-coded `True`. See Providers.
"That {name} account isn't in a domain this kubelatch accepts. Did you pick a personal account?" (`domain_not_allowed`)	The e-mail's domain isn't in `OIDC_ALLOWED_DOMAINS` (exact, subdomains included only if listed), or, with Google, the `hd` claim is missing or names another domain.	The person picks their company account (Try another account), or you add the domain to `OIDC_ALLOWED_DOMAINS`.
"No kubelatch account has the e-mail address of that {name} account." (`unknown_account`)	An account has that e-mail, but `OIDC_AUTOLINK` doesn't allow linking it: `off`, or `invited` and the account has signed in before and has no pending link. Someone whose e-mail no account has lands on Ask to join instead, or a domain rule signs them up (Domain access and join requests).	Generate a Linking link.
Someone with an account lands on Ask to join	No account has their e-mail as the provider sends it: outside the letters A to Z, case counts (`Álvaro@acme.com` isn't `álvaro@acme.com`), and an account without an e-mail is never found by it.	Fix the account's e-mail (in lower case) or generate a Linking link, and deny the request.
"More than one kubelatch account has your e-mail address. Ask an administrator to fix it." (`ambiguous_account`)	Two or more enabled accounts have that e-mail. The `login.failure` event lists them in `candidates`.	Change the e-mail of the ones that aren't the person's, or disable them.
"Your kubelatch account is linked to another {name} identity. Ask an administrator to unlink it." (`identity_mismatch`)	The account with that e-mail is linked to another identity of the provider: the person's account at the provider was recreated, or the address now belongs to someone else.	If it's the same person, Unlink {name} on the account (Unlink the OIDC identity) and they link again on their next sign-in. If the address went to someone else, don't unlink: disable the previous owner's account and create one for the newcomer.
"That {name} account is already linked to another kubelatch account." (`oidc_taken`)	That identity already opens another kubelatch account.	Unlink {name} on the other account first.

CI with GitHub Actions¶

Symptom	Cause	Fix
The exchange responds `401 invalid token`	The `id_token` fails verification: signature, `iss`, `aud` different from `KUBELATCH_BASE_URL`, expired, or `jti` already redeemed.	The reason is in the log (`ci: token rejected reason=…`, or `ci: token replayed` if the `jti` was already redeemed) and in the control plane (`ci.exchange.failure`). Common causes: the action's `url` (or `KUBELATCH_URL`, without the action) isn't the server's `KUBELATCH_BASE_URL`, the token was redeemed twice, or the server's clock is off by more than 30 s.
`403 no trust rule matches this workflow …`	There's no rule for that repository, ref, and environment.	The message carries `repository_owner_id`, `repository_id`, `ref`, and `environment`: create the rule with those values, or drop the ref or the environment.
`403` naming the bot	The bot is disabled or has no active permissions.	Enable it or grant it permissions.
`429`	More than 60 exchanges per minute from the same IP.	Reduce the frequency.
`could not reach kubelatch` in the workflow (`curl: (7)` or `(35)` without the action)	The runner can't reach kubelatch over https, or doesn't trust its certificate.	You need a name reachable from the runners and a certificate they trust: with a private CA, its certificate in the runner's system store or in `NODE_EXTRA_CA_CERTS` in the job's `env`. `KUBELATCH_KUBECONFIG_CA` then lets the job's later steps trust it too.

The server log¶

kubelatch writes one JSON object per line on standard output (kubectl logs); with KUBELATCH_LOG_FORMAT=text, key=value pairs. At startup it logs listening with the version, the address and whether TLS and GitHub login are on. With OIDC configured, it also logs OIDC login enabled, with the issuer, whether the client is public, the domains and the link policy, and then OIDC discovery fetched or OIDC discovery failed; sign-ins will retry it.

Every request outside /clusters/ (the API, the probes, the CI exchange and the GitHub webhook) leaves an http request line, except the web UI's files:

{"time":"2026-10-02T17:32:41.767Z","level":"INFO","msg":"http request","method":"GET","route":"GET /api/me","status":200,"duration_ms":3,"bytes":412,"client_ip":"203.0.113.7","subject_id":"01999c1e-5b7a-7d43-9a51-3c2f0e8d4a10","request_id":"01a0fd3f-08a7-744c-8bad-6e34fc981ad1"}

route is the API's route pattern, never the requested path or its query: no id or token in a URL reaches the log.
request_id is the response's X-Request-Id header (the browser's developer tools show it). Every other line that request logs carries it too, such as the cause of a 500 (request failed). kubelatch generates the id; one sent by the client is ignored.
subject_id appears when a session or a CLI token authenticated the request.
/healthz and /readyz log at DEBUG.
The proxy writes no http request line: every kubectl request is already a row in Audit. The lines it logs about a request (proxy: upstream error, proxy: load cluster…) carry audit_id, the response's Audit-ID.
Lockouts and rate limits log at WARN with client_ip and the login (login: account locked, login: rate limited, ci: exchange rate limited…). A password, a code or a token never appears.
A failed OIDC sign-in logs oidc login failed with kind, code (the code of the login page's ?error=) and err, the cause. The provider's tokens, the code it returns and the client secret never appear.

To see more, set config.logLevel: debug in values.yaml (KUBELATCH_LOG_LEVEL) and run helm upgrade: the change rolls the pods. debug adds a line for each probe, so go back to info when you're done. warn keeps only warnings and errors.

What to watch¶

kubelatch doesn't expose /metrics yet. With what's there:

/readyz in your external monitoring: 503 means Postgres is unreachable.
The JSON log (kubectl logs): tls: the served certificate expires soon, tls: reload failed, reconciler: pass failed (with the cluster and the reason), namespace watch down (with the cluster and the cause; namespace watch up when it's back), namespace watch: cannot build the cluster's client, retention: pass failed, membership sync: pass failed, OIDC discovery failed; sign-ins will retry it, oidc login failed, ci: token rejected, login: account locked, login: rate limited.
GET /api/clusters: status and last_error for each cluster (the same as Clusters).
GET /api/oidc/status: discovery.error, the last failed read of the provider's discovery document (the same as Users).
GET /api/control-events: several login.failure events in a row from the same IP.