Secret rotation

When each secret is read decides whether a rotation needs a restart. The table was checked against the code.

Secret	Read by	When it is read	Rotation needs
`PUSHLEFT_DEV_TOKEN` (development mode only)	API	startup	API restart
API tokens and console sessions (production)	API	per request, by hash	nothing; revocation applies to the next request
Registry password file `PUSHLEFT_BUILD_REGISTRY_PASSWORD_FILE`	gateway	on every build lease (`attach_registry_auth`)	no gateway restart
same file	scheduler	startup (`Retention::from_env`)	scheduler restart
Vault token file `PUSHLEFT_VAULT_TOKEN_FILE`	agent, mesh	startup (`VaultClient::new`)	agent or mesh restart
Cloudflare token	controller (env value), mesh (file)	startup	restart
`DATABASE_URL` passwords	every service	startup (pool)	restart
Gateway/telemetry TLS keys, agent CA key	gateways	startup	restart (CA/key compromise)

Development token

Development mode is loopback-only and must never be the production mode. To rotate the token in a checkout:

rm .env.herdr && scripts/dev setup      # writes a new 0600 token; the old value is never printed
# restart `scripts/dev api` and any shell that exported PUSHLEFT_TOKEN

The token was once tracked in git history; it was rotated and the history rewritten on 17 October 2026 before the repository ever had a remote (claude-handoff.md). If .env.herdr ever shows up in a diff again, rotate it in every checkout before any push.

Production API tokens

# list (admins see all tokens of the org), revoke, issue a replacement
curl -fsS -H "Authorization: Bearer $ADMIN" https://api.pushleft.dev/v1/auth/tokens
curl -fsS -X DELETE -H "Authorization: Bearer $ADMIN" -H 'X-Pushleft-CSRF: 1' https://api.pushleft.dev/v1/auth/tokens/<id>
curl -fsS -H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' -d '{"name":"ci","service":true,"role":"member"}' https://api.pushleft.dev/v1/auth/tokens

Revocation is checked on the next request, and SSE streams close within about 750 ms. Lost owner token: DATABASE_URL=$OWNER_URL pushleft-api bootstrap-owner --email <owner> reissues a token for that same owner (authentication.md). Each revocation leaves a token_revoked row in auth_events.

Registry credential

This is one operator credential shared by every build lease (unscoped; build-executor.md).

Add a second user at the registry (htpasswd supports several). Keep the old one.
Write the new password to both files, owned by pushleft, mode 0600: /etc/pushleft/agent-gateway/registry-password and /etc/pushleft/scheduler/registry-password. If the username changes, also update PUSHLEFT_BUILD_REGISTRY_USERNAME in both env files and restart the gateway.
systemctl restart pushleft-scheduler. The gateway hands the new password to the next build lease without a restart.
Wait at least 30 minutes, the build timeout: leases issued before the switch carry the old password in their payload until they push. Then remove the old user at the registry.
Verify: a test pl up reaches build_completed, and the next retention pass records image_release_status = 'deleted' for an eligible artifact.

Vault/OpenBao token

Issue the new token with the same policy. Write it atomically to each host’s PUSHLEFT_VAULT_TOKEN_FILE (root 0600): agents /etc/pushleft/credentials/vault-token, mesh /etc/pushleft/vault/mesh-token.
Restart: systemctl restart pushleft-mesh on the mesh host, then systemctl restart pushleft-agent node by node. An agent restart drops its in-flight execution, the same as certificate renewal (fleet.md). Their leases expire and retry. Do this one node at a time, and check fleet list between nodes.
Revoke the old token in Vault only after every host has restarted.
Verify: on a database node, a manage_database health job succeeds, and pushleft-mesh logs no Vault 403.

Database passwords

ALTER ROLE pushleft_<service> PASSWORD '...', update DATABASE_URL in /etc/pushleft/<service>.env, and restart that one service. Existing connections keep working until the restart.

Do not

Do not put secret values into jobs.payload, deployment_events or results. The gateway rejects secret-shaped result fields; nothing else checks.
Do not loosen file modes to share a file between users. The binaries refuse group- or world-readable key files by design.