Security incident checklist

Use this for a suspected credential leak, tenant data exposure, a compromised node or control-plane host, or abuse. Work in order. Keep a timestamped log in the ticket. Use the read-only pushleft_ops role for every query until you decide to act.

1. Scope (first 15 minutes)

• What was reported, by whom, and when. Which organization(s), node(s) and credential(s).
• Freeze evidence: take a pg_dump now (restore runbook), and copy the journals of the gateway, API and affected nodes (journalctl -u pushleft-* --since ...).
• Pull the audit trail for each affected organization:

  DATABASE_URL="$OPS_URL" pushleft-api audit --org <org> --since 2026-10-01T00:00:00Z > audit-<org>.jsonl

  It merges auth_events, deployment_events (with actor_user_id/actor_credential_id from migration 0034 on) and agent_job_results, oldest first. Log tails are omitted, and nothing is deleted.

2. Credentials and identity (auth_events)

-- Who logged in, created or revoked tokens, changed memberships, and every failure.
SELECT occurred_at, kind, user_id, detail FROM auth_events WHERE org_id = '<org>' AND occurred_at > '<since>' ORDER BY occurred_at;
-- Failure reasons across all organizations (revoked/expired tokens still being presented, reused login codes).
SELECT detail->>'reason' AS reason, org_id, count(*) FROM auth_events WHERE kind = 'failed_auth' AND occurred_at > now() - interval '24 hours' GROUP BY 1, 2 ORDER BY 3 DESC;
-- Deployments and teardowns attributed to a credential.
SELECT occurred_at, kind, deployment_id FROM deployment_events WHERE actor_credential_id = '<token or session id>' ORDER BY occurred_at;

• Revoke leaked tokens (DELETE /v1/auth/tokens/{id}) or disable the user: UPDATE users SET disabled_at = now() WHERE id = '<user>' ends every credential on the next request.
• Unknown tokens and unknown login codes are logged, not stored (by design: guessing cannot grow the table). Search the API journal for unauthorized bursts. Rate limiting belongs at the proxy; the API has none (gap).
• Limits of the trail: development-mode requests have no actor; events from before migration 0034 have no actor; auth_events.user_id is NULL for actions taken with a service token (gap: the service token id is not recorded there).

3. Nodes and execution (agent_job_results)

-- Everything a node reported, in order. The output is bounded and refuses secret-shaped fields.
SELECT r.reported_at, j.org_id, j.kind, j.deployment_id, r.attempt, r.outcome, r.output - 'log' AS output
FROM agent_job_results r JOIN jobs j ON j.id = r.job_id WHERE r.node_id = '<node>' AND r.reported_at > '<since>' ORDER BY r.reported_at;
-- Which organizations had work on the node (blast radius of a node compromise).
SELECT DISTINCT j.org_id FROM jobs j WHERE j.target_node_id = '<node>';

• Compromised node: revoke its certificates, drain it, replace it (CA/key runbook). Rotate the Vault token if the node held one (secret rotation).
• Build node: assume the registry push credential leaked, because every build lease receives it (build-executor.md). Rotate it, and audit the registry for pushes outside organizations/<org>/projects/<project> repositories.
• Gateway or control-plane host: the agent CA key, the registry password and the database credentials are all on it. Treat all of them as leaked.

4. Tenant data

• Telemetry and analytics in ClickHouse are organization-scoped by the gateway. Raw IPs and full referrers are never stored.
• Managed database contents are not readable by the control plane. Secret values are in Vault, and only references are in PostgreSQL. Check Vault audit logs for the affected paths.

5. Recover and close

• Redeploy affected applications from known-good commits (pl up), then tear down the suspect deployments (pl down).
• Re-run the step 2 and 3 queries after the fix to confirm no new activity from revoked identities (failed_auth with reason = revoked is expected).
• Write the timeline, root cause and customer notification decision in the ticket.

Do not

• Do not delete or edit auth_events, deployment_events or agent_job_results. The triggers refuse it, and they are the evidence.
• Do not grant the API role BYPASSRLS or ownership to “see more”. Use pushleft_ops.
• Do not export audit JSON into tickets that tenants can read. It contains other identities’ ids.