Agent gateway CA, server key or agent key compromise

There are three keys. Identify which one leaked before you act, because the response differs:

Key	Where	What a holder can do
Agent CA key `PUSHLEFT_AGENT_CA_KEY_FILE`	gateway host, 0600	Mint client certificates the TLS layer accepts. These certificates do not authenticate on their own: the gateway also requires the fingerprint to be an unrevoked `fleet_node_certificates` row (fleet.md). If the same CA also signed the gateway server certificate, the holder can impersonate the gateway to agents from a network position, and hand them typed jobs.
Gateway server key `PUSHLEFT_AGENT_GATEWAY_KEY`	gateway host	Impersonate the gateway (and the enrollment listener) to agents, given a network position.
One node’s agent key `/var/lib/pushleft/pki/agent.key`	that node	Act as that node: lease its targeted jobs and report results, until the certificate is revoked or expires (24 h by default).

Symptoms

failed_auth_burst, or unknown fingerprints in the gateway log (401).
Certificates in fleet_node_certificates that no enrollment or renewal you know of explains.
The alert agent_certificate_not_rotating, or a host compromise report for the gateway host or a node.

Diagnosis

-- Every certificate issued recently, with lineage (replaces_id is set by renewal).
SELECT c.id, c.node_id, n.name, c.purpose, c.created_at, c.not_after, c.revoked_at, c.replaces_id
FROM fleet_node_certificates c JOIN fleet_nodes n ON n.id = c.node_id
WHERE c.created_at > now() - interval '7 days' ORDER BY c.created_at DESC;

-- What a suspect node did: every result it reported.
SELECT r.reported_at, j.kind, j.org_id, j.deployment_id, r.outcome, r.output - 'log' AS output
FROM agent_job_results r JOIN jobs j ON j.id = r.job_id
WHERE r.node_id = '<node>' AND r.reported_at > now() - interval '7 days' ORDER BY r.reported_at;

Also check the gateway’s journal for enroll requests: enrollment refusals do not reach the database.

Actions

One node’s agent key

Revoke at once. This takes effect on the node’s next request:

UPDATE fleet_node_certificates SET revoked_at = now() WHERE node_id = '<node>' AND revoked_at IS NULL;

pushleft-controller fleet drain <node>, then pushleft-controller fleet replace <node>. Renewal needs a valid certificate, so a revoked node cannot recover by itself. There is no re-enroll command for an existing node (gap); replacement is the supported path.
Its workloads keep running on the old host. Users redeploy (pl up) so the work lands on healthy nodes. Allocations are not moved (node drain), and fleet decommission refuses while any remain.

Gateway server key (agent CA intact)

Issue a new server certificate from the CA that agents trust (PUSHLEFT_AGENT_CA on nodes). Agents pin the CA, not the key, so nodes need no change.
Install the new files at PUSHLEFT_AGENT_GATEWAY_CERT/_KEY, owned by pushleft, mode 0600. Run systemctl restart pushleft-agent-gateway. Agents reconnect within 5 s.
If you cannot rule out traffic interception, treat every node as suspect and follow the agent-key steps for each.

Agent CA key

Contain. The CA key alone does not authenticate, so first stop new certificates from being recorded. Unset PUSHLEFT_AGENT_CA_KEY_FILE and PUSHLEFT_AGENT_ENROLL_BIND in /etc/pushleft/agent-gateway.env, then run systemctl restart pushleft-agent-gateway. Enrollment and renewal now return 503. Existing certificates keep working until they expire (24 h by default).
Audit recent certificates with the query above. Revoke any you cannot explain. issuer_ca_sha256 names the CA that issued each row.
Restore issuance with the leaked key still in place (set PUSHLEFT_AGENT_CA_KEY_FILE and PUSHLEFT_AGENT_CA_CERT_FILE again, separate from PUSHLEFT_AGENT_CLIENT_CA), then rotate it out with no fleet outage (fleet.md):
```
pushleft-controller fleet ca rotate --begin     # new CA issues; gateway trusts old + new
# wait until every node renewed (default check every 600 s; certificates from the old CA are due at once)
pushleft-controller fleet ca rotate --complete  # refused while a node is still on the old CA
```
--complete revokes every old-CA agent certificate, re-signs the gateway server certificate with the new CA and deletes the old key. The old CA is trusted for the overlap only, and the attacker’s minted certificates still need a database row to authenticate. If you cannot accept even that window, keep issuance off and replace nodes (fleet replace) instead.
If the compromised CA signed the gateway server certificate, agents could have been talking to an impostor during the exposure. --complete moves the server certificate to the new CA, and agents trust only the bundle their last renewal delivered. Treat results reported during the exposure as suspect (query above).
Keep the CA key on the gateway host only, mode 0600. An HSM/KMS signer is the planned upgrade (gap).

Verification

The old fingerprints get 401 on POST /v1/agent/lease.
pushleft-controller fleet list shows only intended nodes heartbeating.
SELECT count(*) FROM fleet_node_certificates WHERE revoked_at IS NULL AND not_after > now() matches the number of live nodes times their purposes (agent, telemetry).

Do not

Do not delete certificate rows. Revocation is revoked_at, and the rows are evidence.
Do not reuse a node ID or hand-edit fleet_nodes.status back to provisioning to force a re-enrollment. The enrollment secret flow, and its single-use guarantee, live in fleet provision.
Do not disable mTLS or client-CA checking to keep the fleet running.