Node drain and replacement

Operator CLI: pushleft-controller fleet list|drain|replace|decommission|admit (fleet.md). Run it from an operator host with DATABASE_URL set to pushleft_controller, and HETZNER_API_TOKEN set for replace/decommission.

Symptoms

node_heartbeat_stale, node_data_disk_low, node_capacity_high, a host alert from the provider, or planned maintenance.

Diagnosis

pushleft-controller fleet list        # pool, status, schedulable, CPU/memory/disk used/total, heartbeat age, agent version

-- What lives on the node.
SELECT a.resource_kind, a.status, count(*) FROM fleet_allocations a WHERE a.node_id = '<node>' AND a.status IN ('reserved','active') GROUP BY 1, 2;
SELECT i.org_id, i.deployment_id, i.role_name, i.status FROM application_instances i WHERE i.active_node_id = '<node>' AND i.status <> 'removed';
SELECT j.id, j.kind, j.status, j.lease_expires_at FROM jobs j WHERE j.target_node_id = '<node>' AND j.status IN ('queued','leased');

Actions

Drain: pushleft-controller fleet drain <node>. This sets draining and schedulable=false. The gateway refuses new leases from a non-active node (403). Heartbeats and completions of jobs already leased continue. The command prints the active allocations and live leases.
Replace: pushleft-controller fleet replace <node>. This provisions a like-for-like node, waits for enrollment, and drains the old node. Then mount the data volume, join the management mesh, check with pushleft-agent doctor, and run pushleft-controller fleet admit <new-node>.
Move the work. Allocations are not relocated (gap):
- Application instances: users redeploy (pl up), which places the new deployment on schedulable nodes. Then they tear down the old one (pl down).
- Managed databases: relocation is not implemented (managed-databases.md). Do not drain a database node that holds live instances unless you are prepared to restore from the tenant’s own backups.
Known trap: draining blocks teardown. A draining node cannot lease. The teardown_application, remove_ingress and stop_application jobs targeted at it therefore wait forever, its allocations stay active, and fleet decommission keeps refusing. The only lever today is break-glass and must be recorded in the ticket: return the node to active while keeping it unschedulable, so that it can lease the work already targeted at it, and nothing new is placed:
```
UPDATE fleet_nodes SET status = 'active', updated_at = now() WHERE id = '<node>' AND status = 'draining' AND NOT schedulable;
```
This also lets it lease any queued retries of its existing instances. Watch fleet list until the allocations reach zero, then drain again. A drain mode that still leases teardown work is a gap.
Decommission: pushleft-controller fleet decommission <node> --wait 600. It waits for zero allocations and zero live leases, revokes every certificate and marks the node retired, then deletes the server. A failed provider delete is retried by running it again.

A node that is gone for good

A node that stops leasing and heartbeating is marked lost after PUSHLEFT_NODE_LOST_AFTER_SECONDS (default 300), whether it was active, unhealthy or draining. Its leases are fenced, its application capacity is released and its instances are replaced elsewhere (node lost). Database allocations stay held. If you also want a revived disk unable to act as the node, revoke its certificates (it then also stops receiving its cleanup list):

UPDATE fleet_node_certificates SET revoked_at = now() WHERE node_id = '<node>' AND revoked_at IS NULL;

Verification

fleet list shows the replacement active and heartbeating, with the old node at zero used capacity and then retired. node_heartbeat_stale clears.

Do not

Do not set fleet_allocations.status = 'released' or delete allocations by hand. Capacity accounting and subnet non-reuse depend on them.
Do not retarget jobs to the replacement node. A job’s identity, lease token and generation are bound to its node.
Do not delete fleet_nodes rows. Retirement keeps history.