Node lost

The scheduler marks a node lost when it has not leased or heartbeated for PUSHLEFT_NODE_LOST_AFTER_SECONDS (default 300, minimum 30) and reclaims its work (fleet.md). This runbook covers what is automatic, what is not, and what to check.

Symptoms

• node_heartbeat_stale, then pushleft-controller fleet list shows the node as lost with a line such as lost app-3 since …: fenced 4 job(s), 3 instance(s) lost, replacements 2 ready / 1 queued, 0 database instance(s) stranded.
• Deployment events node_lost, then application_ready for the replacement instances, ingress_relocating and ingress_route_relocated. Or deployment_degraded.
• pushleft-controller fleet events shows node_lost (and database_instance_stranded for a database node).

What already happened (automatic)

1. The node is lost and unschedulable. Every queued or leased job targeted at it is failed with last_error='target node lost; lease fenced'. A late heartbeat or completion from it gets 409.
2. Its application instances are lost (rows kept). Each instance of an active deployment has a replacement row (replaces_instance_id, generation + 1) with a normal ensure_application job.
3. Its application and environment-gateway capacity is released. Subnet slots are not reused.
4. A live deployment stays live on its old route. Once the replacement web instances are ready on a healthy node, a relocation route is published there; the old route drains only after it is active.
5. Database instances on the node are error and keep their allocation. Nothing is moved.

Diagnosis

-- The reclamation record.
SELECT created_at, payload FROM fleet_events WHERE node_id = '<node>' AND kind = 'node_lost';

-- Replacements and where they are.
SELECT o.id AS lost, o.role_name, r.id AS replacement, r.generation, r.status, n.name AS node
FROM application_instances o JOIN application_instances r ON r.replaces_instance_id = o.id
LEFT JOIN fleet_nodes n ON n.id = r.active_node_id
WHERE o.active_node_id = '<node>' ORDER BY o.deployment_id, o.role_name;

-- Why a replacement is not placed yet.
SELECT j.id, j.status, j.target_node_id, j.available_at, j.payload->>'last_error' AS error
FROM jobs j JOIN application_instances r ON r.id::text = j.payload->>'application_instance_id'
JOIN application_instances o ON o.id = r.replaces_instance_id
WHERE o.active_node_id = '<node>' AND j.kind = 'ensure_application';

-- Routes still on the lost node.
SELECT deployment_id, hostname, route_generation, status FROM ingress_analytics_routes
WHERE node_id = '<node>' AND status IN ('publishing','active','draining');

A replacement waits (queued, retried every 15 s) until a fresh, schedulable application node in the deployment’s regions has capacity and fresh network admission evidence for the environment. In an environment with a sealed topology only nodes that are already members qualify. Otherwise the placement is rejected (application_placement_rejected) and the deployment is degraded.

Actions

1. Ingress DNS. The wildcard *.<base> record is operator-managed. If the lost node was the ingress host, point the record at the node that now carries the relocation route (SELECT DISTINCT n.name, n.public_ipv4 FROM ingress_analytics_routes r JOIN fleet_nodes n ON n.id = r.node_id WHERE r.relocation AND r.status = 'active').
2. Capacity. If replacements are queued and the deployment is degraded, add capacity: fleet provision --pool application …, fleet admit. Placement continues by itself, and the deployment returns to live when its long-lived roles are ready.
3. Degraded with a failed relocation. A failed relocation (deployment_degraded, reason ingress relocation off a lost node failed) is retried per hostname after five minutes, up to three failures; each hostname drains its lost route only once its own replacement is active. After the third failure automatic relocation stops. Ask the user to run pl up again, which publishes a fresh generation (ingress host failure).
4. Database node. Data is on the lost host’s volume. Recover the host and its volume (then decommission and replace following node drain), or restore from the tenant’s own backups. Pushleft does not relocate database data.
5. The host comes back. It authenticates but is refused work (403 node_lost) and receives a cleanup list; its agent removes the containers of every fenced instance (node was marked lost; removed fenced instance … in the agent journal). Check with docker ps --filter label=com.pushleft.managed=true on the host. It never leases again. Decommission it: pushleft-controller fleet decommission <node> (accepted for lost; refused while a stranded database allocation is held).

Verification

• fleet list: replacements N ready / 0 queued, the deployment live.
• SELECT status FROM ingress_analytics_routes WHERE id = … for the relocation route is active, the lost node’s route is draining.
• curl -sI https://<app>.<base> answers from the new node.

Do not

• Do not set a lost node back to active or edit a lost instance. The status is terminal (a trigger refuses it), and the replacement already owns the role slot.
• Do not retarget the fenced jobs or reuse their lease tokens.
• Do not release a stranded database allocation by hand; the data is still on that disk.
• Do not run fleet ca rotate --complete while a lost node that may come back still runs containers: completion revokes its certificate, and it then cannot receive its cleanup list.