Stuck deployment

A deployment is stuck when it stays queued, building or deploying and no new deployment_events arrive (alert deployment_stuck, after 30 min). An expired lease nobody reclaims shows up separately as job_lease_orphaned.

Diagnosis

Work from the deployment down. $D is the deployment id.

-- 1. State and the event trail (what was the last thing that happened?)
SELECT status, created_at, updated_at FROM deployments WHERE id = '$D';
SELECT sequence, kind, occurred_at, payload - 'log' AS payload FROM deployment_events WHERE deployment_id = '$D' ORDER BY sequence;

-- 2. Every job, where it is targeted, its lease, and its last error.
SELECT id, kind, status, target_node_id, attempt, max_attempts, available_at, lease_expires_at, payload->>'last_error' AS last_error
FROM jobs WHERE deployment_id = '$D' ORDER BY created_at;

-- 3. Results the agents reported (append-only).
SELECT r.reported_at, j.kind, r.node_id, r.attempt, r.outcome, r.output - 'log' AS output
FROM agent_job_results r JOIN jobs j ON j.id = r.job_id WHERE j.deployment_id = '$D' ORDER BY r.reported_at;

-- 4. Source, artifact, instances, reservations, routes.
SELECT id, verified_at FROM source_snapshots WHERE deployment_id = '$D';
SELECT id, image FROM release_artifacts WHERE deployment_id = '$D';
SELECT id, role_name, status, active_node_id, generation FROM application_instances WHERE deployment_id = '$D';
SELECT a.node_id, a.resource_kind, a.status FROM fleet_allocations a JOIN application_instances i ON i.id = a.resource_id WHERE i.deployment_id = '$D';
SELECT hostname, route_generation, status FROM ingress_analytics_routes WHERE deployment_id = '$D';

-- 5. The nodes involved.
SELECT id, name, pool, status, schedulable, now() - last_seen_at AS heartbeat_age FROM fleet_nodes
WHERE id IN (SELECT target_node_id FROM jobs WHERE deployment_id = '$D');

Failing a deployment safely

The user path is pl down <id> (POST /v1/deployments/{id}/teardown). It removes traffic first, then queues fenced teardown jobs to the nodes that hold the instances.

An operator without the tenant’s credentials can only use the database. Use the break-glass steps below, as the owner role, and record why:

• Nothing was ever dispatched (no job has a target_node_id, a lease or a result): mark it failed. The scheduler then stops placing it, the gateway refuses its leases, and the reaper releases never-dispatched reservations, as long as no network attachment exists in the segment yet (reap_unstarted_application):

  BEGIN;
  UPDATE deployments SET status = 'failed', updated_at = now() WHERE id = '$D' AND status IN ('queued','building','deploying');
  INSERT INTO deployment_events(sequence, id, org_id, deployment_id, kind, payload)
    SELECT COALESCE(max(sequence), 0) + 1, gen_random_uuid(), (SELECT org_id FROM deployments WHERE id = '$D'), '$D',
           'operator_failed', jsonb_build_object('reason', '<ticket and reason>')
    FROM deployment_events WHERE deployment_id = '$D';
  COMMIT;

• Something was dispatched (any targeted, leased or reported job, or any instance with an active_node_id): request teardown instead. This is the same state change the API makes. The scheduler then drives the fenced teardown:

  BEGIN;
  UPDATE deployments SET status = 'teardown_queued', updated_at = now() WHERE id = '$D' AND status NOT IN ('teardown_queued','destroyed');
  INSERT INTO deployment_events(sequence, id, org_id, deployment_id, kind, payload)
    SELECT COALESCE(max(sequence), 0) + 1, gen_random_uuid(), (SELECT org_id FROM deployments WHERE id = '$D'), '$D',
           'teardown_requested', jsonb_build_object('operator', true, 'reason', '<ticket and reason>')
    FROM deployment_events WHERE deployment_id = '$D';
  COMMIT;

  Teardown jobs go to the instance’s active_node_id. If that node is dead, they wait for it (see the table).

Verification

The deployment reaches failed, or teardown_queued and then every instance removed. deployment_stuck clears, and capacity on the node drops (pushleft-controller fleet list).

Do not

• Do not set jobs.target_node_id, status, attempt or lease_* by hand. Targeting is the scheduler’s admission decision, and the lease token is the fence for every result.
• Do not mark application_instances removed, or release fleet_allocations, by hand. Only an accepted teardown result may, because a container may still be running.
• Do not delete events, jobs or results. The triggers refuse it, and they are the audit trail.