Caddy ingress host failure

Public traffic enters through the host Caddy on the application node that runs a deployment’s web instances. The agent writes each route into the operator bucket pushleft-ingress-router, with route ID pl-ingress-<hostname>-g<generation>. The durable record is ingress_analytics_routes (one row per hostname, node and generation, with the exact task). The wildcard *.<base> DNS record points at the ingress host’s public address. That record is operator-managed (architecture.md “Ingress publication”).

Symptoms

• ingress_publication_stuck or ingress_route_failed, node_heartbeat_stale for the application node, or external probes of https://<app>.<base> failing while /readyz is green.
• Caddy down on a live node: TLS or connection errors on 443 for every hostname on that node.

Diagnosis

-- Routes on the suspect node, newest generation first.
SELECT hostname, route_generation, status, deployment_id, created_at, activated_at, url
FROM ingress_analytics_routes WHERE node_id = '<node>' AND status IN ('publishing','active','draining')
ORDER BY hostname, route_generation DESC;

-- Is the node itself alive?
SELECT name, status, schedulable, now() - last_seen_at AS heartbeat_age FROM fleet_nodes WHERE id = '<node>';

On the host, curl -s http://127.0.0.1:2019/id/pushleft-ingress-router | jq '.handle[0].routes[]."@id"' lists the routes Caddy holds. Compare them with the active rows above.

Case A: Caddy restarted or lost its config, node healthy

Routes are pushed once per publication. There is no periodic route reconciliation (gap), so a Caddy that restarts with an empty config serves 404 for every Pushleft hostname, even though the database says active.

1. Restart Caddy with its last autosaved config: caddy run --resume, or the distribution’s unit configured with --resume. Caddy autosaves admin-API changes, so this restores the routes it had.
2. Check that the router bucket and the terminal 404 route exist, and that route IDs match the active rows.
3. If the autosave is gone, there is no supported republish command (gap). Each affected deployment must be redeployed (pl up), which publishes a new generation. Do not hand-write routes from ingress_analytics_routes.task into Caddy: the agent’s ETag compare-and-swap, the label checks and the readback are the only checks that the upstream is the right container.

Case B: the application node is lost

Automatic since migration 0037: see node lost. After PUSHLEFT_NODE_LOST_AFTER_SECONDS the scheduler replaces the web instances on another node and publishes a relocation route there; the old route drains only when the new one is active. You still repoint the wildcard DNS record (step 3 below). The manual redeploy below remains the fallback when the deployment is degraded (no eligible node, or the relocation failed):

1. pushleft-controller fleet drain <node>, so nothing new is placed there.
2. Provision and admit a replacement application node (fleet provision --pool application ..., then fleet admit). Give it a host Caddy configured as in architecture.md (router bucket, terminal 404, wildcard certificate, admin on loopback).
3. Point the wildcard *.<base> A/AAAA record at the replacement’s public address in Cloudflare. Pushleft publishes no per-deployment DNS.
4. Ask affected users to run pl up again. List them with the query below. The new deployment reserves capacity on a healthy node, publishes the next route generation for the same hostname, and supersedes the old deployment. Hostname ownership is per environment, so the same hostname is accepted.
5. The old routes and instances on the dead node stay active/draining, and their remove_ingress/stop_application jobs stay targeted at that node. Only that node can lease them. Leave them. They do not block the new deployment. fleet decommission refuses while the node’s allocations stay active, so the dead node remains draining until those allocations are released by a fenced teardown (gap for an unrecoverable host).

SELECT DISTINCT r.org_id, d.id AS deployment_id, r.hostname
FROM ingress_analytics_routes r JOIN deployments d ON d.id = r.deployment_id
WHERE r.node_id = '<node>' AND r.status = 'active';

What to tell users

The server hosting your application’s public route failed. Run pl up again from the same commit to move it to a healthy server; the URL stays the same. Data in managed databases is not on this server.

Verification

• curl -sI https://<app>.<base> returns the application’s response from the new node.
• SELECT status FROM ingress_analytics_routes WHERE hostname = '<app>.<base>' ORDER BY route_generation DESC LIMIT 1 returns active for the new node.

Do not

• Do not edit ingress_analytics_routes.status or node_id. A trigger refuses identity rewrites and backward moves.
• Do not insert into ingress_hostnames to free or move a hostname.
• Do not enable on-demand TLS, or point the wildcard at a host without the terminal 404 route.