Fleet substrate

Private operator tooling for Pushleft-managed hosts: Hetzner provisioning, agent enrollment, short-lived certificates and rotation, signed agent updates, inventory/health/capacity, drains and replacement. None of this is exposed to the developer API or console. Operators use pushleft-controller fleet … with direct database and Hetzner credentials.

Live Hetzner provisioning has not been executed. No credentials were available. Everything below was exercised against a local Hetzner API stub (scripts/fixtures/hetzner_stub.py) built from the official OpenAPI document (https://docs.hetzner.cloud/cloud.spec.json). See First live run.

Components

Provisioning

pushleft-controller fleet provision --pool build|application|database --region fsn1 --type cx32 [--image ubuntu-24.04] [--placement-group NAME]
pushleft-controller fleet provision --resume <node-id>

This is an operator-run CLI workflow, not a queued job. jobs.org_id is mandatory and platform nodes belong to no organization. A controller-only job kind would have needed a schema exception for that. Durability comes from fleet_nodes.provision_state:

1. pending: insert the fleet_nodes row (provisioning, provider='hetzner'). Validate the server type, location, image (matching architecture) and SSH keys. Converge the pool firewall. Generate a 256-bit enrollment secret. Store only its SHA-256 with a 15-minute expiry. Record capacity from the server type, less a fixed host reserve of 0.5 vCPU, 1 GiB and 20 GB. Create the server with cloud-init user_data and the labels pushleft.node_id, pushleft.org=platform and pushleft.pool.
2. server_requested: poll until the server is running. Record the server ID, type, location, image and public/private IPs.
3. awaiting_enrollment: wait until the gateway marks the node enrolled. If the secret expires first, the state becomes failed.
4. enrolled: done. fleet admit <node> then sets schedulable once the node is active, has heartbeated within 60 s and has capacity.

Replay safety: the CLI looks up a server by its pushleft.node_id label before it creates one. A crash after creation therefore adopts the existing server, whose user-data holds the secret already hashed in the row. A name held by a server without our label is a conflict, and the CLI never adopts it. A 429 is retried with Retry-After, or with RateLimit-Reset when RateLimit-Remaining is 0. Idempotent calls also retry 5xx and transport errors, at most 4 attempts with backoff. Server creation never retries blindly. After an ambiguous failure the CLI looks up the label again. The token is read from HETZNER_API_TOKEN, sent only as a bearer header and redacted from Debug and error output.

Configuration (environment):

Cloud-init

The user-data is a bash script, at most 32 KiB, with quoted heredocs. Each value is validated before rendering: UUID, hex secret, https URLs without shell metacharacters, version grammar, base64 and PEM-only CA. The script:

• writes /etc/pushleft/agent.env, the gateway CA, the release public key (SPKI PEM) and the enrollment secret (/var/lib/pushleft/enrollment-secret, 0600);
• installs ca-certificates curl openssl zstd docker.io;
• downloads SHA256SUMS, SHA256SUMS.sig and the release tarball, then verifies them with openssl pkeyutl -verify -rawin and sha256sum -c --strict before it installs /usr/local/bin/pushleft-agent (see Signed updates);
• installs both systemd units, runs pushleft-agent enroll and enables pushleft-agent.service.

The agent service has ConditionPathIsMountPoint=/var/lib/pushleft/data, so it does not start until the operator has attached and mounted the XFS data volume.

Firewall

Host precondition: br_netfilter loaded with net.bridge.bridge-nf-call-iptables=1 (cloud-init writes /etc/modules-load.d/pushleft.conf and /etc/sysctl.d/90-pushleft.conf). Environment rules sit in the inet forward hook, which bridged frames between the mesh gateway’s port and a workload only traverse with br_netfilter. pushleft-agent manage-firewall refuses to apply or verify rules otherwise, the recurring gateway health_check fails, and every heartbeat reports the live value as host.bridge_netfilter. Gap: a module unloaded after admission is reported, not acted on; running environments keep their now-bypassed rules until an operator drains the node (bridge-family rules that do not depend on br_netfilter are the upgrade). Found on native Ubuntu 24.04 with Docker 29, which no longer loads the module itself; the Linux worker suites pass --sysctl net.bridge.bridge-nf-call-iptables=1 and need the module on the host.

Each pool has one Hetzner firewall, pushleft-<pool>. It is attached in the create request, so a server never boots without it. Inbound rules are exact and converged with set_rules:

• 22/tcp from PUSHLEFT_FLEET_OPERATOR_CIDRS only;
• 80/tcp and 443/tcp from anywhere on application (ingress) pool nodes only;
• nothing else. Agent, telemetry and management traffic is outbound: the agent dials the gateway, and Tailscale/Headscale uses the management mesh with outbound NAT traversal or DERP. Outbound traffic is not restricted by the Hetzner firewall.

Enrollment

The gateway serves POST /v1/enroll on a separate listener (PUSHLEFT_AGENT_ENROLL_BIND). It uses TLS with server authentication only and no client certificate. The listener has only that one route. It cannot lease, heartbeat or complete jobs.

Request: {node_id, enrollment_secret, csr_pem}.

1. The CSR is parsed and its signature verified (proof of key possession). Its CN must equal the node ID, and any SANs must be DNS names equal to the node ID. Otherwise the gateway returns 400. The CSR is checked before the secret, so a malformed request does not use up the secret.
2. One UPDATE consumes the secret: matching hash, enrollment_used_at IS NULL, unexpired and node provisioning. The node becomes active with a fresh last_seen_at. An unknown node, a wrong, expired or used secret and a non-provisioning node all return the same 401 enrollment_refused.
3. The gateway signs a client certificate with the operator CA key and inserts it as fleet_node_certificates purpose agent. The certificate has CN and SAN DNS set to the node ID, EKU clientAuth only, KeyUsage digitalSignature, no CA flag and a random serial. The X.509 notBefore is backdated 60 s for replica clock skew. The stored window is the nominal lifetime. The response is {certificate_pem, ca_pem, not_after}.

Gateway configuration:

The agent (pushleft-agent enroll) generates an ECDSA P-256 key locally. The key never leaves the host. The agent stages key and certificate atomically: temp file, fsync, rename, 0600, then directory fsync. It then deletes the secret file. Re-running it with an identity already staged is a no-op. Certificates live in /var/lib/pushleft/pki, because /etc/pushleft is read-only to the agent service.

Rotation

• POST /v1/agent/certificate/renew (mTLS, {csr_pem}) returns a new certificate only when the presenting certificate has less than a third of its stored lifetime left, or when it was not issued by the gateway’s current CA (409 renewal_not_due otherwise). Every issued row records issuer_ca_sha256; certificates from before migration 0037 have none and renew at the next check. The new row records replaces_id. The response carries ca_pem, the gateway’s trust bundle (PUSHLEFT_AGENT_CLIENT_CA).
• The agent’s run loop calls it every PUSHLEFT_AGENT_RENEW_CHECK_SECONDS (default 600). When renewal succeeds, the agent restages key and certificate atomically, stages the bundle as gateway-ca.pem beside the certificate (/var/lib/pushleft/pki), builds a new client, checks it against /healthz and swaps it in place. Every request takes the current client, so in-flight jobs keep their leases and heartbeat with the new certificate. If the check fails, the agent exits instead and systemd (Restart=always) starts it from the staged files; abandoned leases expire and retry under token fencing.
• Once gateway-ca.pem exists it replaces the bootstrap PUSHLEFT_AGENT_CA (read-only under /etc/pushleft). It arrived over an mTLS session the previous trust verified. After a CA rotation the old CA is no longer trusted at the agent’s next renewal.
• On the first lease that presents a certificate issued by renewal, the gateway revokes (revoked_at) every older agent certificate of that node in the same transaction. From then on the old fingerprint gets 401 on every route: lease, heartbeat, complete, source, identity and renew. Until that first lease the old certificate keeps working, so a lost renewal response does not lock the node out. Each later renewal attempt issues a fresh certificate, and the first one used retires all earlier ones.
• pushleft-agent renew-certificate performs one attempt by hand.

A build that is downloading its source when the swap happens keeps the snapshot client for that download; the old certificate stays valid until the node’s next lease.

This covers agent mTLS certificates. Renewal of environment-mesh (Headscale) pre-auth keys is a separate workflow and is unchanged.

CA rotation

The fleet CA (issuer of agent client certificates and of the gateway server certificate) rotates in two phases without a fleet outage. Run the controller on the gateway host, with the gateway’s own file variables and DATABASE_URL:

1. pushleft-controller fleet ca rotate --begin archives the current CA as <file>.previous (key 0600), generates a new ECDSA P-256 CA, writes the bundle as new + old, then the new key and certificate. Records ca_rotation_begun with both CA fingerprints. Every certificate not issued by the new CA is now due for renewal, and the renewal response carries the two-CA bundle, so each agent trusts the new CA before anything switches. The gateway server certificate is still from the old CA.
2. Agents renew at their next check (PUSHLEFT_AGENT_RENEW_CHECK_SECONDS) and swap clients in place. Old-CA certificates keep authenticating during the overlap.
3. pushleft-controller fleet ca rotate --complete is refused while any active, unhealthy or draining node has no unrevoked, unexpired agent certificate from the new CA (it lists them). Otherwise, in one transaction, it revokes every agent certificate not issued by the new CA and records ca_rotation_completed. Then it re-signs the gateway server certificate with the new CA (same common name and DNS/IP names, server-auth only, existing key), trusts only the new CA and deletes the archived old key and certificate. A failure after the commit is finished by running --complete again.
4. pushleft-controller fleet ca rotate --abort restores the archived CA, only while no certificate has been issued by the new one (ca_rotation_aborted).

The gateway re-reads its PKI files every 2 s (no restart, no SIGHUP). A changed set is loaded in full and self-tested before it replaces the running one: the CA key must sign a certificate the CA certificate verifies, and the issuing CA must be in the trust bundle. A half-written set therefore keeps the previous configuration. New handshakes use the new configuration; established connections keep their session. Certificate issuance takes a shared PostgreSQL advisory lock that the rotation phases take exclusively, and refuses to issue from a CA whose rotation was aborted, so an un-reloaded gateway cannot issue from it. Keys never enter the database; fleet_events holds only CA fingerprints and counts.

Limits:

• Nodes provisioned during the overlap get PUSHLEFT_FLEET_GATEWAY_CA_FILE in cloud-init. Point it at the two-CA bundle until --complete, then at the new CA.
• lost nodes do not block --complete, and their old-CA certificates are revoked by it. A lost node that returns after a completed rotation cannot authenticate, so it never receives its cleanup list. Stop its containers by hand or reimage it.
• Telemetry certificates (purpose='telemetry') are not part of this rotation.
• A compromised CA key needs the incident runbook first (gateway CA compromise). A rotation alone does not show which certificates the holder minted.

Lost nodes

The scheduler marks an active, unhealthy or draining application, build or database node lost once its last_seen_at is older than PUSHLEFT_NODE_LOST_AFTER_SECONDS (default 300, minimum 30). Mesh coordinator nodes are excluded. One transaction per node:

1. status='lost', schedulable=false, lost_at, and a node_lost row in fleet_events (unique per node) with silence, fenced job count and kinds, lost and replacement instance counts, released allocations and stranded databases.
2. Every queued or leased agent job targeted at it is fenced: failed, lease token cleared, last_error='target node lost; lease fenced'. Jobs are never retargeted. The gateway answers every later heartbeat or completion from a lost node with 409 lease_conflict (the stale-lease path), and no result row is written.
3. Fenced side effects: a build whose artifact was not yet published fails its deployment (build_failed, code=node_lost). A publishing ingress route becomes failed and is republished once the replacement is ready.
4. Its application instances (deploying, ready, error, draining) become lost. That status is terminal; rows are never deleted. Reserved-only instances go back to pending_placement.
5. Its deployment_role and network_attachment allocations are released. Subnet slots (network_attachment_requests) are kept and never reused.
6. Each lost instance of an active deployment gets a replacement row: same role slot, generation + 1, replaces_instance_id, and a normal ensure_application job. Placement then follows the usual rules: a fresh, schedulable node in the deployment’s regions, a network attachment request on that node, and fresh network admission evidence before the job is targeted. With a sealed environment topology only existing member nodes qualify; otherwise the placement is rejected. Release and job roles that already ran are not run again.
7. A live deployment stays live while it waits. If no eligible application node exists, or a replacement is rejected or fails, it becomes degraded with a deployment_degraded event. It returns to live (deployment_recovered) when its long-lived roles are ready and no route on a lost node is still active.
8. Database instances on the node become error with a database_instance_stranded fleet event. Their allocations stay held and nothing moves: the data is on that disk.

Ingress: a live deployment keeps its active route on the lost node until every primary web instance (including the replacement) is ready on an active allocation of a healthy node. Then the scheduler publishes a relocation route (relocation=true, the hostname’s next generation) through the normal publish_ingress path. Only when that route activates does the gateway move the lost node’s route for that hostname to draining (ingress_route_relocated); the deployment is live again only when no hostname is still served from a lost node and every long-lived role is ready. A failed relocation leaves the old route, degrades the deployment and is retried per hostname after five minutes, at most three failures. The wildcard DNS record is operator-managed and must point at the new ingress host (runbook).

A lost node that comes back authenticates, but every lease returns 403 node_lost with a cleanup list: one TeardownApplicationTask per instance fenced away from it (at most 64 per response). The agent runs the normal fenced teardown for each (ownership labels and generation checked), so containers of fenced generations are removed. This is idempotent: the list repeats on every lease, and a removed container is simply not found again. The node never leases again. Decommission it (fleet decommission accepts lost). It refuses while a stranded database allocation is held.

pushleft-controller fleet list shows lost nodes with their reclamation result. pushleft-controller fleet events lists recent fleet events.

Signed agent updates

Release layout under <release_url>/<version>/:

• pushleft-agent-<version>-linux-<amd64|arm64>.tar.zst, containing exactly one regular file, pushleft-agent (build with tar --zstd -cf … pushleft-agent);
• SHA256SUMS in sha256sum format;
• SHA256SUMS.sig: base64 of the raw Ed25519 signature over the exact SHA256SUMS bytes.

Signing (operator, offline key):

openssl genpkey -algorithm ed25519 -out release.key            # once; keep offline
openssl pkey -in release.key -pubout -outform DER | tail -c 32 | base64   # PUSHLEFT_RELEASE_PUBKEY
sha256sum pushleft-agent-0.2.0-linux-*.tar.zst > SHA256SUMS
openssl pkeyutl -sign -rawin -inkey release.key -in SHA256SUMS | base64 > SHA256SUMS.sig

pushleft-agent self-update (run by the oneshot pushleft-agent-update.service after the operator changes PUSHLEFT_AGENT_VERSION in agent.env):

1. Downloads all three files over HTTPS, with bounded sizes.
2. Verifies the signature with the pinned key (ring Ed25519, already in the lock through rustls).
3. Finds the exact file name for the requested version in the signed sums. The version is part of each signed name, so an old signed release cannot be served as a newer version.
4. Checks SHA-256, then extracts only pushleft-agent. Any other entry, link or path is refused.
5. Writes it next to the target (fsync, 0755), runs --version and renames it over /usr/local/bin/pushleft-agent.
6. With --restart, runs systemctl restart pushleft-agent.

pushleft-agent.service keeps /usr read-only. The update unit gets ReadWritePaths=/usr/local/bin. --release-ca / PUSHLEFT_RELEASE_CA_FILE adds a trust root for a private release mirror.

Downgrades are possible by explicit operator choice of version. No rollback marker or update rollout controller exists yet.

Inventory, health, capacity

• Every lease request carries capabilities.host: cpu_count, memory_mb, data_disk_free_gb (on /var/lib/pushleft/data), kernel and docker_version, plus agent_version. The gateway stores these, bounded, in fleet_nodes.host_* with host_facts_at. The Docker version is probed once per agent process.
• last_seen_at is refreshed by leases and heartbeats, as before. Schedulers keep their 60 s freshness rule.
• pushleft-controller fleet sweep marks active nodes whose last_seen_at is older than PUSHLEFT_NODE_UNHEALTHY_AFTER (seconds, default 90) as unhealthy. The controller daemon runs it every 2 s. Allocations are not released, because moving work is drain’s job. Leases are refused while unhealthy. The node’s next authenticated lease returns it to active.
• pushleft-controller fleet list prints pool, region, status, schedulable, CPU, memory and disk used/total from active fleet_allocations, heartbeat age and agent version, then one line per lost node: fenced jobs, lost instances, replacements ready and queued, stranded database instances.
• After PUSHLEFT_NODE_LOST_AFTER_SECONDS without a lease or heartbeat the scheduler marks the node lost and reclaims its work (Lost nodes).

Drain, decommission, replacement

• fleet drain <node>: status='draining', schedulable=false. The gateway already refuses leases to non-active nodes (403). Heartbeats and completions of already-leased jobs continue. The command reports active allocations and live leases.
• fleet decommission <node> [--wait SECONDS]: waits for zero active allocations and zero live agent leases, then refuses unless the node is draining, provisioning, offline, revoked or already retired. In one transaction holding fleet_nodes … FOR UPDATE (which conflicts with the FK lock a new allocation takes), it checks allocations again, revokes every certificate (all purposes) and sets retired/retired_at. Only after commit does it delete the Hetzner server, by recorded ID or label lookup. A failed delete is retried by running the command again. Rows, certificates, jobs and results are kept. The CLI never deletes a server whose node still has an active allocation.
• fleet replace <node>: provisions a like-for-like node (same pool, region, type and image), waits for enrollment, then drains the old node. Moving allocations off a draining node is not automated. Only a node that goes silent is reclaimed (Lost nodes). Database relocation is a separate workflow. Decommission the old node once fleet list shows it empty.

What is manual

• Hetzner project, API token, SSH keys and the operator CA/gateway certificates.
• Attaching and mounting the XFS data volume at /var/lib/pushleft/data, the Vault token file and joining the management tailnet (pushleft-agent join-management-mesh). The agent service does not start before these are done.
• fleet admit, after the operator has checked the bootstrap (pushleft-agent doctor).
• Releasing or relocating allocations of a drained (still heartbeating) node before decommission. Silent nodes are reclaimed automatically; their databases are not.
• Pointing the wildcard ingress DNS record at the replacement ingress host after a node is lost.
• Building, signing and publishing agent releases. Starting pushleft-agent-update.service per node.
• Starting each CA rotation phase (fleet ca rotate --begin, then --complete) on the gateway host.

Threat model

• Stolen enrollment secret. It works once, for 15 minutes, for one node ID in provisioning. The plaintext exists only in the provisioning process memory, the Hetzner user-data and the 0600 file the agent deletes. Any process on the new VM can read the Hetzner metadata service until the secret is used. The database stores only the hash. Brute force is impractical (256 bits).
• Stolen agent key. The certificate has a 24 h lifetime by default and is revoked by revoked_at. It is retired automatically at the next rotation. fleet decommission or a manual revoked_at takes effect on the next request, as before. A node can renew only for its own ID.
• CA key compromise. An attacker can mint certificates, but every certificate must also exist as an unrevoked fleet_node_certificates row, so the CA key alone does not authenticate. Keep the CA key on the gateway host only, 0600. An HSM/KMS signer is the upgrade.
• Malicious release mirror or MITM. Without the offline release key it cannot pass signature, checksum and single-file extraction. A replayed old release cannot be renamed to a newer version.
• Enrollment listener abuse. It is unauthenticated by design but has one route. A bad CSR is refused before any database write, and refusals do not reveal whether a node exists. It is not rate-limited yet. Keep it reachable only from the provider network or put it behind a proxy limit.
• Hetzner token. It has full project access. It is never stored in the database or logged. Run fleet commands from an operator host, not from the developer API.

First live run

Verify, in a throwaway Hetzner project:

1. fleet provision creates exactly one server with the three labels, the pushleft-<pool> firewall attached at boot, and the expected type, location and image. A second --resume creates nothing.
2. Hetzner accepts the user-data. Check /var/log/cloud-init-output.log: the apt install, release download, openssl pkeyutl -verify -rawin on Ubuntu 24.04’s OpenSSL, sha256sum -c and tar --zstd all succeed, and enrollment completes within the 15-minute TTL.
3. The firewall rules as shown in the console. SSH from outside the operator CIDR fails. 80/443 are open only on application nodes. Nothing else answers.
4. Recorded public IPv4/IPv6 and server ID match the console. The capacity reserve is plausible against real Docker/agent usage.
5. After mounting the data volume and joining the mesh, the agent heartbeats and host facts are populated. Rotation with a short PUSHLEFT_AGENT_CERT_LIFETIME_SECONDS restarts the service, and the old fingerprint then gets 401.
6. Signed self-update on the node, including the systemd unit sandbox (ReadWritePaths=/usr/local/bin).
7. Rate-limit headers on a real 429, if one can be provoked safely, and the error codes for quota (resource_limit_exceeded) and a sold-out type (resource_unavailable).
8. fleet decommission deletes the server and leaves the node retired with all certificates revoked.

Verification

npm run test:fleet runs the real gateway with a disposable database and an ephemeral EC CA, a stub Hetzner API and an HTTPS release server. It checks:

• a group-readable CA key is refused;
• the enrollment listener accepts TLS without a client certificate, and agent routes still reject such clients;
• the provision CLI creates the server with only operator SSH on a build node, and stores only the secret hash;
• a real agent enroll succeeds and the secret cannot be used twice;
• leases record host facts, and renewal before the two-thirds point is refused;
• after rotation the old certificate works until the new one is used, then gets 401;
• the sweep marks the node unhealthy and the next lease recovers it;
• admit uses the server-type capacity and list shows used/total;
• a draining node gets 403, and decommission refuses while an allocation is active, then retires and revokes the node and deletes the server;
• the openssl-signed release is accepted by both the cloud-init command (using the controller-rendered PEM) and self-update, and a foreign key or tampered archive is refused.

The enroll::tests::enrollment_rotation_and_revocation database test (run by scripts/mesh-controller-e2e.sh) covers the same rules at handler level, plus a revoked current certificate and a retired node.

npm run test:fleet-reclaim runs:

• database tests: reclaim::tests (silence below and above the threshold, one node_lost event across replays, fenced leases, released capacity, lost rows kept and terminal, one replacement per lost instance at generation + 1, no target before admission on the new node, relocation only after the replacement is ready, degraded without capacity and back to live, database instance stranded with its allocation held); gateway lost_node_completion_is_stale_and_its_lease_returns_cleanup, relocation_drains_the_lost_route_only_once_active_and_recovers_live, failed_relocation_degrades_a_live_deployment, ca_rotation_overlap_renews_onto_the_new_ca; controller ca::tests (begin, abort, refused complete, refused abort, complete, re-signed server certificate, phase events) on a dedicated database;
• live, with a real scheduler, mTLS gateway, two pushleft-agent run processes and Docker (network admission stubbed in SQL, no Caddy): agent A runs the web instance and is SIGKILLed; after 30 s the node is lost, agent B runs the replacement at generation 2 and the deployment stays live; a restarted agent A gets 409 on a late heartbeat, 403 on lease, and removes its fenced container while B’s keeps serving;
• live CA rotation: --begin, gateway reload without restart, agent B renews onto the new CA and swaps its client in place (same PID), --abort and --complete refusals, --complete, the gateway serves a new-CA server certificate, an old-CA client certificate fails the handshake, and agent B keeps leasing throughout.