Agent identity and job leases

pushleft-agent-gateway is a separate, agent-only HTTPS surface. It does not share the browser API’s authentication or listener. TLS client authentication is mandatory at handshake time; there is no header-based fallback.

Identity

The gateway validates the client chain against PUSHLEFT_AGENT_CLIENT_CA, hashes the leaf certificate DER with SHA-256, and resolves that fingerprint to an active database record. This second lookup provides immediate revocation without waiting for certificate expiry. A certificate must be inside both its X.509 validity window and its control-plane validity window.

Fleet nodes may be platform-owned (owner_org_id IS NULL) or customer-owned. A customer-owned node can lease only jobs for its owning organization. A platform node can lease only work explicitly assigned to it by the placement controller.

Authentication holds a shared certificate lock and a node NO KEY UPDATE lock through the handler transaction, then checks certificate validity again using a fresh post-wait snapshot. Revocation and node reassignment cannot commit between authorization and lease publication. Heartbeats and completions also enforce current node ownership against the job’s organization. The migrated database regression suite exercises concurrent ownership/status/certificate changes and revocation waiting for request commit. Fleet reassignment still needs the explicit operational identity-rotation/drain workflow; changing ownership metadata is not a host-wipe mechanism.

Gateway transactions require PostgreSQL 17 or newer and use request-local limits: 1 second per lock wait, 3 seconds per statement, and 10 seconds for the entire transaction. A completion blocked by provider-held attachment locks rolls back promptly, releasing its authorization locks so another heartbeat can proceed. Contention is retryable; no partial result is committed. A real-database regression holds the provider lock while verifying that an authenticated heartbeat remains available after the bounded rollback.

The gateway needs a dedicated PostgreSQL service role with BYPASSRLS because it operates across organizations. Startup fails if the current role is neither a superuser nor BYPASSRLS. The browser API must use a different, non-owner, non-bypass role.

Lease protocol

POST /v1/agent/lease updates bounded node capabilities and atomically leases one queued, node-targeted job with FOR UPDATE SKIP LOCKED.
A random lease token and 45-second expiry fence every mutation.
POST /v1/agent/jobs/{id}/heartbeat extends an unexpired lease. A single attempt cannot be extended past four hours.
The agent cancels its local child process when it loses the lease or cannot renew safely. Docker child processes use kill-on-drop.
POST /v1/agent/jobs/{id}/complete accepts one result for the current node, token, and attempt. Stale completion returns 409.
Retryable failures return to the queue with bounded exponential backoff. Terminal results set completed_at.

Results are append-only, limited to 64 KiB, and reject recursively nested fields whose names suggest passwords, secrets, tokens, credentials, or authorization data. Raw logs use the telemetry/log stream rather than job results.

Network jobs also bind the attachment’s credential revision. Missing credential_revision means legacy revision 1; explicit null or malformed values are not accepted as revision 1. Selection and post-wait lease publication both check the current revision. A superseded heartbeat or completion returns 409 without inserting a result or changing attachment evidence. Queued superseded jobs are terminalized, and exhausted/stale failures cannot erase newer health. This is a database publication fence, not proof that an already dispatched remote Docker operation has stopped; automatic credential renewal remains disabled pending agent-side restaging/quiescence verification.

The agent runs two independent serial execution lanes. Lease requests with lane: network select only manage_network jobs; lane: workload excludes them. Both expired-job cleanup and new lease selection respect the lane. Network polling refreshes node liveness without overwriting the workload lane’s capabilities, and does not run storage quota probes. A database command waiting on Docker therefore does not stop recurring mesh checks. Network work remains serial per agent, with existing environment filesystem locks and generation fences intact. Each executing job independently renews its own lease. Omitted lane defaults to all for older clients; unknown values are rejected. Upgrade the gateway before deploying the dual-lane agent.

Completion and transport errors stay local to the attempt: the lane logs the failure and resumes polling, leaving the unreported lease to expire/retry under its existing token fencing. It does not cancel the other lane’s host operation. A single outer shutdown handler (SIGINT or SIGTERM, crates/pushleft-ops) cancels both lanes, including their in-flight child commands and polling/backoff waits; lane-local error recovery cannot swallow shutdown. Fatal malformed/misrouted lease responses can still stop the agent.

The integration regression blocks a database Docker command behind an explicit test gate, runs a network health attempt through the same authenticated agent, verifies its failure result arrives while the database remains leased, then injects HTTP 500 during a second network completion and checks the database lane stays alive and later completes after release. A second gated run verifies SIGINT exits promptly without releasing the database gate. This proves independent dispatch, reporting, failure isolation, and process shutdown, not successful real mesh health or fleet-scale throughput.

Build-source delivery

POST /v1/agent/jobs/{id}/source with {"lease_token": "<token>"} returns the exact verified source archive for the caller’s current targeted build_image lease, and nothing else. It is served only on the mandatory-mTLS listener.

200: Content-Type: application/vnd.pushleft.source+tar.zstd, exact Content-Length, X-Pushleft-Source-Sha256, X-Pushleft-Source-Snapshot and X-Pushleft-Artifact-Id; the body is the archive bytes.
404: any authorization failure. Another node, another organization’s node, a non-build_image job, a non-build-pool node or one without oci_build/isolated_build capabilities, an inactive/torn-down deployment, or any claim/snapshot/artifact/plan provenance mismatch. Existence of a job that is not the caller’s is not disclosed.
409: the job is targeted to this node but the lease token, leased status or wall-clock expiry (clock_timestamp()) is no longer current.
503: source delivery is not configured (PUSHLEFT_SOURCE_ROOT unset; the gateway logs that delivery is disabled at startup) or all download slots are busy.
401: an unknown/revoked certificate or revoked node, as on every other gateway route.
500: the stored object is missing, not a private regular file, truncated, or its bytes differ from the verified digest. No bytes are returned.

Authorization runs in a bounded transaction (certificate, node, job, deployment locks in the usual order), captures the node, object key, digest, length, snapshot and artifact IDs, and commits. The object is then read and verified in blocking work outside every database lock (SourceStore::read_verified: store key grammar only, no symlinks or hard links, exact length and SHA-256). A second bounded transaction repeats the full authorization; any difference withholds the bytes. Concurrent downloads are bounded by PUSHLEFT_GATEWAY_SOURCE_SLOTS (default 2, 1 to 64) per gateway process. A slot is held by the blocking read even if the request is cancelled, and by the response body until it has been sent or the connection drops, so buffered archives stay bounded at slots x 64 MiB.

The gateway reads the same private local store the API writes. Its service identity must therefore be able to read that directory tree (0700 directories). The final reauthorization narrows, but does not close, the window between the check and the agent receiving the bytes. A lease lost during transmission is fenced by heartbeat and publication, not by this route.

Verified behavior

scripts/agent-gateway-e2e.sh creates an ephemeral CA, server certificate, known and unknown client certificates, and a clean PostgreSQL database. It verifies:

clients without a certificate fail the TLS handshake;
a CA-valid but unknown certificate receives 401;
only the targeted node can lease the job;
wrong/stale lease tokens receive 409;
heartbeats fence execution and completion is idempotent;
secret-shaped results are rejected;
the real agent leases and executes a typed database health task;
result rows are immutable;
database certificate revocation takes effect on the next request.

Enrollment (POST /v1/enroll on the separate server-auth listener), renewal (POST /v1/agent/certificate/renew) and rotation completion are described in fleet substrate. Every agent route on this listener still requires a client certificate. retired nodes are refused like revoked ones. Until placement assigns jobs.target_node_id, queued agent work is intentionally invisible to every node.