Environment mesh coordinators

pushleft-mesh runs only on trusted managed coordinator hosts in the mesh fleet pool. Do not colocate untrusted customer workloads: this process controls Docker, reads platform job state, and publishes scoped enrollment credentials. Fleet administration remains private; this is not a public fleet management API.

Each environment has its own digest-pinned Headscale container, private Docker bridge, Noise key, policy, and SQLite database. Runtime inspection refuses foreign ownership, changed specifications, extra ports/mounts/networks, and weakened container restrictions. The only published Headscale socket is a random loopback HTTP port. Caddy terminates public HTTPS for the exact environment hostname and denies administrative API, metrics, debug, and Swagger paths.

The coordinator uses a private Caddy admin Unix socket where available. Provision Caddy’s managed route bucket with ID pushleft-mesh-router, whose first handler is a subroute containing a routes array. Other routes in that array are preserved using ETag/If-Match compare-and-swap. Operators must provision valid TLS certificates and restrict the Caddy admin endpoint; the worker does not bootstrap certificates. Headscale DNS records are DNS-only A records; application ingress proxy settings are separate.

Durable reconciliation

Migration 0018 separates capacity reservation from workload admission. The scheduler can reserve a database node while its segment is still provisioning, creating a durable network_attachment_requests membership and an immutable globally unique /24 from 10.128.0.0/9. Both database capacity and gateway overhead (500m CPU/256 MiB RAM per selected host/environment) are reserved, not execution-ready; reserved + active count against capacity and share the unique live-resource index. Repeated polling reuses membership without consuming another subnet slot. Slots never cycle or expire; pool exhaustion fails closed rather than reusing a prefix that stale remote work might still reach.

The database job stays untargeted, and its instance has no active node, until the existing independent network admission predicate passes. Only then are its allocations activated and its typed execution job targeted. Existing reservations recheck node ownership, region, pool, schedulability, capabilities and heartbeat freshness. These requests are desired membership, not proof of applied topology. Capacity/CIDR reservations must not be released on a timeout alone.

Migration 0019 adds immutable initial topology sealing and prepared host tasks. With PUSHLEFT_TAILSCALE_IMAGE/--gateway-image configured, the mesh worker waits for all active initial database intents to have reservations, then seals at most 65 members with exact request/node/prefix bindings, configured image/state root and host interface. Membership insert/delete takes the segment lock and refuses changes after sealing. Additional host placement fails explicitly with a fenced-rollout requirement. This is fixed initial membership, not automatic live topology expansion.

The online Linux agent advertises its configured --public-interface/PUSHLEFT_PUBLIC_INTERFACE; the setting must name an existing non-loopback Linux interface. An absent interface prevents initial sealing. Managed VM bootstrap must configure it; application manifests cannot choose host interfaces. --gateway-state-root defaults to the agent’s /var/lib/pushleft/mesh root and must match the host agents.

Sealing creates blank attachment skeletons and existing receipt-bound enrollment jobs. No fake Vault reference or partial desired hash is persisted. After actual issuance, gateway/firewall/host-route/provider specifications and exact task payloads initialize in one transaction. The ordered host queue prepares an empty owned bridge plus nftables first, then gateway, then host routes. Firewall checks never create a bridge. Gateway lease requires current firewall evidence; prepared jobs must exactly match persisted payloads, and destroying/destroyed segments refuse new host work. Provider checks enqueue only after fresh host evidence. Database hosts publish no ports; ingress targets require a separate placement/mapping workflow.

Deduplication is per operation family, so a queued downstream gateway/route job cannot suppress the prerequisite refresh needed to make it leaseable. Saved topologies maintain eligible healthy peers even when another member is offline. Topology reconciliation runs in a separate bounded worker with a 150ms database lock timeout and 3s statement timeout; a busy segment cannot block global coordinator/provider health scheduling.

Admission remains closed for a newly bootstrapped segment. Aggregate activation and host-wide/provider firewall prerequisites are not implemented by this preparation slice. Migration 0020 adds receipt-bound credential revisions without relaxing runtime task/generation fences. A consumed key’s expiry must not invalidate an enrolled gateway’s identity. The agent now has independent serial network and workload execution lanes, verified with a database command blocked while a mesh health attempt reports its result. Fleet-scale health latency and saturation remain unverified. Release blockers still include fenced expansion and combined reservation-to-workload transport/fault tests.

Automatic deployment credential renewal is not production-ready yet. The private issuer implements its provider-side renewal barrier in enrollment.rs; database publication and host-job/result revision fencing are implemented, but automatic scheduling and agent-side credential restaging/remote-operation quiescence still require implementation and verification. An unused-key inventory observation alone is never used as a registration barrier.

The pinned upstream implementation separates key expiration and consumption CAS from registration persistence. Expiry readback alone is therefore not used as proof that earlier registration work has stopped.

The issuer persists enrollment-write.pending.json before renewal mutation, bound to the previous receipt and immutable coordinator ID/specification/verified loopback port. Its bounded expire → quiesce → issue → publish state machine retires an uncertain runtime before retrying expiry, verifies committed expiry, removes the immutable runtime again, then reads provider node/key provenance. Runtime recreation preserves SQLite/Noise data and the exact proxy port. Exact consumption preserves its matching receipt; only conclusively absent registration plus expired/absent old key permits a successor. A pre-create inventory distinguishes unpublished keys created by a lost response within the exclusively managed per-node provider user. Recovery quiesces before expiring those keys, then quiesces/readbacks again before retrying creation. Out-of-band mutation of these private provider users is unsupported; unknown consumed or unsafe keys fail closed.

Successor metadata is journaled before unique-path Vault CAS/readback. The active receipt changes only from the recorded predecessor or retained intermediate chain; historical receipts are immutable. Candidates remain in the chain through retirement, so registration during publication fallback is resolved against the correct receipt after quiescence. A pending journal blocks coordinator publication, routes, revocation and other enrollment intents until the exact intent recovers.

First-key issuance also uses this journal, before user creation or key creation. Initial journals bind initial_identity without inventing a predecessor receipt; existing renewal journals retain their previous-receipt format. A fresh request for a node with any issued-key or registered-node provenance is rejected before journaling: a new request is not an identity-rotation operation. The initial prepare_user stage retires an uncertain immutable runtime, rechecks empty provenance and reconciles the unique per-node user. Conflicting provenance discovered at this pre-key stage releases only that empty initial journal after quiescence; it does not adopt the other request or leave a global dead fence. The shared issue/publish stages expire newly created unpublished keys, quiesce/read back expiry, and journal candidate metadata before Vault CAS. A lost user-create response is recovered by exact user-name inventory after runtime retirement, not by blindly repeating creation. Missing initial active receipts are allowed; an existing unrelated receipt cannot be overwritten. A registered key without durable matching receipt provenance fails closed. This remains provider-side recovery, not proof of agent restaging, live topology expansion or aggregate admission.

Immutable metadata/history publication writes and fsyncs private staging bytes, then installs a hard link without replacement and fsyncs the containing directory. Retries acknowledge only identical regular files; foreign content and symlinks are refused. First identity-marker staging lives in the private state root, outside the not-yet-marked segment directory, so a crash cannot make its own temporary file look like foreign segment data. Unmarked segment data is still never adopted. Old partial final files produced by earlier implementations are not automatically overwritten; operators must investigate their provenance privately before attempting recovery.

Upgrade direction matters: the new worker can read prior renewal journals, but older binaries cannot recover the new initial stage. Complete pending journals with the compatible worker before a binary rollback; never delete a journal to make a downgrade proceed. The real provider fixture covers committed initial user/key creation with lost responses, unpublished-key expiry, exact-request recovery and conflicting-initial-request rejection/release. Seeded conflict boundaries are not proof of every concurrent transport interleaving.

The real fixture verifies committed expire/create lost-response recovery, orphan expiry, fixed-port/immutable-ID recreation, seeded crash-after-active-rename followed by successor expiry, and a retained consumed candidate during Vault outage. It also pauses the actual expiry command after an unused-key observation, registers a real Tailscale gateway with that key, then verifies recovery preserves its receipt/provider identity without extra keys/Vault writes and removes the paused immutable runtime. This proves preservation of a registration that commits before expiry is released, not an internally paused registration committing after expiry.

Database publication preserves enrollment identity, provider node ID and runtime generation/specification hashes while tracking a separate monotonic credential revision. Explicit renewal jobs bind credential_revision, expected_secret_ref and expected_provider_key_id to the old receipt. Preflight acknowledges exact committed history before contacting the provider. publish_mesh_credential checks authorization and the live lease again after resource-lock waits, compare-and-swaps the expected receipt, records append-only history, and atomically replaces only the auth references in all three prepared intents. Replacement clears all independent health evidence; readiness never transfers to a new credential. Same-receipt acknowledgment does not advance the revision. Superseded host/provider jobs cannot publish or invalidate newer evidence, and queued superseded work is terminalized without suppressing current repair. Cleanup never revokes a shared candidate while another eligible publisher could commit it; terminal revocation requires a committed eligibility fence.

publish_mesh_credential is invoker-security and has no PUBLIC execute permission. Provisioning must explicitly grant its exact function signature to the private mesh worker role, together with its required table permissions; never grant it to the browser role. Missing revision fields retain legacy revision 1, but explicit malformed/null revision values are rejected. Migrated-database tests cover publication/replay, strict parsing, prepared-task immutability, health reset, blocked-lock lease expiry, shared-candidate cleanup and stale gateway heartbeat/completion rejection. These database fences do not prove that a previously dispatched remote Docker operation has stopped.

Failed route checks convert to ensure only for the exact current task, revision and assigned coordinator. Queued superseded route jobs are retired with bounded job-only SKIP LOCKED selection; locked, leased, current and other-coordinator jobs are preserved. Completion holds the job lock before resource locks and performs its final live-lease CAS against clock_timestamp() after resource waits. Expiry rolls back status changes, retry conversion and health invalidation together. Real two-connection regressions prove publication waits for current retry conversion and an expired completion cannot erase health after waiting on an attachment lock.

Additional production proof is still required for registration paused inside provider validation across expiry, every journal crash/cancellation boundary, delayed Create surviving a Docker-client timeout, combined provider/database publication loss, stale host execution and fleet-wide recovery latency. Seeded crash-boundary tests are not a substitute for concurrent registration transport tests. None of this work opens aggregate admission.

Reservation publication holds a shared source-deployment lock and accepts only active lifecycle states; teardown/failed/unknown states stop only that deployment’s queued job without marking a shared database instance erroneous. Agent leasing independently repeats the lifecycle check under a shared deployment lock for deployment-owned database work. Null-deployment service/catalog operations remain independent. Work already running when teardown is requested still needs the explicit cancellation/fencing workflow; this change does not forcibly stop it or recycle its capacity.

Migration 0015 adds coordinator health and tenant-scoped enrollment intents; 0017 adds independently fenced provider-route evidence. The run command leases provision_tailnet, issue_mesh_enrollment, and reconcile_mesh_routes jobs, binds each segment to its coordinator node, and verifies actual HTTPS health before recording coordinator readiness. This readiness is not environment network readiness. Attachment, host routing, provider firewall, and periodic host enforcement gates must still pass before deployment admission opens.

Fleet authorization uses private, shared guard rows acquired in UUID order before segment/attachment locks. Status, pool, ownership changes and deletion take exclusive guards via database triggers; heartbeat/capability updates do not. Eligibility is read again after guard acquisition in a separate statement. This permits concurrent environments and heartbeats while serializing revocation with publication. Enrollment identity is immutable; rotation requires a new intent. Revocation clears affected coordinator/attachment evidence, and changing coordinator identity clears its health rather than transferring readiness. Provisioning also checks current managed/mesh/active/fresh authorization before publishing health.

Coordinator evidence expires no later than its observed fleet heartbeat plus 60 seconds; admission also requires its HTTPS check within 120 seconds. The tenant-invoker admission view reads only tenant-scoped segment/attachment evidence, not global fleet inventory. No readiness or provider proof is backfilled during migration.

The runner independently schedules due health every two seconds, even when jobs are queued. Up to eight workers drain available work without a fixed per-job sleep, with a 16-connection database pool. These bounds are not a demonstrated capacity/SLO: a realistic 100-attachment provider/HTTPS sweep remains a release test requirement.

Enrollment keys are non-reusable, non-ephemeral, and expire after ten minutes. Raw keys go only to Vault KV v2 with create-only CAS writes; job payloads, database rows, receipts, and command output contain references/metadata, never raw keys. Each provider key has a distinct Vault path. Durable receipts bind the request and secret fingerprint; retries check provider ownership, expiry, usage, and Vault integrity. An ambiguous write is read back before success is reported. Missing/expired keys are revoked/rotated instead of adopting unknown credentials. Final publication repeats authorization under row locks; revoked nodes, changed ownership, stale attachments, and destroying segments cannot be certified as issued.

Vault policies must deny workload identities access to other nodes’ enrollment paths. Do not expose the coordinator token or the full organization mount to agents. Receipt metadata and database backups require private access as operational security data, even though they do not contain raw credentials.

Subnet route approval

The private approve-routes operator command and CoordinatorManager::reconcile_routes implement explicit provider-side route approval. The durable worker now consumes exact persisted attachment intents, rechecks job lease, enrollment provenance, node ownership, coordinator identity and generation/hash before provider operations, and publishes approval evidence only after fresh independent host evidence and real HTTPS checks. A JSON intent contains node_id, enrollment_id, a typed gateway_spec and enabled. The command takes the same organization/environment/segment identity arguments as ensure, plus --task-file; --check verifies without changing route approvals and cannot certify an unapplied generation. The complete controller that prepares every attachment/host intent from a deployment remains unfinished; this wiring does not establish zero-touch deployment readiness.

Recurring route checks are scheduled for persisted intents after 30 seconds. Failed checks invalidate current approval evidence and retry as repair/ensure operations. Exhausted expired leases are terminalized so they cannot permanently suppress health scheduling. Withdrawal can run for revoked/offline workload nodes and destroying segments, but still requires current enrollment provenance, exact desired intent and an authorized live coordinator. Gateway, firewall, host-route and provider-route evidence must independently match the same desired generation and be fresh before admission opens.

Approval requires the canonical environment coordinator URL, an organization/environment-scoped secret reference matching the persisted enrollment receipt, and exactly one provider node for the issued fleet-node user. The node must match that receipt’s pre-authentication key, advertised hostname and auth-key registration method. Hostname/provider ID alone are not ownership proof. Enabled routes require a live, unexpired node advertising exactly its assigned local /24; extra prefixes and exit/default routes are refused. An overlapping approved route on another node also blocks approval. The command replaces approvals with the exact assigned prefix and verifies fresh provider readback; it never appends whatever a node advertises.

A private per-node provider-ID/generation/hash fence is persisted before mutation under the environment coordinator lock. It prevents rollback, same-generation policy changes and silent node replacement, including after route withdrawal. Check-only verification requires exactly the already persisted generation/hash; a higher generation is not proof of application. A changed provider node requires an explicit rotation workflow; do not remove fence files to adopt it. enabled: false with a newer generation withdraws all of that node’s approvals even when it is offline, then verifies the empty set. Retain enrollment provenance and fences for withdrawal/recovery. Issuer retries never replace a consumed intent’s receipt: they acknowledge the identical receipt only when exactly one provider node and one consumed, single-use, non-ephemeral key match its node/user/key provenance and auth-key registration method. A consumed bootstrap key’s expiry is not node expiry, so this acknowledgment neither reads Vault nor issues a new key. Missing, ambiguous, unsafe, or mismatched provenance fails closed. Explicit re-enrollment still requires a new intent and rotation workflow. Check-only failure does not repair drift; an ensure operation is required.

Provider mutations use the verified immutable coordinator container ID. A fsynced environment-wide pending-write marker is created before each mutation and removed only after exact provider readback. Errors, cancellation and lost responses leave it in place; coordinator readiness, key issuance/revocation and route checks fail closed while it exists. An explicit route ensure recovers by verifying ownership and removing the old immutable coordinator runtime, confirming its absence, then recreating it from preserved SQLite/Noise/config state. This quiesces even delayed Docker exec requests; killing the Docker client or merely accepting a newer generation is insufficient. Never manually delete the pending marker. Recovery does not remove workload containers or coordinator data.

The pending marker preserves the verified loopback proxy port, and recovery binds exactly that port so existing HTTPS handlers can reconnect without a blocked publication operation. Port conflicts fail closed instead of silently allocating a different upstream. An enabled retry may need to wait/retry while the gateway reconnects; the marker remains until its exact route policy converges. The route operation alone does not attest public DNS/ACME or every proxy path. Offline withdrawal remains possible without a gateway connection. A crash after runtime removal is recovered from the same durable marker on retry.

This is provider-approval evidence only. The actual provider fixture and database authorization/concurrency regressions are separate tests, not proof of a complete distributed controller deployment. Combined fault-injected orchestration and actual multi-host transport still need end-to-end proof. An ambiguous provider write must not be treated as successful merely because a local fence exists.

Configuration and verification

Use pushleft-mesh --help and pushleft-mesh run --help for explicit runtime arguments. Required settings include the pinned PUSHLEFT_HEADSCALE_IMAGE from deploy/images.lock.env, PUSHLEFT_MESH_DOMAIN, PUSHLEFT_DERP_MAP_URL, DATABASE_URL, managed coordinator PUSHLEFT_NODE_ID, scoped Cloudflare token file/zone/public IPv4, and Vault address/private token file. Secret token files must not be group/world accessible. Keep state under a dedicated root-owned directory, backed up consistently with the coordinator SQLite data and Noise keys. Database URL credentials must be scoped to this trusted worker’s requirements.

npm run test:mesh requires local Docker, OpenSSL, and a local Caddy binary (PUSHLEFT_CADDY_BINARY can select it). It starts real pinned Headscale instances and a real HTTPS proxy with temporary certificates, plus an emulated Vault KV v2 endpoint. It verifies instance/network separation, replay, HTTPS route readback/admin denial, short-lived issuance, failures before/after Vault commit, provider revocation recovery, secret substitution refusal, and tenant ownership rejection. The real agent then tests untrusted-CA rejection, expired-first-key failure, same-container enrollment with a replacement key, secret cleanup, and enrolled restart while Vault GETs return 503. Enrollment uses the production-generated route handlers with a fixture-only reachable hostname/IP alias; no host DNS/daemon configuration is changed. Linux binds the existing Docker host-bridge IP; Docker Desktop uses its host alias. This does not prove a live Cloudflare account, public DNS/ACME, multi-host workload packet routing, hostname-mismatch rejection, or complete fleet rollout. All fixture resources are explicitly named and cleaned up. This suite is included in test:all; CI installs checksum-pinned Caddy and runs it on native Linux when the workflow is executed.

npm run test:mesh-controller is included in test:all. It creates a uniquely named disposable PostgreSQL database and applies all migrations. Lease-reaper and enrollment regressions use transaction-local fixture tables; route/admission, topology and two-connection guard regressions use the actual migrated tables/triggers. Tests cover changed authorization sources, expired jobs, invalid evidence, offline withdrawal, immutable identities, concurrent heartbeats/shared readers, a revoker already holding the fleet tuple, job/segment lock ordering, blocking until provider publication releases its guard, and immediate readiness invalidation after revocation. Bootstrap tests cover blank skeletons, receipt-bound atomic preparation, ordered host jobs, all-initial-reservations waiting, frozen multi-node peers, offline-peer maintenance, bounded contention and prerequisite-repair starvation. The same command runs agent-gateway authorization races against real PostgreSQL. API E2E separately verifies tenant positive/negative invoker-view access without SELECT on fleet inventory or guards. These regressions do not simulate every distributed crash/interleaving across Docker, Vault, and PostgreSQL.

On native Linux, the real mesh runtime fixture must own its private bind state as root, matching the production coordinator contract. A UID-0 container with all capabilities dropped cannot traverse another user’s mode-0700 directories. The test refuses this mismatch rather than weakening directory permissions or adding DAC capabilities. CI explicitly opts into PUSHLEFT_MESH_E2E_SUDO=1, which runs only the disposable Python runtime fixture with noninteractive sudo; Cargo compilation stays unprivileged. Local Linux operators can instead invoke the fixture as root. Docker Desktop’s file-sharing permission behavior does not prove this native-Linux ownership property; remote CI execution remains a separate verification requirement.

The mesh fixture also exercises explicit receipt-bound subnet approval, exact readback/replay, refusal of future-generation checks and broader advertisements, approval-drift repair, consumed-receipt preservation and offline withdrawal. A temporary standard-library Go helper built with the pinned toolchain runs inside the disposable coordinator’s executable data bind: it remains alive past a real Docker-client timeout and attempts a delayed write. The test requires immutable-ID quiescence, same-enabled-intent recovery through the preserved proxy port, no stale re-enable after withdrawal, committed-write/lost-response recovery, crash-after-marker recovery, SQLite node-identity retention and real HTTPS verification. The production noexec tmpfs restrictions are not weakened. Occupied-port/foreign-runtime and concurrent recovery/cancellation regressions remain additional release proof gaps.

Production release remains gated on complete attachment orchestration, firewall enforcement/reverification, managed service operations, restore/HA exercises, and passing runtime image vulnerability scans. This slice does not make the overall platform production ready.