Pushleft architecture

Product boundary

Pushleft converts a local checkout or Git revision into an isolated preview or a production release. The managed service runs first on Hetzner in fsn1 and hel1; customer-owned VMs follow. Kubernetes is an adapter later, not the first runtime abstraction.

pl / Git hooks / web
          │
          ▼
  Rust control plane ─── PostgreSQL (desired state, events, leases, usage)
       │      │
       │      ├──────── Vault/OpenBao (secret values; control plane stores refs)
       │      ├──────── OCI registry + object storage
       │      ├──────── Cloudflare (authoritative DNS, proxy/WAF, certificates)
       │      ├──────── Headscale (management plane + isolated environment control planes)
       │      └──────── ClickHouse (90-day telemetry and web analytics)
       │
       ▼ outbound mTLS pull
  VM agents ── management tailnet
       │
       └──── environment bridge + namespaced mesh gateway ── web/worker/cron/data

Core invariants

A deployment is desired state plus an immutable release. Traffic assignment is mutable and auditable.
Every mutating API call and agent job carries an idempotency key.
Every row, query, event, and usage record is organization-scoped. Production database roles use RLS; application predicates remain mandatory defense in depth.
Every public API route resolves exactly one organization from the credential (AuthContext), checks a minimum role, and sets app.org_id from that context. Production credentials are looked up by hash on every request; nothing about the tenant is taken from the request body or path. Development auth is loopback-only. See authentication.
Secret values live in Vault/OpenBao. Manifests and PostgreSQL contain references only.
Agents initiate outbound mTLS connections. The gateway verifies the client CA and an active certificate fingerprint, then leases only work targeted to that node. Agents execute a versioned allowlist of typed jobs, never arbitrary control-plane shell strings.
Deployment events are append-only. Log and event streams resume from a cursor.
Resource limits, budgets, expiry, placement, and pricing checks happen before work is leased.
Teardown is a reconciled state transition: traffic is removed, processes stop, ephemeral resources are deleted, persistent resources follow retention policy, and the audit trail remains.
Preview hosts are flattened as <branch>--<project>.preview.pushleft.dev, so one *.preview.pushleft.dev certificate covers them.
Domain purchases use Openprovider from the backend and authoritative DNS uses zones in one configured Cloudflare account. Customer ownership, registrant consent, accepted prices, payment state, delegation, and retries remain durable Pushleft state; provider credentials never reach the browser.
Mail, DKIM, SPF, MX, and provider-verification records are always DNS-only. Only eligible A, AAAA, and CNAME web records may enable the Cloudflare proxy.
Every deployment environment has a dedicated Headscale instance and database. A shared Headscale instance plus tags is not considered a tenant boundary. The host agent never joins these workload tailnets.
Public-port declarations name an exact workload role and destination port; only 80/443 may implicitly target the managed ingress router. Deployment admission stays closed until provider and host firewall generations plus the selected node’s environment attachment are ready.
The node agent and local OpenTelemetry collector never receive arbitrary shell or SQL from the control plane. Typed operations, digest-pinned images, Vault references, and fencing tokens are mandatory.

Fleet nodes move through provisioning → active ⇄ unhealthy → draining → retired (plus the older offline/revoked). Only active nodes can lease. unhealthy (missed heartbeats) keeps allocations, and drain is the only path that moves work. Retirement revokes every certificate and keeps all history. Enrollment uses a one-time, 15-minute secret on a separate server-auth-only listener. Client certificates are short-lived (24 h by default) and rotate through the mTLS gateway. The old certificate is revoked on the first lease with the new one. See fleet substrate.

The placement scheduler reserves CPU, memory, and persistent disk before targeting a job. A node is eligible only while active, explicitly schedulable, seen within 60 seconds, in the requested region and pool, advertising the required capability, and either managed by Pushleft or owned by the requesting organization. Capacity exhaustion leaves work queued for later reconciliation; malformed or currently unsafe requests fail closed. Allocations remain durable until an explicit teardown or move releases them.

DNS and domain ownership

Openprovider supplies availability, authoritative prices, registrant handles, registration, renewal data, and nameserver delegation. The controller creates a new full Cloudflare zone only after registration is confirmed, records its exact account, zone ID, and assigned nameservers, and waits for delegation before making the domain active. It never adopts a pre-existing zone based only on a matching name. Cloudflare tokens are restricted to the configured account with Zone and DNS edit permissions.

Record changes are reconciled from desired state. Cloudflare batch operations reduce partial updates but propagation across its distributed store is not atomic, so migrations must tolerate old and new answers during the convergence window. Pushleft tags owned records and never deletes unowned records.

Provider presets generate reviewable record sets for Microsoft 365, Google Workspace, Amazon SES, and inbox2. Tenant-specific verification, MX, and DKIM values remain required inputs. SPF is merged into the existing single SPF record rather than publishing multiple SPF TXT records.

Private deployment networks

Host precondition for every application node: br_netfilter loaded and net.bridge.bridge-nf-call-iptables=1, otherwise the environment firewall below is bypassed for frames the bridge switches between the mesh gateway port and workload containers. The agent fails closed without it (docs/fleet.md, “Firewall”).

The host agent joins exactly one private, self-hosted management tailnet. That identity is used only for fleet administration and is never attached to a workload bridge. Running global tailscale up for a deployment is forbidden because one ordinary host daemon cannot be active in several isolated tailnets at once.

Each environment receives a dedicated Headscale control server/data store and therefore a distinct workload tailnet. On every selected host, the agent will reconcile one digest-pinned Tailscale subnet-router container in its own namespace. That gateway attaches only to the environment-specific Docker bridge and advertises only that host’s globally unique private /24. Releases in the same environment share the bridge and stable service discovery; cross-host traffic uses the environment mesh. Peer routes and return paths are reconciled from durable node attachments. Different environments have separate Headscale state, gateway namespaces, bridges, routes, and firewall identities.

Enrollment uses a one-use, short-lived pre-authentication key read from a tenant- and environment-scoped Vault reference. The key is delivered as a 0600 file and passed to tailscale up through file: so it is not exposed in process arguments or Docker environment metadata. Tailscale SSH, exit-node advertisement, management-tailnet routes, and unrelated route acceptance are disabled. Kernel mode is required for subnet routing; the gateway receives only NET_ADMIN, NET_RAW, and /dev/net/tun, with no Docker socket, host network, or agent credentials.

Gateway admission verifies effective runtime state, not just desired labels. The agent inspects the pinned image, entrypoint/command/environment, sole static bridge attachment, private namespaces, exact capabilities/TUN device, read-only filesystem, exact private mounts/tmpfs, unpublished ports, bounded CPU/RAM/swap/PIDs, and rotating local logs. After Tailscale reports Running, it independently reads effective preferences and rejects foreign control URLs, extra advertised routes, SSH/web/DNS enablement, exit-node use, and weakened forwarding filters. Health also requires the exact local state ownership marker. Reconciliation is serialized per environment; stopped containers retain their configured static IP for pre-start validation, and subsequent live health requires the actual IP. Starts/restarts use inspected container IDs. Ambiguous create failures never remove a container by name.

The gateway runs with TS_DEBUG_FIREWALL_MODE=nftables: its root filesystem is read-only without a writable /run, and in iptables mode tailscaled cannot take /run/xtables.lock and never opens its control socket (seen on Ubuntu 24.04 with ip_tables loaded; Docker Desktop picked nftables by itself). Gateways created before this variable existed still verify; ensure_gateway recreates them on their own state directory, so enrollment is kept and TS_AUTH_ONCE skips a new login.

First-enrollment provenance is durably recorded before key retrieval/start and cleared only after verified health and key removal. Failed first login retains the short-lived key; a stopped pending runtime refreshes it from the authorized Vault reference before start. A running NeedsLogin runtime can recover only with the expected coordinator, or an uninitialized default profile backed by matching pending provenance. Already connected profiles are never reauthenticated as a recovery shortcut. Normal enrolled restarts do not require bootstrap Vault reads. test:network-agent exercises real Docker runtime/preference readbacks, stopped static-IP validation, live preference drift, and stopped pending-key refresh. test:mesh, now included in test:all, also proves real TLS enrollment through production-generated proxy handlers, untrusted-CA rejection, expired-first-key recovery of the same stopped container, consumed-key cleanup, and enrolled restart during a Vault read outage. This is not yet proof of cross-host workload routing, public DNS/ACME, or production fleet rollout.

Publicly trusted coordinator certificates need no extra agent setting. Self-hosted private authorities use operator-only PUSHLEFT_MESH_CA_FILE (or --mesh-ca-file); manifests cannot select trust roots or disable TLS checks. The agent validates PEM and X.509 data, rejects private keys/mixed blocks and non-regular, relative, or group/world-writable source files, and creates an immutable content-addressed trust snapshot. Only that exact snapshot may be mounted read-only with the expected TLS trust setting. Changing an operator trust bundle changes its snapshot path and fails existing runtime inspection until an explicit fenced replacement is performed; CA rotation is not silently applied to a running gateway.

Firewall identity binds the gateway’s runtime specification hash and resolved ingress policy, not its bootstrap credential reference. Reissuing a one-use key therefore cannot invalidate otherwise unchanged firewall enforcement. The environment_firewall_v2 renderer admits declared peer replies only when IPv4, both interfaces equal the environment bridge, destination is local, and conntrack is reply-direction established/related. Unsolicited peer traffic remains denied. Existing persisted firewall fences/desired records require an explicit newer generation when upgrading; do not rewrite or bypass an old fence at the same generation.

The typed network task validates pinned images, immutable generations, canonical private subnets, non-overlapping peers, credential-free HTTPS Headscale origins, state paths, and Vault tenant scope. The agent can reconcile an owned Docker bridge with a deterministic host interface and a constrained kernel-mode Tailscale gateway using a file-backed one-use key, persistent state, exact Docker labels, and health checks. It can apply and independently verify scoped environment nftables policies and host peer routes. The isolated Headscale runtime, HTTPS publication, scoped Vault enrollment recovery, and durable coordinator worker are implemented; see coordinator operations. Per-node attachment publication/orchestration, complete partial-agent-enrollment recovery, host-wide input policy and provider firewall enforcement remain unfinished, so network_segments.status must not become ready in production.

Database placement, reconciliation of existing allocations, logical-database/user operations, and agent leasing require a host-specific network_attachments record. Gateway, firewall, host routes, and coordinator-approved provider routes must each match the desired generation and separately hashed policy, with verification timestamps within the last 120 seconds; future timestamps are rejected. The shared ready_network_attachments view uses invoker security so tenant RLS remains effective without exposing global fleet inventory. Coordinator health is separately projected with an expiry no later than its observed fleet heartbeat plus 60 seconds; revocation and coordinator replacement invalidate cached health. There is no readiness backfill. Authenticated network-job results update only the matching host/environment/gateway/generation, durable desired specification hash, and the exact pl_<segment UUID> bridge used by database placement. Database triggers prevent attachment identity changes, generation rollback, and specification changes without a generation increase. Health tasks carry the complete expected gateway specification; the agent checks actual environment/specification labels (including generation), sole bridge attachment and IP, bridge ownership/IPAM, and Tailscale state. Gateway health cannot certify an unapplied generation or mark firewall/routes ready. Route leasing and successful route results additionally require fresh matching gateway and firewall evidence; check-only jobs cannot certify unapplied routes. Retry failures invalidate their respective evidence. The durable mesh worker now periodically verifies exact persisted provider-route intents and repairs failed checks; a future attachment controller must prepare all host/provider desired state before these gates can open in production. Admission prevents new work; stopping existing traffic after network drift still requires host enforcement.

Host routing uses an exclusively owned numeric table derived from the environment UUID, protocol 196, and priority 1000 rules restricted to the local source subnet and incoming environment bridge. It never modifies main/local/default or management-tailnet tables. Numeric JSON reads prevent operator table/protocol aliases from concealing foreign state. Root-owned locks, table ownership records and durable generation/hash fences prevent adoption, collisions and rollback, including after kernel rule loss. Validation rejects overlapping higher/equal-priority policies and local-table peer collisions, then verifies actual kernel lookups. Route hash environment_routes_v2 binds the priority/protocol; upgrades require a newer generation. Online route tasks attest gateway health first; the offline manage-routes diagnostic command only verifies host routing and does not attest mesh enrollment.

npm run test:linux-routes proves real TCP/UDP peer replies, unsolicited-peer denial, environment isolation, numeric aliases, narrow source/destination shadow detection, local-table collision refusal, unchanged main/unrelated tables, drift repair, peer removal and durable fencing. Its decrypted remote link is simulated: actual multi-host Tailscale transport, route approval and SNAT remain separate release proofs. Linux CI schedules this gate alongside the firewall and quota gates; remote CI execution has not been verified locally.

Upgrade caveat: a previously installed v1 priority-10000 route rule is deliberately rejected as foreign by v2, even with a newer desired generation. There is no automatic in-place v1 rule migration. Keep the attachment closed and quiesce its workloads, verify the persisted environment/table ownership and exact old rule, remove only that old rule through operator maintenance, and apply the newer v2 generation before restoring admission. Never clear ownership/fence files, flush shared tables, or bypass readiness to perform an upgrade.

Public traffic is intended to reach only the Cloudflare proxy/router path or explicitly selected workload ports. NetworkPolicy rejects custom ports without a destination role/port and rejects source CIDRs overlapping private, shared-overlay, documentation, benchmark, multicast, or reserved space. The agent owns one nftables table per environment, leaves Docker/other owners’ tables alone, checks ownership, validates a complete transaction, atomically replaces its table, and compares normalized kernel readback with desired rules. A root-owned persistent generation/hash fence and an exclusive per-environment file lock prevent stale retries from restoring older rules, including after table loss. Workload-to-host access, private/metadata egress, source spoofing and IPv6 are denied. Same-environment traffic and public IPv4 egress are allowed. Public ingress requires an exact interface, protocol, source range, translated destination IP/port, DNAT status, and original published port; direct routing and undeclared ports are denied. The firewall specification has its own durable hash and cannot refresh gateway readiness. Stale network jobs cannot acquire an agent lease.

npm run test:linux-firewall builds a Linux agent and runs real namespace packet tests in a disposable Docker worker without host networking, host devices or writable host mounts. It covers TCP/DNAT allow and deny behavior, cross-environment and host/metadata isolation, same-environment traffic, internet egress, source ranges, replay/drift repair, generation rollback, fence persistence after table loss, and ownership refusal. Broader UDP/IPv6/spoofing and native Docker-rule coexistence validation remain release work. Host-wide public-input policy, provider firewall reconciliation, boot-time ordering and periodic re-verification are separate requirements; this scoped table does not certify them. The Docker Desktop kernel used locally has XFS but lacks XFS quota support, so the real XFS quota release gate still needs a quota-enabled Linux worker.

Observability and analytics

Every node runs pushleft-agent and one OpenTelemetry Collector Contrib system service. The agent renders and validates the node-scoped collector configuration from its authenticated mTLS identity, then reconciles the systemd unit with rollback on failed activation. Host metrics and application OTLP signals enter over loopback. The collector runs under a dedicated UID, uses a telemetry-only client certificate, and is denied access to the Docker socket and the agent/Vault credential directory. Container cgroup attribution and cost-allocation metrics are still pending runtime integration.

The Rust mTLS OTLP gateway accepts protobuf over HTTP from telemetry-purpose certificates only, resolves revocation against the control-plane database, overwrites node and region identity, rejects tenant/deployment claims, applies per-node request and byte limits, and forwards to a loopback-only collector. The collector has a persistent retry queue and batches logs, metrics, and traces into ClickHouse with a 90-day TTL. Workload-scoped OTLP ingestion remains closed until deployment identity can be validated against scheduler placement. Attribute/body redaction policy is still pending, so applications must not emit secrets in telemetry. Control-plane audit retention is separate and longer-lived.

The built-in web analytics ingress accepts events only when the telemetry certificate’s node, deployment, and hostname match an active ingress_analytics_routes assignment. Organization, environment, service, and region are derived from trusted state. Before insertion into the 90-day ClickHouse table, the gateway drops query strings and fragments, reduces referrers to hostnames and user agents to coarse families, and replaces the source address with a daily HMAC. Raw IP addresses and full referrers are never written. Duplicate event_id values converge through ReplacingMergeTree. Runtime route publication, a durable edge-side retry queue, and the organization-scoped analytics query API remain to be implemented before the feature is end-to-end.

Managed databases

The managed host image, rather than an inbound database job, installs Docker Engine, XFS tooling, and the agent. The agent then manages PostgreSQL and ClickHouse as digest-pinned containers with explicit CPU, memory, XFS project-quota disk limits, data-path, and private-network policy. Database container roots are read-only; ClickHouse configuration/logs live inside its quota tree, temporary files are bounded tmpfs mounts, and Docker stdout/stderr logs rotate. Quota project IDs are durable and globally unique; startup fails closed unless accounting and enforcement are active, and the scheduler requires and revalidates the matching node capability. The online executor supports idempotent ensure/install, restart, health checks, database/user creation, and least-privilege grant reconciliation. Credential rotation and removal exist as typed/local maintenance operations but remain rejected by the online lease loop until database-level fencing is implemented. It fetches secret values just-in-time from Vault/OpenBao KV v2 and supplies bootstrap credentials through root-only bind-mounted files; user credentials travel only over child-process stdin. Values never appear in command arguments, Docker labels, jobs, events, or control-plane rows.

The control plane keeps one durable database instance per environment and engine, separate from releases. Logical databases and principals are idempotent desired-state resources. PostgreSQL owners are explicit; ClickHouse users receive a normalized allowlist of grants. The agent refuses floating images, unowned containers, paths outside its configured roots, implicit specification upgrades, and data deletion without a path-bound removal task. Database containers publish no host ports and attach only to the environment network.

PostgreSQL WAL archiving, verified base backups and point-in-time restore are executed by the agent. Restore always creates a new generation (<instance>_g<N>); it is never done in place. ClickHouse native backups are executed too. Daily restore drills verify both engines and record their results as fenced instance evidence (migration 0030). Backups use a private local BackupStore; an S3/R2 adapter and encryption are not built yet. Replica bootstrap, promotion, major-version upgrade and relocation cutover remain rejected: a restored generation runs beside its source, and writers are not moved. Application roles can declare host-local persistent volumes with one writer and same-environment read-only readers. Volumes are quota-bound, retained on teardown, backed up nightly and released only explicitly or after a retention window. See managed-databases.md.

PostgreSQL relocation

Pushleft does not promise multi-writer PostgreSQL. A move between Pushleft and Shiftleft uses a controlled single-writer handoff:

Provision and restore the target from a base backup. The agent’s restore task does this today, into a new generation with PITR, but only on the same host’s local backup store.
Start one-way logical or physical replication from the authoritative source.
Bring up the target application against a read-only or shadow target and pass health checks.
Apply a short write fence and wait for replication lag to reach zero.
Promote the target, switch edge traffic/DNS, and release the write fence.
Drain the old application while it rejects writes.
Keep reverse replication or a rollback window until the move is accepted, then retire the source.

The application should use a stable database endpoint so most moves require no connection-string change. DNS TTL reduction is preparation, not the consistency mechanism.

Planned relocation targets zero acknowledged-write loss through the write fence. An unplanned regional failure uses asynchronous replication and therefore has a non-zero RPO target. A third failure-domain witness owns quorum; the database controller owns fencing and must revoke the old writer before promotion.

Delivery slices

Slice 1 — control-plane contract (implemented)

.pl.yaml parser, validation, inference, and deterministic planner.
Rust API with idempotent deployment creation, immutable events, job enqueueing, teardown, and SSE cursors.
pl up, pl list, pl down, pl logs, and pl plan.
Production authentication: hashed API tokens, console sessions with a CSRF header, owner/admin/member/viewer roles on every route, bootstrap onboarding, immediate revocation, an append-only auth audit and pl login/whoami/logout. It requires a trusted TLS proxy. SSO, 2FA and password reset are not implemented (authentication).
Agent host safety checks, secure management-Headscale join, node-scoped telemetry reconciliation, and mTLS typed-job leasing.
PostgreSQL migration and local developer stack.
Dedicated network-segment, DNS-record-set, and telemetry-tenant desired-state tables with RLS.
Cloudflare zone/owned-record execution, DNS provider preset validation, and 90-day ClickHouse schemas.
Managed PostgreSQL/ClickHouse instance, catalog, and principal state with RLS; online typed agent execution for install, health, users, and grants.
Mandatory-mTLS agent gateway with certificate revocation, node-targeted leases, heartbeat fencing, bounded results, retry backoff, and append-only execution records.
Immutable source-snapshot/release-artifact persistence, private lease-bound build publication, and per-role/replica application intents. This is a contract foundation; source ingestion, isolated builds and application execution are not wired yet.

Slice 2 — single-node runtime

Automated agent enrollment, short-lived certificate issuance/rotation, and drain workflows. Implemented as private operator tooling: see fleet substrate. Hetzner provisioning is tested only against a local API stub; no live Hetzner run has been made. Capacity-bounded database placement is implemented; application-role placement is not.
OCI build, registry push, typed Docker lifecycle jobs, resource limits, health checks, and local volumes.
Wildcard preview DNS/TLS and rollback-safe router configuration.
Environment mesh gateway, selected-node attachment readiness, host route reconciliation, and atomic provider/nftables firewall enforcement.
Structured stdout/stderr and OTLP ingestion with bounded disk queues and cursor persistence.

The next production-facing increment is an authenticated, single-node stateless preview from actual pl up to a verified HTTPS URL. The CLI now uploads a verified immutable source bundle when ingestion is configured, but application build/runtime jobs still have no executable worker path. Generic application-role reservations prepare their own environment membership; a web-only application does not need a dummy database. A private pinned-image runtime test is an implementation checkpoint, not completion of this user flow.

Migration 0021 supplies the first checkpoint. source_snapshots binds one immutable archive digest/object key, byte/file bounds, project, plan hash and scoped image repository to an active deployment. Migration 0023 and the configured API ingester now verify bounded archive bytes/manifest before reserving one exact digest and stable artifact identity under organization quotas, privately storing it without overwrite, and atomically attaching verified provenance to the queued build. SQL metadata alone still cannot prove object contents, and the eventual isolated builder must checksum-verify downloaded objects. See source ingestion for limits, cancellation/replay behavior, local-storage assumptions and missing retention/GC. Deployment manifest/plan/source identity cannot be edited in place; a changed checkout requires a new deployment.

publish_release_artifact is a private invoker-security function with PUBLIC execution revoked. The future agent gateway/controller, not an untrusted build container or browser, must receive the necessary database privileges. It accepts only an exact digest under the snapshot’s organization/project-scoped repository, a matching targeted build job/source snapshot/plan and preassigned artifact_id, and an active, fresh authorized build-pool node with isolated-build capability. Ingestion persists the output artifact identity before dispatch so retries never invent another identity after a lost response. Authorization and the wall-clock lease are rechecked after waits. Exact replay is allowed only under current authorization; substitutions are rejected. Artifact, immutable role/replica intents and untargeted ensure_application jobs commit together. They do not allocate capacity, start containers, publish routes or signal ready. There is no public adopt-an-image endpoint. The configured ingester selects the registry origin; a scoped repository suffix alone is not registry trust, and actual build provenance still requires the isolated builder. The scheduler now targets verified builds to isolated build-pool nodes, the agent builds them with rootless BuildKit and pushes the exact digest, and the gateway calls this function inside the completing transaction; see build executor for the isolation decision, the unscoped registry credential boundary and the missing native Linux proof.

Tenant policies permit reading these records but not asserting verification, inserting artifacts or creating runtime intents. npm run test:release-artifact, included in test:all, uses a disposable migrated database to test atomic publication/replay, invalid build/source/image/tenant provenance, immutable input/history, tenant read/write isolation, and real resource-lock waits crossing lease/node-freshness boundaries. Typed ApplicationTask contracts reject unknown nested role fields, tags/unscoped images, unsafe resource limits/health paths and mismatched execution identities. Their scope must come from authenticated durable placement state; intrinsic shape validation is not authorization. The long-lived runtime contract covers web/private/worker roles; cron/job/release require their own execution lifecycle before they can run. Runtime implementation remains absent, so no application workload is advertised as deployed by this checkpoint.

Migration 0022 and the scheduler reserve application-pool capacity plus one gateway allocation per host/environment and prepare immutable membership without executing workloads. The initial stateless envelope reserves 500 millicores/512 MiB per role replica, plus 500 millicores/256 MiB gateway overhead; configurable resource plans remain to be connected. Existing reservations do not silently move when a host becomes unavailable. Initial topology eligibility covers application and database pools, and sealing waits for active deployment artifacts and all application-role reservations as well as database reservations. This prevents a fast database reservation from sealing before a slower build produces application intents. Migrated PostgreSQL tests cover application-only bootstrap, mixed build/placement barriers, capacity accounting, foreign-owned host exclusion, and reservation replay without granting execution. Targeting runtime jobs must still wait for independent admission and agent-side support; do not make currently unhandled ensure_application jobs executable by setting their target manually. Cron/job/release execution lifecycles, resizing, and live topology expansion remain incomplete.

Application runtime admission (critical-path step 3). The scheduler targets an ensure_application job to the host holding its deployment_role reservation only when ready_network_attachments holds fresh independent evidence for that host/environment, the same predicate database work uses. Targeting activates the role and gateway allocations, marks the instance deploying with that active_node_id, and publishes a typed EnsureInstance task. The agent gateway (crates/pushleft-agent-gateway/src/applications.rs) leases, renews and completes the job only for the node owning the active allocation of a live deployment, rebuilding the task from durable instance/artifact/snapshot/segment/allocation state and requiring exact payload equality; leasing additionally requires fresh admission. Results update the instance only for the matching generation and owning node (healthy success is ready, everything else error). The agent (crates/pushleft-agent/src/application.rs) validates the task and its leased tenant envelope, pulls only repo@sha256: images, runs one container per generation (pl-app-<instance>-g<generation>, labelled with organization/environment/deployment/instance/generation/spec hash) attached only to pl_<segment>, with --cpus/--memory/--memory-swap/--pids-limit, --read-only plus a small tmpfs, --cap-drop ALL, no-new-privileges, no published ports and no mounts (the task has no secrets/volume policy yet, so none are ever added). Web roles are healthy after an in-network HTTP probe (busybox wget) from the digest-pinned PUSHLEFT_APPLICATION_PROBE_IMAGE in deploy/images.lock.env (Alpine 3.24.2, scanned by scripts/runtime-image-scan.sh; --application-probe-image overrides it); workers and private roles without a health path after process liveness, both within 60 seconds. Missing that deadline is a permanent unhealthy failure, because the same release would fail the same way; Docker and pull errors stay retryable. Replay adopts an exact matching container, replaces a drifted one of the same generation, refuses a stale generation, and removes older generations only after the new one is healthy. Lease loss, heartbeat conflict or completion conflict stops the container only if this attempt created or started it and it was not yet healthy (an unfinished rollout); an adopted running or already-healthy generation keeps serving and the loss is only logged (application::stop_on_lease_loss). The gateway alone decides instance status. npm run test:application-runtime exercises this on real Docker with a local registry and a per-run private network; its control-plane admission is out of scope. The migrated tests in npm run test:application-placement stub admission evidence in SQL exactly as scripts/scheduler-e2e.sh does. Still missing: secrets/volume policy, configurable resource plans (the 500m/512 MiB/128 PID envelope is fixed), ingress/DNS/TLS publication, and native multi-host proof.

One-shot roles and teardown (step 4b). Cron/job/release roles reserve the same 500m/512 MiB envelope and wait for the same admission; on admission the scheduler marks the instance ready on its host and keeps the durable container spec on the completed placement job instead of publishing a service task. crates/pushleft-scheduler/src/runs.rs then enqueues run_application jobs (typed RunApplicationTask, run_id = job id) targeted to that host: release and job run once per deployment; cron runs on its five-field UTC schedule (CronSchedule, Vixie day-of-month/day-of-week OR), never with a second run of the role queued or leased (overlap prevention under the instance row lock), and after downtime or overlap only the most recent missed tick runs. Long-lived roles are not targeted until the deployment’s release run has completed successfully; a terminal release failure fails the deployment. The gateway (src/runs.rs) leases runs only to the owning node with fresh admission and a spec equal to durable placement. The agent (src/runs.rs) runs the same hardened container (--restart no, read-only, cap-drop ALL, limits, environment network only, no mounts/ports) for at most MAX_RUN_SECONDS (3600), captures a bounded 16 KiB log tail, reports exit code/duration/log, and removes the container on every exit path including lease loss. A non-zero exit or timeout is a permanent failure (never retried automatically); agent/infrastructure errors retry, so delivery is at-least-once. Every result is an application_run_succeeded/application_run_failed deployment event. A run is adopted, not repeated, if the agent restarts while its container exists.

pl down sets teardown_queued; once no ingress route of the deployment is still publishing, active or draining (traffic is removed first), the scheduler enqueues one teardown_application per targeted instance and recorded generation (idempotency application:<id>:teardown:<generation>), aimed at active_node_id, marks the instance draining and cancels its queued ensure/run work. The gateway leases it only to that node, only for the recorded generation, only while the deployment is teardown_queued (no network admission is needed to stop containers). The agent removes every container of that instance up to that generation plus its probe and run containers, refusing a newer generation or a container labelled for another organization/deployment. Only an accepted removal marks the instance removed (the sole path besides pre-seal reservation reclaim) and releases its deployment_role allocation; a terminal failure leaves it error with capacity held. Artifacts, snapshots, events and allocation history are retained; environment gateway capacity and membership are released only by the environment network teardown below. npm run test:application-lifecycle (scripts/application-lifecycle-e2e.sh) drives the real scheduler, mTLS gateway, agent and Docker through release-before-web, a failing job (exit 7), a cron run with the next minute skipped while it runs, and teardown mid-run leaving no labelled Docker objects; admission evidence is stubbed in SQL. Not done: cron catch-up policy beyond “latest missed tick”, time zones/@macros, per-role resource plans and runtime limits, releasing one-shot capacity between runs, parallel runs (the agent workload lane is serial, so a long run delays other work on that host), and live log streaming.

Environment network teardown (migration 0035, crates/pushleft-scheduler/src/network_teardown.rs, crates/pushleft-agent-gateway/src/network_teardown.rs, NetworkManager::teardown in the agent). Order: routes removed, then containers (above), then the network. When every instance of a teardown_queued deployment is removed and none of its routes is publishing/active/draining, the scheduler looks at the environment’s current segment. If another deployment of the environment is still active (queued through degraded) or a database instance of the environment holds capacity, the shared network is retained and the deployment becomes destroyed at once (deployment_destroyed, environment_network: retained). If another torn-down deployment is still removing containers, it waits so both share one network teardown. Otherwise, in one transaction, the segment becomes destroying (irreversible; a trigger refuses any other move), queued network work for its attachments is failed, and each unreleased attachment gets one manage_network job targeted at its node with the task remove_environment_gateway built by the SQL function network_teardown_task (gateway name, bridge pl_<segment>, state path from the prepared gateway task or none, generation) and idempotency network:<attachment>:teardown. The gateway leases, renews and accepts it only while network_teardown_authorized holds: the authenticated node owns the unreleased attachment, the segment is destroying, environment, generation, credential revision and the task equal durable state, no deployment of the environment is active, and no other network attempt for the attachment holds a live lease. The agent’s network lane removes, in order and only when labelled for this environment: the gateway container (after a best-effort tailscale logout), host routes and the nftables table (Linux only; owned-protocol rules/routes and the pushleft:<environment> table, then their fences so a later network starts fresh), the bridge (Docker refuses while a running container is attached: a retryable failure), the mesh state directory (marker-checked, confined to the state root) and key material. Each step treats “already gone” as done, so a crash between steps and a re-lease converge; foreign objects are refused (container, state) or left and reported (bridge: foreign_retained). The receipt (status: removed plus per-step evidence) sets the attachment gateway_status='removed', released_at and teardown_evidence; when no unreleased attachment remains, finish_environment_network_teardown releases the memberships (network_attachment_requests.released_at; the subnet slot is kept and the sequence never rewinds) and their network_attachment capacity, marks the segment destroyed, and moves every finished teardown_queued deployment of the environment to destroyed with deployment_destroyed. A segment without attachments never dispatched host work and is released the same way without a job. Retries are bounded by the job’s attempts; a terminal failure or an expired final lease runs fail_environment_network_teardown: the deployments become teardown_failed (deployment_teardown_failed), the segment stays destroying, capacity stays held, and a repeated pl down does not mask it. Released rows refuse any update or delete. A destroyed segment is history: network_segments is unique per environment only while not destroyed, so the next deployment of the environment gets a new segment (new bridge, gateway names and subnet slots); a deployment created while the segment is destroying is refused with 409. Tests: npm run test:application-placement (scheduling waits for removal, shared network retained, attachment-free release, lease fencing against other nodes/forged tasks/active siblings, receipt release and slot burning, bounded failure), the agent unit test teardown_is_replay_safe_and_owner_scoped (interrupted teardown converges, foreign state refused), npm run test:application-lifecycle (live: labelled gateway stand-in and bridge removed, destroyed, slot burned) and npm run test:pl-up-demo (destroyed, then a redeploy on a new segment with a higher slot). Not done: deleting the node record on the Headscale coordinator (the agent only logs the gateway out), any Linux run of nftables/route removal, operator retry of teardown_failed, releasing role capacity held by failed deployments, and pl down of a deployment whose instances were never placed (they stay pending_placement/reserved, so it stays teardown_queued).

Ingress publication (critical-path step 4). The scheduler (crates/pushleft-scheduler/src/ingress.rs) publishes only after durable health: every web instance of a deployment, across all web roles, must be ready (written by the gateway from a fenced result) on the node holding its active deployment_role allocation. The hostname is derived, never taken from the manifest: <branch>--<project>.<base> for previews (the preview/ prefix of the environment slug is stripped; over-long labels keep a stable hash suffix) and <project>.<base> for production, where <base> is the operator’s PUSHLEFT_PREVIEW_BASE_DOMAIN. Without that variable nothing is published. The first environment to publish a hostname owns it in ingress_hostnames (migration 0025); any other environment or organization deriving the same name gets ingress_rejected and a failed deployment. Each publication takes the next route generation for the hostname and writes one publishing row in ingress_analytics_routes plus one typed publish_ingress job per node, holding the exact task. Only the first web role in plan order is routed. Its replicas are the upstreams. The agent gateway (crates/pushleft-agent-gateway/src/ingress.rs) leases and completes that job only while the payload equals the durable task, the route is still publishing, no newer generation for the host/node was ever active, the node is not owned by another organization, and each upstream instance is still ready at the routed generation on that node with an active allocation. A lease that fails these checks makes the job, route and not-yet-live deployment fail (ingress_failed). A completion that fails them returns 409 and records nothing. The agent (crates/pushleft-agent/src/ingress.rs, --caddy-admin) inspects each pl-app-<instance>-g<generation> container, requires its organization/environment/deployment/instance/generation labels and running state, reads its address on pl_<segment>, and writes the route through CaddyClient (ETag compare-and-swap plus exact readback) into the operator bucket pushleft-ingress-router. Caddy route IDs are pl-ingress-<hostname>-g<generation>. A newer generation already in Caddy makes the task permanently stale. A new generation is inserted in front of older ones (first match wins), so the old route keeps serving until its own exact-generation removal. On success the gateway demotes the previous active route to draining and activates the new one. A trigger rejects activation behind a newer generation, identity rewrites and backward status moves, and a unique index allows one active route per host/node. Only when all of the deployment’s routes are active does it set the deployment live and append deployment_url_ready with the URL. The previous live deployment becomes superseded, and older deployments of the environment that are still building become cancelled, so they can never publish later. The scheduler then queues an exact-generation remove_ingress for each draining route, or for a failed publication that may have run. After the route is removed, it queues one stop_application per node. The agent stops the owned containers with docker stop and does not delete them; the gateway marks the instances removed and releases their role capacity. Artifacts, instances, routes and events remain as history. Teardown (teardown_queued) drains active routes first. Stopping a torn-down deployment’s containers belongs to the fenced teardown work. A release that fails before live never touches the previous route. The API returns url on GET /v1/deployments[/{id}] only from an active route row, and pl list prints it. The tenant RLS policy on routes is now read-only. pl up follows the SSE stream by default (--detach returns after queueing). It prints build_*, application_ready/application_failed (now appended by the gateway), ingress_* and the final URL, and exits non-zero with the bounded log tail on a terminal build_failed, application_placement_rejected, application_failed, ingress_failed or ingress_rejected.

Ingress TLS (custom domains and wildcard previews, migration 0036). The ingress Caddy runs the base configuration printed by pushleft-agent ingress-caddy-config (ingress_base_config in crates/pushleft-mesh/src/caddy.rs): one server pushleft-ingress listening only on :443, whose routes are the pushleft-ingress-router bucket followed by a terminal 404; automatic HTTPS stays on, so Caddy adds the TLS connection policy and a port-80 server that only answers ACME HTTP-01 and redirects everything else with 308 to https://; apps.tls.certificates.load_files and .automate start empty and belong to the agent; on-demand TLS is never configured, so no SNI name can make Caddy order a certificate by itself. --acme-ca/--acme-email/--acme-trusted-root/--acme-challenge set the issuer for custom domains (default: Caddy’s Let’s Encrypt then ZeroSSL). With PUSHLEFT_INGRESS_TLS=1 or the wildcard files, the agent fails a publication closed (ingress_tls_unconfigured) unless the server holding the bucket listens only on the HTTPS port with automatic HTTPS and redirects on; it never falls back to plain HTTP.

Preview wildcard, mode (a), implemented and tested. PUSHLEFT_INGRESS_WILDCARD_CERT_FILE/_KEY_FILE (absolute paths that the agent and Caddy both read). Before every preview publication, and every 30 s in pushleft-agent run, the agent parses the certificate, refuses an expired one or one whose DNS SANs do not cover the hostname (wildcard_certificate, RFC 6125 single-label *), and keeps exactly one load_files entry tagged pushleft-ingress-wildcard plus sha256:<certificate digest>. A renewed file changes the tag, which makes Caddy reload both files; the key is never read by the agent, logged or sent through the admin API. Names covered by a loaded certificate are never ACME-managed.
Preview wildcard, mode (b), operator option, not implemented by Pushleft. The pinned image (caddy@sha256:13b7fb…, v2.11.6) has no dns.providers.* module (caddy list-modules), so DNS-01 cannot run in it. An operator who wants Caddy to obtain the wildcard builds xcaddy build v2.11.6 --with github.com/caddy-dns/cloudflare (zone-scoped Zone.DNS:Edit token), adds *.<base> to apps.tls.certificates.automate with an automation policy {"subjects":["*.<base>"],"issuers":[{"module":"acme","challenges":{"dns":{"provider":{"name":"cloudflare","api_token":"{env.CF_API_TOKEN}"}}}}]}, and runs the agent with PUSHLEFT_INGRESS_TLS=1 and no wildcard files. Pushleft then checks only that the server is HTTPS-only. The repo’s pushleft-cloudflare client is not wired into Caddy.
Custom domains. POST /v1/custom-domains {hostname, project, environment} (member+, pl domain add) records a pending_verification claim with a random 128-bit token; GET/DELETE and pl domain list|show|remove read and remove it (RLS lets the tenant only insert pending claims and set removed). The scheduler’s own loop (crates/pushleft-scheduler/src/domains.rs, every 5 s, at most 8 checks, never inside a placement tick) asks the resolver in PUSHLEFT_DOMAIN_VERIFY_NAMESERVER (default: /etc/resolv.conf, then 1.1.1.1) over UDP with a random query ID, a 3 s timeout and 2 tries, for TXT _pushleft-challenge.<host> equal to the token, else a CNAME from the host to the environment’s derived preview hostname. Failures back off from 15 s to 10 min and the claim fails after verify_by (72 h); each distinct reason, the verification and the failure are custom_domain_events. A verified claim is unique across organizations (partial unique index); a name that is another environment’s preview hostname, or lies under the preview base domain, fails. Both sides of a custom/preview collision take one advisory lock. Verified names ride along with every new release (one route per hostname/node, managed_certificate: true in the task) and a name verified after go-live is attached to the live release once. The gateway leases such a route only while the claim is still verified/active for that organization/environment, and the route fence accepts custom-domain ownership on insert and on activation only. Activation sets the claim active; only then does the agent add the name to Caddy’s automate list, where Caddy obtains the certificate with HTTP-01 or TLS-ALPN-01. A generation switch never touches automate, so the certificate is not re-issued; removing the last route of a name removes it from automate. DELETE sets removed; the scheduler drains (or fails) its routes, the deployment stays live on its preview URL, and the claim’s history is kept. deployment_url_ready and pl up/pl list keep the derived preview URL. A custom route that fails on an already-live release leaves the release live and records custom_domain_route_failed.

Operator duties that remain: the DNS record *.<base> (A/AAAA) to the ingress host (created by hand; the record-set API has not been exercised for a wildcard), ports 80 and 443 reachable from the internet for custom-domain ACME, the wildcard certificate’s renewal (mode a), and Caddy’s reachability to the environment bridges (not yet proven on native Linux with the environment nftables policy).

URLs use https://; only the RFC 6761 .localhost development domain stays plain HTTP, unless the scheduler runs with PUSHLEFT_INGRESS_HTTPS=1. Tests: npm run test:ingress (plain HTTP on .localhost, routing and generation fencing). npm run test:ingress-tls (scripts/ingress-tls-e2e.sh): the generated config in the pinned Caddy, fail-closed on a plain-HTTP listener, wildcard served and verified with curl --cacert, 308 redirect, uncovered names refused, hot reload, the scheduler’s DNS check against scripts/fixtures/dns-stub.py, no certificate for unrouted SNI names, a real ACME HTTP-01 order from a local ACME CA (a second pinned Caddy running acme_server, rooted in a per-run CA), a generation switch without re-issuance, automation removed with the last route. npm run test:pl-up-demo runs the whole path over HTTPS, including pl domain add → stub TXT → verified → attached → ACME → HTTPS → pl domain remove. The migrated tests (npm run test:application-placement, domains_tests and gateway ingress::) cover the verification state machine, cross-tenant collision, unverified or removed names never leased, and drain/supersede/rollback with custom routes. Not proven: public DNS, Let’s Encrypt/ZeroSSL, internet-reachable 80/443, DNS-01. Gaps:

A custom domain is active once routed; Caddy issues its certificate asynchronously. ingress_certificate_invalid/_expiring (scripts/ops-alerts.sh with PUSHLEFT_OPS_INGRESS_ADDRESS) is the evidence that it was issued and renews.
A verified custom-domain claim blocks every other organization until its owner removes it; there is no takeover by re-verification.
TXT proves control of the zone, not that the name points at the ingress; ACME keeps failing (Caddy retries) until A/AAAA or the CNAME is in place.
Pushleft does not create the tenant’s DNS records, even for zones it manages in Cloudflare.
Multiple web roles: only the first is routed.
A deployment without a web role never becomes live.
A restarted container whose bridge IP changes needs a new publication; there is no periodic route reconciliation.
The agent verifies by admin readback, not by an HTTP probe through Caddy.
Hostname ownership is first-claim. A managed service needs reserved project namespaces per organization.
Retained stopped containers have no garbage collection.

Terminal deployments have a deliberately narrow reservation-recovery path: the scheduler may release reserved application capacity only before topology sealing, with no network attachment/enrollment anywhere in the segment and no instance execution target, lease history, attempt, executable task or agent result. It marks the unused instance removed and preserves released allocation history. Shared gateway capacity/membership is retained until no database/application allocation remains for that environment on the host; subnet sequence slots are never recycled. Sealed/prepared networks and previously dispatched workloads still require fenced teardown, not this shortcut. Tests cover shared live deployments, rejection of an unsupported second role, prior-dispatch/sealed refusal, concurrent capacity reservation and an uncommitted membership transaction racing topology sealing.

Completion requires production-capable authentication and tenant authorization, isolated source ingestion/build with immutable image provenance, capacity-bounded application placement, private networking with independent firewall/admission evidence, healthy ingress/DNS/TLS publication, and streamed real build/runtime events. Verify with a web-only repository from the real CLI, tenant-negative tests, failed build/start/health/route rollback, restart/reconciliation replay, and fenced teardown. Do not advertise pl up as operational merely because a job is queued or an image-only fixture is green. Existing credential/network safety gates remain required; they are not substitutes for this vertical slice.

Status (17 October 2026): npm run test:pl-up-demo (pl up demo) now runs this verification on real processes: production auth behind the TLS-evidence proxy, pl login/up/list/down, rootless BuildKit to an htpasswd registry, scheduler placement, the mTLS gateway, the Docker runtime and Caddy ingress. It covers a URL that serves the built image, agent and scheduler kills (including one mid-rollout that the re-leased task adopts), a second release that drains the old route and keeps its artifact, cross-organization/viewer/revoked-token refusals, failed build, start, health, hostname and tampered-source cases that leave the live route untouched, and fenced teardown back to Caddy’s 404. Remaining gaps before this slice is complete: network admission is stubbed in SQL (no Headscale, gateways, nftables or host routes), the run uses Docker Desktop and not native Linux, there is one host per pool and no multi-host transport, traffic is HTTPS with a per-run wildcard and a local ACME CA (no public DNS, Let’s Encrypt or internet-reachable 80/443), all services share one database role, and environment network teardown is only exercised against stub networks (no nftables, host routes or Headscale).

Slice 3 — production substrate

Two-region placement, rolling/canary traffic, quotas and metering.
Managed PostgreSQL/ClickHouse backup/restore, PITR and restore drills (implemented on a local store); object-store adapter and encryption, relocation cutover, major upgrades, and volume replication/migration policy.
Vault/OpenBao transit and dynamic credentials.
Cloudflare proxy/WAF automation, custom-domain ACME, and outbound MTA/ESP routing. Openprovider registration and Cloudflare DNS onboarding are implemented in slice 1.
AI provider gateway with BYOK and metered-proxy modes.