pl up demo: the end-to-end vertical slice

npm run test:pl-up-demo (scripts/pl-up-demo-e2e.sh, also in test:all) runs the complete path from an ordinary web-only checkout to a URL and back down, with the real binaries. It is critical-path step 5 from docs/claude-handoff.md.

How to run it

export CARGO_BUILD_JOBS=4
nice -n 15 bash scripts/pl-up-demo-e2e.sh

It needs Docker, the dev PostgreSQL (compose.dev.yaml, 127.0.0.1:55432), jq, htpasswd, openssl, python3, git and network access to Docker Hub for the pinned base images. Every name is per run: a disposable database pushleft_plup_<run>, containers plup-<run>-registry/-caddy/-acme, a DNS stub process, networks labelled com.pushleft.e2e=<run>, and free loopback ports. PUSHLEFT_PLUP_E2E_REGISTRY_PORT, PUSHLEFT_PLUP_E2E_HTTP_PORT and PUSHLEFT_PLUP_E2E_HTTPS_PORT override the registry and Caddy ports. On failure the script prints the last log lines, the jobs, the instances and the events, and keeps its temporary directory.

Wall clock on the development Mac (Docker Desktop, warm Cargo and image caches): about 6 minutes (369 s and 366 s on two consecutive runs). Most of it is the BuildKit builds (including the five attempts the broken build gets) and the two 60-second health deadlines of the failed start and failed health cases.

What runs

A node has one pool, so the demo uses two agent processes. Both nodes are shared (no owner_org_id), so the second organization is placed on the same hosts.

What is stubbed, and what is not proven

• Network admission. A background loop writes, for every environment segment, the same evidence scripts/application-lifecycle-e2e.sh writes: a ready coordinator on a mesh-pool node, and fresh gateway, firewall, route and mesh-route rows for the application node. It also creates an internal Docker network named like the environment bridge (pl_<segment>) and connects Caddy to it. No Headscale, Tailscale gateway, nftables or host routes run. Nothing else is stubbed: the build, the registry push and pull, placement, health, ingress, supersession and teardown (including the network-lane teardown job, which finds nothing it owns to remove) all go through the real code.
• The second organization is inserted in SQL with a hashed token, because there is no org-creation endpoint (only bootstrap-owner, which creates the first one).
• Docker Desktop, not native Linux. Containers run in Docker Desktop’s VM. This says nothing about host-kernel hardening, rootless BuildKit on a Linux host, or nftables.
• Local CA and DNS stub, not the internet. The URL is https://<branch>--<project>.pl<run>.localhost/, served with a per-run wildcard and reached with curl --cacert <run CA> --resolve on the Caddy HTTPS port. The custom domain’s certificate comes from a real ACME HTTP-01 exchange, but against a local ACME CA, and its ownership TXT record comes from a DNS stub. Public DNS, Let’s Encrypt/ZeroSSL and internet-reachable 80/443 are not exercised.
• Single host. One build node and one application node on one machine. Multi-host placement, packet transport and failover are not exercised.
• One database role. Every service connects as the pushleft superuser, so missing grants for separate service roles would not show up here.

Evidence, step by step

1. pl login. bootstrap-owner prints a token; pl login stores it and prints owner@plup.test · Demo org.

2. pl up, following events. The checkout is scripts/fixtures/web-only in a git repo on branch demo, with no .pl.yaml. Inference gives one web role on port 8080 with health /healthz. The CLI prints build_queued, source_verified, build_completed, application_ready, ingress_publishing, ingress_route_active and deployment_url_ready, ends with live https://demo--demo-app.pl<run>.localhost/, and exits 0. The hostname returned 404 before the deployment.

3. The URL serves the build over HTTPS. curl --cacert through Caddy returns pushleft build fixture, so the handshake verified the wildcard for that name. Another name under the base domain returns 404, and plain HTTP answers only 308 to the https:// URL. The artifact is <registry>/organizations/<org>/projects/<project>@sha256:…, built by the build node. The container runs that digest read-only, with CapDrop ALL, no mounts and no published ports.

4. pl list shows the deployment live with the URL.

5. Restart replay. The application agent and the scheduler are killed with SIGKILL. The route keeps serving while they are down. After the restart the deployment is still live, the container has the same ID and start time, and no job was added for the deployment.

6. Second release, with a crash mid-rollout. The checkout changes and is committed. The application agent is down while the release builds and is targeted. Once restarted, it is killed again as soon as it has started the new container, before it reported health (the instance is still deploying), and the scheduler is killed with it. The old release keeps serving. After the restart, ensure_application is leased again (attempt 2) and adopts the same container (same ID). pl up exits 0 with the same URL, which now returns pushleft demo release 2. The old route is removed, Caddy holds only pl-ingress-<host>-g2, the old deployment is superseded and its container stopped but kept, and its artifact row and registry manifest remain. pl list shows the new deployment live with the URL and the old one superseded without one.

7. Tenant isolation. A second organization’s owner token gets 404 for the first organization’s deployment on GET, the SSE stream, source upload and teardown. pl down <id> fails with not_found, and pl list shows nothing. The deployment stays live, no teardown_requested event exists, and the URL still serves.

8. Viewer and revocation. A viewer onboarded with a login code cannot pl up (forbidden), and no deployment row is created. A token that worked for pl list fails on the next call after DELETE /v1/auth/tokens/{id} (revoked).

9. Failed build. A Dockerfile whose RUN exits 3 ends pl up non-zero at build_failed (after the five attempts the job allows), printing the log tail with the failing step’s output. The deployment is failed, with no artifact and no route row.

10. Failed start. An image whose process exits at once ends pl up at application_failed. The deployment is failed and has no route row.

11. Failed health. An image whose /healthz returns 500 ends pl up at application_failed, with the same result.

12. Route refused. The second organization deploys a checkout that derives the same hostname. pl up ends at ingress_rejected, the deployment is failed, and the hostname stays owned by the first organization.

13. Tampered source. With the build agent stopped, a deployment is queued, and one byte of its verified source object is changed on disk. When the build agent comes back, the gateway refuses the download (500, integrity check), every attempt fails, the build ends with a terminal build_failed, and no artifact exists.

    In steps 9 to 12 the failing release goes into the live environment, and each time the script checks that the live deployment is still live, that Caddy’s routes did not change, and that the URL still returns the second release.

14. Custom domain. A viewer’s pl domain add is forbidden. The owner’s pl domain add www.demo-<run>.test prints the claim with the TXT _pushleft-challenge.… token and the CNAME target (the preview hostname). A duplicate claim is refused, and the other organization’s pl domain list is empty. With no DNS record the scheduler records a failed check, the claim stays pending_verification, no route row exists, Caddy’s automate list is empty and the name gets no TLS certificate. Once the stub serves the TXT record, the claim becomes verified, is attached to the live release and becomes active. Caddy obtains the certificate by HTTP-01 from the local ACME CA, and curl --cacert returns pushleft demo release 2 on the custom name. The release stays live on its preview URL. scripts/ops-alerts.sh probes both names through the ingress: it reports the 12-hour ACME leaf as expiring under a 1-day threshold, and neither name as invalid. pl domain show lists custom_domain_added, _check_failed, _verified, _routing and _route_active. pl domain remove drains the route, Caddy is back to the single preview route, and automate is empty.

15. pl down. pl down prints Teardown queued. Then the routes are removed, Caddy has no routes, the URL returns Caddy’s 404, no container of the deployment exists, the instance is removed, and its deployment_role allocation is released. The artifact, the snapshot and the events (including application_removed) remain. No other deployment of the environment is still active (release 1 is superseded, the failing releases are failed), so the scheduler queues one remove_environment_gateway job on the app node’s network lane. Its receipt releases the attachment (gateway_status='removed', released_at, evidence gateway: absent, bridge: foreign_retained because the stub bridge is not agent-owned and Caddy is attached to it, routes/firewall: not_applicable on macOS), releases the membership (the subnet slot stays burned) and the gateway capacity, marks the segment destroyed, and the deployment destroyed with a deployment_destroyed event. pl list shows it destroyed.

16. pl up again into the torn-down environment. The API creates a new segment for the environment; placement gets a new membership with a higher subnet slot; the URL serves release 2 again.

Integration bugs this demo found

• Build retries reused the previous attempt’s workspace. The name was pl-build-<job>-<first 8 hex of the lease token>. Lease tokens are UUIDv7, so those 8 hex digits are the millisecond timestamp and stay the same for about 65 seconds. A retry inside that window recreated the same path, and Docker Desktop refused the bind mount (bind source path does not exist). The first real failure was then hidden behind four infrastructure failures, and the CLI printed the wrong log tail. The suffix now uses the random tail of the token (crates/pushleft-agent/src/build.rs, workspace_name; test retries_get_a_fresh_workspace_name).
• A failed health check was retried five times. The agent reported an image that never became healthy as a retryable failure, so the gateway re-leased it four more times with a 60-second deadline each. pl up waited about six minutes for a result that could not change. A missed health deadline is now a permanent failure with code unhealthy. Docker and pull errors stay retryable (crates/pushleft-agent/src/application.rs, unhealthy; test a_failed_health_deadline_is_a_permanent_failure).
• The test proxy buffered SSE. forwarded-https-proxy.py read the whole upstream response, so pl up could never follow events in production mode. It now streams text/event-stream.
• The web-only fixture served nothing. Its image had no command (busybox sh exits at once), so it could build but never pass health. It now serves /www with busybox httpd on 8080, including /healthz.

Remaining before this is production-proven

• Native Linux run of the same flow: rootless BuildKit on a Linux host, the host Caddy reaching the bridge address, nftables and the environment firewall.
• Real network admission (Headscale, Tailscale gateways, firewall and route evidence) instead of the SQL stub, and multi-host placement, transport and failover.
• Public wildcard DNS (*.<base> record), a public CA (Let’s Encrypt/ZeroSSL) for custom domains with 80/443 reachable from the internet, and DNS-01 for the wildcard (needs a Caddy build with a DNS provider module; the pinned image has none).
• Separate database roles per service with least-privilege grants.
• Scoped, short-lived registry credentials: build nodes share one push credential, and application nodes need an operator-provisioned pull credential.
• Org creation, so that tenant tests do not need SQL.
• Environment network teardown on a real host: the demo bridge is a stub the agent correctly refuses to remove, and no Tailscale gateway, nftables table or host route exists to remove. npm run test:application-lifecycle removes a labelled gateway stand-in container and the bridge; nftables and route removal only run on Linux and are not exercised by any suite yet.
• Garbage collection of retained stopped containers, and of the role capacity still held by failed deployments’ instances.