pushleft Pushleft handbook
The one page to start from. It says what Pushleft is, how the pieces fit, how to run it on a laptop and on servers, and how work on it is done and verified. Everything here points at a longer document; nothing here contradicts them, and when it does, the longer document wins and this page has a bug.
Status (17 October 2026): the whole vertical slice (pl up → isolated HTTPS URL → pl down → destroyed) runs end to end on one Docker Desktop host and on one native Ubuntu 24.04 host with network admission stubbed in SQL. It has never run on more than one host, against live Hetzner, public DNS or a public certificate authority. The honest gap list is in claude-handoff.md.
1. What it is
Pushleft is a self-hostable deployment platform for small teams: run pl up in a checkout and get an isolated URL, with builds, private networking, managed PostgreSQL/ClickHouse, backups, custom domains and telemetry handled by the platform. The product boundary and the invariants every change must keep (identity binding across retries, immutable history, fail-closed admission, no host Docker socket or secrets for tenant builds, subnet slots never reused) are in architecture.md. Read those twenty lines before changing anything.
2. Architecture in one screen
Everything is Rust (edition 2024, sqlx on PostgreSQL 17, axum, rustls) except the React/Vite console. One PostgreSQL database holds all state; row-level security is keyed on app.org_id; history tables are append-only and enforced by triggers.
| Crate | Role | Detail |
|---|---|---|
pushleft-core | Shared contracts: .pl.yaml manifest, plan, typed tasks for build, application, database, volume, network, ingress | — |
pushleft-api | Control plane HTTP API, auth (tokens, sessions, login codes, TOTP, rate limits), SSE event streams, migrations (pushleft-api migrate), bootstrap-owner, audit | authentication.md, source-ingestion.md |
pushleft-cli (pl) | login, up, list, down, domain, totp, whoami, console-code | pl-up-demo.md |
pushleft-scheduler | Places work after admission evidence: builds, application roles, databases, volumes; ingress publication; retention GC; backups; lost-node reclaim; network teardown; custom-domain verification | architecture.md |
pushleft-agent-gateway | The only thing agents talk to: mandatory mTLS, node-targeted leases, heartbeats, fenced completions, enrollment and certificate renewal, PKI hot reload | agent-gateway.md, fleet.md |
pushleft-agent | Runs on every node: rootless BuildKit builds, Docker application runtime with generation fencing, managed databases and backups, volumes, environment mesh gateway, nftables firewall and host routes, Caddy ingress routes, signed self-update | build-executor.md, managed-databases.md |
pushleft-mesh | Headscale coordinator per environment, route approval, Caddy admin client | mesh-coordinators.md |
pushleft-controller | Operator CLI: fleet provisioning and admission, drains, decommission, CA rotation, domain registration and DNS | fleet.md, domain-registration.md |
pushleft-telemetry-gateway, pushleft-ops | mTLS telemetry ingress; health/readiness and SIGTERM handling shared by all services | operations.md |
pushleft-hetzner, pushleft-cloudflare, pushleft-openprovider, pushleft-secrets, pushleft-source | Provider clients, Vault/OpenBao secrets, verified source store | — |
The path of one deployment: the CLI uploads a verified source snapshot → the API records a deployment and resolves the plan → the scheduler queues a build only to a build node with capability and admission evidence → the agent builds in rootless BuildKit from the gateway-delivered exact archive, pushes to the registry, and the gateway publishes the digest under the lease fence → the scheduler places application roles only after the environment network is admitted → the agent starts containers with health checks and generation ownership → after durable health the scheduler publishes Caddy routes and the deployment is live with an HTTPS URL → a later release drains the old route; pl down removes routes, containers, then the environment network, ending in destroyed.
3. Infrastructure
Production topology, ports, service users, database roles and migrations: deploy/README.md. In short:
- Control-plane host: API behind a TLS-terminating proxy that sets the trusted forwarding headers, scheduler, agent gateway, controller, telemetry gateway, central collector, alert timer. Systemd units and env examples in
deploy/systemd/. - Control-plane PostgreSQL: primary plus synchronous streaming replica, WAL archive and base-backup timer, manual fenced failover.
deploy/postgres/, runbooks/control-plane-postgres-failover.md. - Mesh host: one Headscale container per environment behind a Caddy.
- Node pools:
build(rootless BuildKit),application(Docker runtime plus the host Caddy that is the ingress),database(XFS data volume with project quotas). Nodes are provisioned on Hetzner bypushleft-controller fleet provision; cloud-init installs Docker, persistsbr_netfilterand the sysctls, downloads the signed agent, enrolls once and renews its certificate from then on (fleet.md). - External: Vault/OpenBao, an OCI registry with Basic auth, ClickHouse for telemetry, Cloudflare, Openprovider, Hetzner.
- Host preconditions that bit us on native Linux:
br_netfilterloaded withnet.bridge.bridge-nf-call-iptables=1(the agent refuses environment firewalls without it), Caddy 2.7 or newer,psql,jq,htpasswd. Container images are digest-pinned indeploy/images.lock.env.
4. Running it
Laptop (development). Requirements: Rust 1.88+, Node 20+, Docker with Compose. herdr-pushleft opens the tmux workspace; dev/setup writes a private .env.herdr; panes run dev/api, dev/scheduler, dev/web, dev/docker. Development auth is loopback-only and must stay so. Manual steps are in the README.
Demo of the full slice. npm run test:pl-up-demo (about 6 to 7 minutes) runs real processes in production auth mode: CLI, build, registry, runtime, Caddy with a per-run wildcard certificate and a local ACME CA, crash replay, a second release, custom domain, negative tenant/auth/build/health cases, teardown to destroyed and a redeploy. What it proves and what it stubs: pl-up-demo.md.
The website and manual. site/ is an Astro static site: the marketing pages plus the manual, which renders docs/, deploy/README.md and the README in place at build time so it cannot drift from the code, and a three.js topology viewer driven by site/src/data/topology.json (checked against deploy/README.md by npm run test:site). The manual’s sources are a committed copy under site/content/ made by npm run sync-docs; npm test in site/ fails when that copy differs from the repository, so a docs change ships with its sync. Section titles on the home page are pre-rendered by site/scripts/ascii-titles.py (pyfiglet, Pagga). The two colour figures on the home page are real data: npm run figures regenerates site/src/data/figures.json (lines per source file per crate) from the repository and npm test fails when it is stale; site/src/data/demo-timeline.json is a dated record of one measured run. dev/site (or cd site && npm run dev) serves it on port 4720, the site pane of the herdr runtime tab, reachable as https://pushleft-site.test through the local Caddy (the console is https://pushleft.test); cd site && pl up deploys it with site/Dockerfile and site/.pl.yaml.
Servers. Follow deploy/README.md for units, roles and migrations, fleet.md “First live run” for the node checklist, and operations.md for SLOs, alerts (scripts/ops-alerts.sh, SQL in deploy/observability/alerts/), audit and capacity. Runbooks live in runbooks/.
5. Way of working
Trunk and history. Work lands on main. Feature work by parallel agents happened in git worktrees under ../pushleft-wt/<branch> and was hand-merged; none remain. Commits are authored as the repository owner with the Co-Authored-By and Claude-Session trailers. The repository has no remote yet; the dev token that once sat in the baseline commit was rotated and removed from history with git filter-repo.
Verification ladder. Every change: cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, then the suites it touches, one at a time. npm run test:all is the full list (see package.json); each scripts/*-e2e.sh creates a disposable migrated database and real containers with per-run names and ports, so suites can run on a shared machine. --ignored Rust tests need PUSHLEFT_TEST_DATABASE_URL; the scripts set it. CI (.github/workflows/verify.yml) runs the same on Ubuntu, then the Linux-only packet, routing and XFS quota gates.
Native Linux. Docker Desktop hides kernel-level behaviour, so Linux-only suites and the full chain also run on a native host. Locally that was a Lima VM (limactl start pushleft, Ubuntu 24.04 arm64, 4 CPUs, 8 GiB, repo cloned to ~/pushleft, synced with git fetch && git reset --hard origin/main since the Mac checkout is its origin). Suites that need root on Linux (mesh, network agent, XFS quota) escalate only their runtime fixture under PUSHLEFT_MESH_E2E_SUDO=1, never cargo.
Security gate. npm run security: cargo audit, npm audit, gitleaks over history, opengrep, trivy filesystem scan, then the per-platform runtime image scan with no suppression (scripts/runtime-image-scan.sh). The image step is red by decision until 2026-11-16; the time-boxed acceptance is deploy/security/image-risk-register.json, and npm run test:security-policy fails when it expires. Never suppress a package vulnerability to turn it green.
Review. Every turn of agent-produced work was reviewed by an independent Codex review hook, whose findings had to be fixed before the turn could end. Findings that changed behaviour are called out in commit messages and docs rather than silently absorbed.
Machine etiquette on a shared host. CARGO_BUILD_JOBS=4, nice -n 15 for builds and suites, one integration suite at a time per host, at most three implementation agents in parallel. Never kill, renice or reset processes or containers you did not start; never prune Docker.
Rules that do not bend. Development auth stays loopback-only. Nothing unfinished is enabled to make a test pass. Network readiness needs independent durable evidence; reserved capacity is not execution permission. Identities (source, artifact, deployment, node, lease, generation) stay bound across retries. History is immutable; statuses are added, rows are not deleted. Subnet slots are never recycled. Tenant builds never see the host Docker socket, host secrets or privileged mode.
6. Where to look next
- Open gaps and decisions: claude-handoff.md, especially “Production requirements” and “Security release gate”.
- What production readiness still needs that only a live environment can prove: a public domain with real wildcard DNS and Let’s Encrypt, a live Hetzner run with real Headscale admission across more than one host, off-host backup copies, and the decision on the pinned images.