Deploying the Pushleft control plane

This is the layout the code supports today, not a target architecture. Every statement here was checked against the code at the time of writing; gaps are marked gap. Operations, SLOs and recovery are in docs/operations.md; runbooks are in docs/runbooks/.

Nothing here has been run on a live Hetzner fleet. The units were written against the binaries’ configuration code and the local test suites, not exercised under systemd on a real host.

Topology

Node units (pushleft-agent*.service, pushleft-otelcol.service) and their provisioning are described in fleet.md.

Ports and flows

Outbound from the control-plane host: PostgreSQL (all services), the registry (scheduler retention deletes manifests), Cloudflare and Openprovider (controller, and API for domain quotes), ClickHouse (central collector, analytics sink). The mesh host reaches PostgreSQL, Vault, Cloudflare and its local Docker/Caddy. Nodes only dial out: gateway, telemetry gateway, registry, Vault, Headscale.

Service users and files

Application nodes also need br_netfilter loaded with net.bridge.bridge-nf-call-iptables=1 persisted (/etc/modules-load.d/pushleft.conf, /etc/sysctl.d/90-pushleft.conf); pushleft-agent manage-firewall refuses to run without it. Cloud-init does this on Hetzner nodes.

/etc/pushleft/<unit>.env files are root:root 0600; systemd reads them before dropping privileges. Each deploy/systemd/<unit>.env.example lists every variable that binary reads, marks secret values and secret file paths, and gives the default. Secret files go under /etc/pushleft/<unit>/, owned by the service user, mode 0600; the binaries refuse key files with group or other permission bits. Because the three source-store services share a UID, each can read the others’ secret files. That is a consequence of the shared store, not a design goal.

useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-controller
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-telemetry
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-ops
install -m 0755 target/release/pushleft-{api,scheduler,agent-gateway,controller,mesh,telemetry-gateway} /usr/local/bin/
install -d -m 0755 /usr/local/lib/pushleft/scripts /usr/local/lib/pushleft/deploy/observability/alerts
install -m 0755 scripts/ops-alerts.sh /usr/local/lib/pushleft/scripts/   # needs psql and curl on the host
install -m 0644 deploy/observability/alerts/control-plane.sql /usr/local/lib/pushleft/deploy/observability/alerts/
install -m 0644 deploy/systemd/pushleft-*.service deploy/systemd/pushleft-ops-alerts.timer /etc/systemd/system/
install -m 0600 deploy/systemd/api.env.example /etc/pushleft/api.env   # then edit; repeat per unit
systemctl daemon-reload

Database roles

All services connect to one database. Run the SQL below as the database owner after the first migration (the GRANT ... ON ALL TABLES statements only cover tables that exist, so rerun them after every migration that adds tables). scripts/capacity-e2e.sh creates these roles with this exact grant set and runs the API, scheduler, gateway, audit and the alert rules under them. That run exercises deployment creation, placement and leasing. The production-auth, source-upload, build-completion, ingress and teardown paths have not been run under these roles. Their test suites use the owner role.

CREATE ROLE pushleft_api        LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS;
CREATE ROLE pushleft_scheduler  LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_gateway    LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_controller LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_telemetry  LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_mesh       LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_ops        LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;

GRANT USAGE ON SCHEMA public TO pushleft_api, pushleft_scheduler, pushleft_gateway,
  pushleft_controller, pushleft_telemetry, pushleft_mesh, pushleft_ops;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO pushleft_api, pushleft_scheduler,
  pushleft_gateway, pushleft_controller, pushleft_telemetry, pushleft_mesh;
REVOKE ALL ON _sqlx_migrations FROM pushleft_api, pushleft_scheduler, pushleft_gateway,
  pushleft_controller, pushleft_telemetry, pushleft_mesh;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO pushleft_api, pushleft_scheduler,
  pushleft_gateway, pushleft_controller, pushleft_telemetry, pushleft_mesh;
GRANT DELETE ON organization_memberships TO pushleft_api;          -- member removal
GRANT DELETE ON network_attachment_requests TO pushleft_scheduler; -- pre-seal reservation reclaim
-- The pre-tenant auth functions; migrations 0027 and 0038 revoke PUBLIC execute.
GRANT EXECUTE ON FUNCTION pushleft_authenticate_token(bytea), pushleft_authenticate_session(bytea, uuid),
  pushleft_redeem_login_code(citext, bytea), pushleft_ensure_user(citext),
  pushleft_user_memberships(uuid), pushleft_rate_limit_hit(text, text, text, integer, integer),
  pushleft_totp_challenge(bytea) TO pushleft_api;
-- Optional: replication details (lag, sync state) in /readyz and the replay_lag alert.
GRANT pg_read_all_stats TO pushleft_api, pushleft_ops;
-- Migration 0024 creates the NOLOGIN group with EXECUTE on publish_release_artifact.
GRANT pushleft_agent_gateway TO pushleft_gateway;
GRANT EXECUTE ON FUNCTION publish_mesh_credential(uuid,uuid,uuid,uuid,bigint,text,text,text,text,timestamptz,text,uuid)
  TO pushleft_mesh;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO pushleft_ops;
ALTER ROLE pushleft_ops SET default_transaction_read_only = on;

Every table except _sqlx_migrations has RLS enabled (checked on a migrated database), so the broad table grant to pushleft_api is bounded by tenant policies; several tables have read-only or no tenant policy and so refuse API writes regardless of the grant. A per-table least-privilege grant set for the service roles has not been worked out (gap). The fleet CLI deletes nothing; the controller daemon’s DELETE needs were not audited beyond its test suite, which runs as the owner.

Migrations

Migrations are the embedded sqlx set in crates/pushleft-api/migrations/; no external sqlx-cli is needed.

• pushleft-api migrate applies pending migrations with DATABASE_URL and exits. Use the owner role.
• pushleft-api (the service) also applies them at startup unless PUSHLEFT_MIGRATE_ON_START=0. Even with nothing pending, sqlx runs CREATE TABLE IF NOT EXISTS _sqlx_migrations, which needs CREATE on schema public. A non-owner pushleft_api role therefore fails at startup unless this is 0. This was verified on PostgreSQL 17 with a role that had only USAGE.
• pushleft-api bootstrap-owner also migrates before it creates the first owner.

Upgrade procedure:

pg_dump --format=custom --file=pushleft-pre-$(date -u +%Y%m%dT%H%M%SZ).dump "$OWNER_URL"   # see the restore runbook
systemctl stop pushleft-api pushleft-scheduler pushleft-agent-gateway pushleft-controller pushleft-mesh pushleft-telemetry-gateway
DATABASE_URL="$OWNER_URL" /usr/local/bin/pushleft-api migrate
# rerun the GRANT ... ON ALL TABLES / SEQUENCES statements above if the release added tables
systemctl start pushleft-api pushleft-scheduler pushleft-agent-gateway pushleft-controller pushleft-mesh pushleft-telemetry-gateway

Migrations are forward-only; there are no down migrations. Rollback means restoring the pre-upgrade dump (control-plane PostgreSQL restore). Agents keep running during the window: leases and heartbeats fail, and agents retry until the gateway is back. Their in-flight leases may expire and the jobs retry under the normal fencing. Version skew between binaries and schema is not tested, so stop all services across a migration.

Control-plane PostgreSQL HA

Files: postgres/primary.conf, postgres/replica.conf, postgres/pg_hba.replication.conf, postgres/pushleft-pg-ha.sh, systemd/pushleft-postgres-basebackup.{service,timer}, systemd/postgres-ha.env.example. Install the postgres/ directory as /etc/pushleft/postgres/ (root-owned, 0755, files 0644, the script 0755) on both nodes; archive_command and restore_command call the script there.

Setup, once (as postgres, with /etc/pushleft/postgres-ha.env loaded: set -a; . /etc/pushleft/postgres-ha.env; set +a):

1. Primary: pushleft-pg-ha.sh configure-primary, then restart PostgreSQL (wal_level, archive_mode). It includes primary.conf, appends the replication hba line for PUSHLEFT_PG_REPLICA_HOSTS, and creates the pushleft_replicator role and slot.
2. Replica, PostgreSQL stopped and the data directory empty: pushleft-pg-ha.sh bootstrap-replica, then start it. It runs pg_basebackup through the slot, includes replica.conf, writes primary_conninfo (with the password) to postgresql.auto.conf (0600) and standby.signal.
3. Primary: pushleft-pg-ha.sh enable-sync, and check SELECT sync_state FROM pg_stat_replication is quorum.
4. Primary: systemctl enable --now pushleft-postgres-basebackup.timer. Copy /var/backups/pushleft/postgres and the archive off the host.
5. ops-alerts.env: PUSHLEFT_OPS_REPLICA_DATABASE_URL for the standby (the read-only pushleft_ops role exists there through replication).

The migration-owner role also needs CREATEROLE: migration 0024 creates the pushleft_agent_gateway group role. npm run test:control-plane-ha runs all of this against the pinned PUSHLEFT_POSTGRES_IMAGE on Docker (image layout: config in the data directory). The Debian/Ubuntu layout in postgres-ha.env.example (config in /etc/postgresql/17/main, binaries in /usr/lib/postgresql/17/bin) is written from the package layout and has not been run. Failover and rejoin: runbook; PITR: restore runbook.

Health and shutdown

The health listener (crates/pushleft-ops) is plain HTTP with no authentication and no data in the response. Bind it to loopback. It answers only the two exact request lines and 404 for anything else.

SIGTERM, which systemd sends on stop, and SIGINT are handled the same way in every binary:

• API and gateways: stop accepting, finish in-flight requests, then exit. Open SSE event streams end cleanly, and pl up/pl logs reconnect from their cursor after 0.5 s. If no API answers at that moment the CLI exits with an error, and the deployment carries on server-side. A drain deadline (30 s for the API and agent gateway, 20 s for the telemetry gateway) closes whatever is still open; TimeoutStopSec=45s sends SIGKILL after that.
• Scheduler and controller: the current reconciliation tick finishes (each runs in its own transaction), then the process exits. The scheduler’s retention task is dropped mid-pass. Its deletes are designed to be re-run (verified-delete, then record).
• Mesh: the tick loop stops, and in-flight coordinator workers are aborted as before. Their job leases expire and retry under the existing fencing.
• Agent: both lanes are cancelled and in-flight child processes are killed, the same as Ctrl-C (agent-gateway.md).