pushleft Operating Pushleft
How to run, watch and recover the control plane as the code stands on the ops-readiness branch (based on main at 860d332). Deployment units, ports and database roles are in deploy/README.md. Runbooks are in runbooks/. Everything below is either measured, read from code, or labelled plan or gap. No part of this has run against a live fleet.
Where the signals come from
The control-plane services (pushleft-api, -scheduler, -agent-gateway, -controller, -mesh, -telemetry-gateway) export no metrics: no OpenTelemetry SDK, no /metrics. pushleft-core/src/observability.rs is the manifest’s observability policy (90-day retention, analytics privacy), not instrumentation. Node collectors send host metrics and application OTLP through the telemetry gateway to ClickHouse (pushleft_otel, 90-day TTL). That pipeline describes nodes and workloads, not the control plane.
Every SLI below is therefore computed from durable state the services already write to the control-plane PostgreSQL: deployments, deployment_events, jobs, agent_job_results, fleet_nodes, fleet_allocations, ingress_analytics_routes. The alert rules are SQL (deploy/observability/alerts/control-plane.sql), evaluated every minute by scripts/ops-alerts.sh from pushleft-ops-alerts.timer, with HTTP probes of every /readyz. Firing alerts are journal lines (FIRING <severity> <alert> <runbook> :: <detail>) and a failed unit. Paging integration is not included: hook OnFailure= or a journal shipper to the pager (plan). scripts/capacity-e2e.sh runs all 15 rules against a populated database under the read-only pushleft_ops role, and checks that node_capacity_high fires there.
No metrics pipeline was added for the control plane. The SQL rules need nothing new to run, and they can only reference tables that exist. Add OTLP instrumentation to the central collector when per-request API latency or queue-depth time series are needed (plan).
SLOs
The targets are initial proposals. None has been measured in production, because there is no production. “Window” is the evaluation window of the alert rule.
| SLI | Exact measure | Target | Alert (severity) |
|---|---|---|---|
| Deployment success rate | Deployments created 30 min to 24 h ago: those with a deployment_url_ready event, divided by those plus status='failed' ones. Tenant build errors count as failures until failures are classified (gap). | ≥ 95 % over 24 h (at least 20 deployments) | deployment_success_rate_low (page) |
Time from pl up to live | deployment_url_ready.occurred_at − deployment_accepted.occurred_at, p95 | ≤ 10 min over 24 h | up_to_live_slow (ticket); a single deployment silent for 30 min: deployment_stuck (ticket) |
| Build duration | build_completed − source_verified, p95 (queueing for a build node included). The hard timeout is 30 min. | ≤ 15 min over 24 h | build_duration_slow (ticket); infrastructure-wide failures: build_failures_infrastructure (page) |
| API availability | GET /readyz answers 2xx within 3 s, probed every minute. The proxy’s non-5xx ratio is the better user-facing measure, but Pushleft ships no proxy config (gap). | 99.5 % of minutes per month | service_not_ready (page) |
| API latency | p95 duration of non-streaming requests in the TLS proxy’s access log. The API itself logs no per-request durations at info. | ≤ 500 ms | none shipped (gap) |
| Agent lease freshness | now() − fleet_nodes.last_seen_at for schedulable nodes. The scheduler stops placing on a node at 60 s, and the controller marks it unhealthy at 90 s. | ≤ 60 s | node_heartbeat_stale at 120 s (page); job_lease_orphaned (page) |
| Ingress certificate validity | For every hostname with an active route for at least 15 min, a TLS handshake through PUSHLEFT_OPS_INGRESS_ADDRESS with SNI = hostname must verify the chain and name, and notAfter must be at least PUSHLEFT_OPS_CERT_MIN_DAYS (14) days away. Covers the operator wildcard (previews, renewed by the operator) and Caddy’s ACME certificates (custom domains, renewed by Caddy at about a third of their lifetime). Probed every minute by scripts/ops-alerts.sh; disabled when the address is unset. | 100 % of active hostnames | ingress_certificate_invalid (page), ingress_certificate_expiring (ticket) |
| Ingress route publication latency | ingress_analytics_routes.activated_at − created_at, p95 | ≤ 120 s over 24 h | ingress_publication_slow (ticket); any route publishing > 5 min: ingress_publication_stuck (page) |
| Backup freshness | Plan. main at 860d332 has no backup columns: managed-database backup, restore and PITR are rejected by the agent executor (managed-databases.md), and backups are being built in a separate branch. When that lands, the SLI is the age of the newest verified backup per database instance, with target ≤ 26 h for daily backups, and an alert rule must be added against the real columns. The control-plane database has no scheduled dump at all (see DR). | — | none (gap) |
| Login abuse | auth_events.kind='rate_limited': one row per tripped bucket (address, email or user) and window (authentication.md) | ≤ 20 tripped windows per 15 min | auth_rate_limited_spike (ticket) |
| Control-plane database redundancy | On the primary: every pushleft_* physical slot active, under 64 MiB behind, replica replay_lag ≤ 30 s; the standby (PUSHLEFT_OPS_REPLICA_DATABASE_URL) reachable and in recovery | always | control_plane_replication_lag (page) |
| WAL archive freshness | pg_stat_archiver: no failure after the last success; no completed segment waiting over 5 min (archive_timeout = 60) | always | control_plane_wal_archive_stale (page) |
Supporting alerts: node_capacity_high, node_data_disk_low, agent_certificate_not_rotating, ingress_route_failed, custom_domain_unrouted, registry_image_release_failing, failed_auth_burst. Each rule in the SQL file names its runbook.
Service not ready
service_not_ready means a /readyz probe failed. Readiness is: the API’s database answers within 2 s and is not in recovery (pg_is_in_recovery(); a standby or a fenced old primary is not ready, so a service left pointing at one during a failover stays red); a health listener’s database answers the same check within 2 s; and, for the scheduler, controller and mesh, their loop completed an iteration in the last 30 s. The API’s /readyz body reports database.in_recovery, replay_lag_seconds (on a standby) and, on the primary, each replica’s name, state, sync_state and replay_lag_seconds (null without pg_read_all_stats). A database outage makes all of them fail together: go to control-plane PostgreSQL failover, or restore if both nodes are lost. A single service failing means that process is down or its loop is wedged: check systemctl status and journalctl -u <unit>. Restart=always restarts a crashed process after 5 s.
Audit trail
Three append-only tables. Triggers reject UPDATE and DELETE, for the owner too, unless the triggers are disabled.
| Question | Where it is recorded |
|---|---|
| Who deployed what, and when | deployment_events.kind='deployment_accepted' (environment), plan_resolved, build_queued (source revision). actor_user_id/actor_credential_id (migration 0034) carry the user and the API token or session id of the request. Rows from before 0034 and from development mode have no actor. |
| Who tore down what | teardown_requested with the same actor columns. Then application_removed, ingress_route_*, deployment_superseded. |
| Token lifecycle | auth_events: bootstrap, login (with second_factor), logout, token_created (name, scope, role, service), token_revoked, login_code_created, and failed_auth with a reason. Ids only. Actions taken with a service token record no user_id (gap). |
| Second factor and login abuse | auth_events: totp_enrolled, totp_activated, totp_disabled, recovery_code_used, security_settings_changed, failed_auth (totp_invalid, totp_replayed, recovery_code_reused, challenge_*, totp_required, totp_enrollment_required), and organization-less rate_limited rows (pushleft-api audit --org none). Never a TOTP secret, code or recovery code. |
| Membership changes | auth_events.membership_changed: the acting user, plus {user_id, from, to} in the detail |
| Build outcomes | source_verified, build_completed, build_failed (terminal flag, bounded log tail); agent_job_results for every attempt (node, attempt, outcome) |
| Runtime outcomes | application_ready, application_failed, application_run_succeeded/_failed (exit code, log tail), application_placement_rejected, application_stopped, application_removed, application_teardown_failed |
| Ingress outcomes | ingress_publishing, ingress_route_active/_draining/_removed, ingress_failed, ingress_rejected, deployment_url_ready (URL) |
| Custom domains | custom_domains (status, method, attempts, last_error, created_by) and append-only custom_domain_events: custom_domain_added/_removed (with actor_user_id), custom_domain_check_failed (one per distinct reason), custom_domain_verified (method), custom_domain_failed (reason), custom_domain_routing, custom_domain_route_active/_failed/_removed. No certificate or key material is ever recorded. |
Fleet operator actions (fleet drain, decommission, certificate revocation) and break-glass SQL | Not audited beyond fleet_nodes.status/retired_at and fleet_node_certificates.revoked_at. The runbooks require an operator_* deployment event or a ticket entry for every break-glass write. An operator audit table is a gap. |
Operator read path:
DATABASE_URL="$OPS_URL" pushleft-api audit --org <uuid> --since 2026-10-01T00:00:00Z [--until <rfc3339>] [--limit 10000]The command prints one JSON object per line, oldest first, across the three tables: {at, source: auth|deployment|agent_result, kind, deployment_id, actor_user_id, actor_credential_id, detail}. Log tails (log) are stripped from event payloads and results. The transaction is read-only. It refuses to run under a role that RLS would blind (not superuser, BYPASSRLS or the table owner), because otherwise it would return an empty audit. Tenant owners and admins can already read their own auth_events through RLS. There is no tenant-facing audit endpoint.
What the trail never contains: token, session or login-code secrets (only SHA-256 hashes, kept in the credential tables, not in events), registry or Vault credentials, or environment secret values. Agent results reject secret-shaped field names. Build and run log tails are tenant output, and they are bounded to 16 KiB.
Retention. auth_events, deployment_events and agent_job_results are kept for the life of the control-plane database. No Pushleft code prunes them, and the triggers block deletion. Source and registry retention GC releases objects but keeps every row (source-ingestion.md). Node and workload telemetry in ClickHouse has a 90-day TTL. A bounded audit retention period with archival, and a tenant-erasure procedure that is compatible with append-only tables, are plans. Until they exist, the audit trail is only as durable as the control-plane database backups (see DR).
Capacity
Target: 100 deployments on 10 nodes. npm run test:capacity (scripts/capacity-e2e.sh) is the evidence. Measured on a 2026-10 dev Mac (Apple Silicon, Docker Desktop PostgreSQL 17, debug builds, shared with other work), not production hardware.
What the test does: it creates a disposable migrated database and applies migrations with pushleft-api migrate as the owner. The real API runs as the non-owner pushleft_api role, the real scheduler and real mTLS gateway as their BYPASSRLS roles, all with the grants in deploy/README.md. The test seeds 10 application nodes at 15 500 m CPU / 31 744 MiB each (a Hetzner CPX51 less the fleet host reserve), creates 100 single-replica web deployments through POST /v1/deployments, and stubs each build in SQL (snapshot, leased build job, the real publish_release_artifact). Network admission is stubbed in SQL every second, the way scripts/application-lifecycle-e2e.sh does it. It then waits until every web instance is targeted, checks the invariants, has every node lease once through the gateway, runs the alert rules and pushleft-api audit, and stops every service with SIGTERM.
Assertions, all passing: no node’s reserved plus active CPU or memory exceeds its capacity; every instance has exactly one targeted ensure_application job, on the node holding its single active role allocation; one gateway allocation per (environment, node); each node leases only its own work (nodes with none get 204); the scheduler’s database time in every 2 s window stays under the 2 s tick; node_capacity_high fires; the audit returns rows under the ops role and refuses the RLS-bound API role; all three services exit on SIGTERM within 10 s with a logged graceful shutdown.
| Measure (one run, 2026-10-17) | Result |
|---|---|
API POST /v1/deployments, sequential, 100 calls | p50 8.7 ms, p95 11.5 ms, max 25.9 ms |
| Scheduler start → first instance targeted | 16.4 s (a first pass reserves, a second pass 15 s later targets after admission) |
| Scheduler start → all 100 instances targeted | 32.4 s (406.4 s before the tick drained its queue; see below) |
| Scheduler database time per 2 s window (20 windows, 7 455 statements) | p50 9 ms, p95 721 ms, max 811 ms; slowest single statement 35 ms |
Gateway POST /v1/agent/lease (7 nodes with work) | p50 15.7 ms, max 17.8 ms |
| Placement result | 15, 15, 15, 15, 15, 15, 10, 0, 0, 0 instances per node; fullest node 96.8 % CPU reserved |
What the numbers say:
- Throughput was bounded by the loop, not by the database.
place_oncehandled one placement job per 2 s tick, and each web instance needs two passes (reserve, then target once admission exists), so 100 instances took about 400 s. The tick now drains the queue whileplace_oncereturns true, bounded to 64 placements and 10 s per tick so the readiness heartbeat still beats every tick (crates/pushleft-scheduler/src/main.rs); the same run then took 32.4 s, and the remaining time is the 15 s admission-evidence delay plus the two passes. Database time per 2 s window rose from ~2 % to a p95 of 0.72 s, still inside the tick. - Placement packs; it does not spread. Candidates are ordered by (memory, CPU, id), so nodes fill one at a time and three of the ten stayed empty. One node failure takes out up to 15 of the 100 deployments. There is no anti-affinity or spread (gap).
- CPU is the binding resource. A single-replica web deployment reserves 1 000 m (500 m role plus 500 m environment gateway) and 768 MiB, so a CPX51 holds 15. The target needs at least 7 such nodes; 10 leave about 50 % headroom.
- Connection budget. The pools add up to 86 connections (API 20, gateway 20, telemetry gateway 20, mesh 16, scheduler 5, controller 5), against PostgreSQL’s default
max_connections = 100, before operator sessions andops-alerts. Raisemax_connectionsor add a pooler before scaling any service out.
Not part of this test, and still required before claiming the target:
- Runtime execution of 100 containers was not part of this test. No agent ran, no image was pulled and no container started. A real load test must run real agents on 10 Linux nodes: the agent’s workload lane is serial, so 15 instances on one node start one after another (pull, start, health probe up to 60 s each).
- Real network admission: the mesh worker, and one Headscale coordinator per environment, so 100 on the mesh host, plus gateway, firewall and route evidence on each node.
- Real builds: the build pool’s serial lanes, the 30-minute timeout, and registry push concurrency for 100 sources; source upload with only 2 slots per API process.
- Ingress: 100 routes in the host Caddy, publication latency, and HTTP probes through it.
- Concurrent
pl upand SSE followers against the API, telemetry volume through the telemetry gateway, and sustained churn (redeploys, teardowns, cron) rather than a single ramp. - Failure injection during the load: a node lost, a gateway restart, a database failover.
Disaster recovery
What can honestly be claimed today. “Manual” means a runbook step, not automation. No restore drill has been timed, so no RTO below is measured.
| Component | Current state | RPO | RTO | Gaps |
|---|---|---|---|---|
| Control-plane PostgreSQL | Primary + one synchronous streaming replica (ANY 1), WAL archive, daily base backups, PITR (deploy/postgres). Failover is manual: fence, promote, repoint, rejoin (runbook). | One node lost: 0 for acknowledged commits while sync is on (async after promotion until the replica rejoins, or after disable-sync). Both lost or bad data replicated: PITR to any time covered by the archive (≤ 60 s archive lag). | Measured 4.92–4.96 s (three runs) from SIGKILL of the primary to the first API write on the promoted node (npm run test:control-plane-ha, 2026-10-17, Docker Desktop, debug build): readyz red 4.3 s, fenced 4.5 s, promoted 4.7 s. This is scripted time; human detection and decision come on top. PITR into a fresh container is tested, not timed at size. | Automatic failover with a witness, a moving endpoint (VIP/DNS), off-host/object-store copies of archive and backups, encryption at rest for them, pg_rewind rejoin, restore drills at production size, Debian/systemd layout not run (the e2e uses the Docker image layout), automated drift reconciliation after restore |
Source store /var/lib/pushleft/sources | Local 0700 directory on the control-plane host. No backup. | Total for unbuilt sources. Already-built images are unaffected (they are in the registry). | Uploads in flight fail, and users rerun pl up. | Object-store adapter |
| Gateway server key, agent CA key, telemetry TLS | Files on the control-plane host | Operator’s own backup | Restore files, restart. Without the CA key, no enrollment or renewal. Nodes keep working until their certificates expire (24 h by default), then need replacement. A planned CA change is a two-phase rotation with no outage (fleet ca rotate --begin/--complete, fleet.md); the gateway reloads the files without a restart. | HSM/KMS signer |
| OCI registry | External | The provider’s | The provider’s. Builds fail closed during the outage (runbook). | Secondary registry |
| Mesh coordinator host (Headscale per environment) | Root-owned SQLite and Noise keys under /var/lib/pushleft/headscale | No automated backup | Not designed: losing coordinator state means re-enrolling each environment’s gateways | Coordinator backup and restore procedure |
| Managed tenant PostgreSQL/ClickHouse | Containers on one database node’s XFS volume. Backup, restore and PITR tasks are rejected by the executor. | No Pushleft backup: losing the volume loses the data | none | Backups (in progress on a separate branch), PITR, relocation, HA |
| Application nodes | Stateless containers. A node silent for PUSHLEFT_NODE_LOST_AFTER_SECONDS (default 300) is marked lost: leases fenced, capacity released, instances lost, replacements placed on another admitted node at the next generation. The deployment stays live (or degraded with an event if nothing can take the work). | n/a (stateless) | Automatic: lost-after time + placement + network admission + startup on the new node (about 60 s after the threshold in npm run test:fleet-reclaim, not measured in production) (runbook). A drained but alive node is still moved by redeploy (runbook). | Relocation off a draining (alive) node; new hosts for a sealed environment topology; a returning lost node after a completed CA rotation gets no cleanup list |
| Database nodes, node lost | Instances become error with a database_instance_stranded fleet event. Allocation held, data not moved. | As managed databases above | Manual: recover the host and its volume, or restore from the tenant’s backups | Database relocation, HA |
| Ingress (host Caddy) | Routes pushed once per publication. Caddy autosave restores them on the same host. | n/a | Same host: caddy run --resume. Lost host: the replacement publishes a relocation route on its node once healthy; the old route drains only then. Repoint the wildcard DNS (runbook). | Periodic route reconciliation, shared edge, republish command |
| ClickHouse telemetry | 90-day TTL, no backup | Telemetry loss accepted | Recreate schema | — |
| Region | Previews run in fsn1 only. Production manifests may name two regions, but nothing fails over. | — | — | Two-region failover is not implemented. There is no cross-region replication of the control plane or of tenant databases, and no witness or writer fencing (architecture.md describes the plan). |
The control-plane host is still a single point of failure. Its PostgreSQL has a replica, but failing over to it is a manual runbook. While the host or the database writer is down, running workloads and their routes keep serving, because agents never stop healthy containers on gateway errors. Nothing can be deployed, torn down or changed until both are back.