Custom-domain verification and ingress certificates

Use this when a tenant’s custom domain stays pending_verification, turns failed, is verified but not served, or when custom_domain_unrouted, ingress_certificate_invalid or ingress_certificate_expiring fires.

How it works (one paragraph)

pl domain add stores a pending_verification claim with a token. Every 5 s the scheduler checks up to 8 due claims against PUSHLEFT_DOMAIN_VERIFY_NAMESERVER (TXT _pushleft-challenge.<host> = token, or CNAME <host> → the environment’s preview hostname). Misses back off 15 s → 10 min and fail after 72 h. A verified name is routed with the next release, or attached to the live one. The gateway activates the route and sets the claim active, and the agent adds the name to Caddy’s apps.tls.certificates.automate. Caddy then orders the certificate (HTTP-01 on port 80 or TLS-ALPN-01 on 443) and renews it itself. Preview hostnames use the operator wildcard (PUSHLEFT_INGRESS_WILDCARD_CERT_FILE/_KEY_FILE), which the agent reloads within 30 s of a file change. Architecture: architecture.md, “Ingress TLS”.

1. Look at the claim

pl domain show <id>                       # tenant view: status, last_error, events
psql "$PUSHLEFT_OPS_DATABASE_URL" -c "SELECT id,org_id,hostname,status,method,attempts,last_error,next_check_at,verify_by FROM custom_domains WHERE hostname='www.example.com' ORDER BY created_at"
psql "$PUSHLEFT_OPS_DATABASE_URL" -c "SELECT kind,payload,occurred_at FROM custom_domain_events WHERE domain_id='<id>' ORDER BY occurred_at"

2. pending_verification or failed

Run the scheduler’s exact check from the scheduler host (same resolver, same code):

PUSHLEFT_DOMAIN_VERIFY_NAMESERVER=<as configured> pushleft-scheduler check-domain <hostname> <token> <preview-hostname>
dig +short TXT _pushleft-challenge.<hostname> @<resolver>
dig +short CNAME <hostname> @<resolver>

• no TXT ... and no CNAME: the tenant has not created the record, created it in the wrong zone, or quoted it oddly (the value must equal the token exactly, surrounding spaces are ignored). Negative caching can delay a fresh record for the zone’s SOA minimum TTL.
• DNS lookup failed: ... timed out / rcode 2: resolver trouble. Point PUSHLEFT_DOMAIN_VERIFY_NAMESERVER at a working recursive resolver and restart the scheduler. Checks resume on their own.
• DNS answer was truncated: the TXT set is too large for UDP (TCP fallback is not implemented). Ask the tenant to remove unrelated TXT records at _pushleft-challenge.<host>.
• failed with hostname is already verified by another organization or environment: another tenant (or another environment) holds the name. Pushleft has no takeover flow. Confirm who controls the zone today before touching anything. If the other claim is stale, its owner removes it (pl domain remove); an operator does not edit another tenant’s row.
• failed with verification window expired: the tenant fixes DNS, then pl domain remove <id> and pl domain add again (new claim, new token). Claims never move back to pending.
• failed with not a valid custom domain outside the preview base domain: the name is under PUSHLEFT_PREVIEW_BASE_DOMAIN. That is by design.

3. verified but not served (custom_domain_unrouted)

The environment needs a live release with ready web instances.

psql ... -c "SELECT r.id,r.deployment_id,r.status,r.route_generation,j.status,j.payload->>'last_error' FROM ingress_analytics_routes r LEFT JOIN jobs j ON j.idempotency_key='ingress:'||r.id||':publish' WHERE lower(r.hostname)='<hostname>' ORDER BY r.created_at"

• No row: the scheduler attaches one route per (live deployment, hostname), only once. A row that failed stays failed for that release (custom_domain_route_failed holds the agent error). The next pl up publishes the name again.
• Agent error ingress_tls_unconfigured: the agent runs without TLS ingress, or the server holding pushleft-ingress-router is not HTTPS-only. Regenerate the base config with pushleft-agent ingress-caddy-config and load it (step 5).

4. Served but the certificate is wrong (ingress_certificate_invalid)

openssl s_client -connect <ingress>:443 -servername <hostname> -verify_hostname <hostname> </dev/null | openssl x509 -noout -subject -issuer -enddate
curl -s localhost:2019/config/apps/tls/certificates/automate      # must list the custom hostname
journalctl -u caddy | grep '"identifier":"<hostname>"' | tail      # ACME attempts and errors

• Custom domain, no certificate yet: Caddy retries with backoff. The usual causes are DNS not yet pointing at this ingress, port 80/443 blocked from the internet, or CA rate limits (Let’s Encrypt: 50 certificates per registered domain per week, 5 failed validations per hostname per hour). Fix the cause. Caddy picks it up on its next retry, and a restart is not needed.
• The name is missing from automate: the route is not published (step 3). Never add names to automate by hand: Pushleft treats that list as its own.
• Preview hostname: the wildcard does not cover the name (only one label under *.<base>), or the file is expired. Install a renewed wildcard (step 5).

5. Renewing or replacing the wildcard (ingress_certificate_expiring on preview hosts)

1. Write the new certificate and key next to the old ones and mv them over the configured paths (an atomic rename on the same filesystem). Mode 0600, owned by the Caddy user. The agent only needs read access to the certificate.
2. Within 30 s the agent logs ingress wildcard certificate loaded (sha256:…), and Caddy serves the new file. Check with the openssl s_client line above.
3. If the agent refuses it (wildcard_certificate on the next publication), the new certificate is expired or does not cover *.<base>. Restore the old files.

The key must never be pasted into the admin API, events or tickets. The agent passes only its path to Caddy.

Reloading the whole ingress config (lost host, corrupt config): pushleft-agent ingress-caddy-config [--acme-email …] > /etc/caddy/ingress.json, then caddy run --resume on the same host, or load the file and republish routes (see ingress host failure). The agent restores the wildcard entry on its next 30 s tick. Custom-domain certificates come back from Caddy’s storage without being re-issued if /var/lib/caddy survived.

6. Removing a domain

pl domain remove <id> (tenant) sets removed. The next scheduler pass drains its routes, and the agent drops the name from automate after its last route is gone. The certificate stays in Caddy’s storage until it expires. The claim and its events stay as history.