SpeakTrue

SpeakTrue LA/Canada Load Balancer Plan

Status

Implemented as of 2026-07-28.

The Canada warm-standby origin, dedicated Cloudflare Tunnel, and recurring PBS backup are deployed and healthy. Cloudflare Load Balancing is active on https://web.speaktrue.cc with LA first and Canada as the failover and fallback pool. Both pools are healthy, and a controlled LA-endpoint disable/restore drill proved automatic failover to Canada and failback to LA.

Production remained HTTP 200 throughout the drill and reported release 15ca9d55 from both paths. The remaining operational follow-up is a secure dual-site deployment path for the current single-target workflow so future releases promote the same commit to both origins automatically.

Target Topology

Implemented Canada Runtime

Item Value
Proxmox node haddadiosn2
VM 2201 / hl-speaktrue-web-ca
LAN address 192.168.2.218
Resources 4 cores, 8 GiB RAM, 50 GiB local-lvm
Runtime Debian 13, Docker 26.1.5, Compose 2.26.1
Checkout /home/codex/SpeakTrue, main, commit 15ca9d55
App origin http://127.0.0.1:8000
App status Docker healthy; local /health HTTP 200
Tunnel speaktrue-web-ca
Tunnel ID 465696ec-f132-4f6f-ae4d-c72b4b55e352
Cloudflared 2026.7.3, service active and enabled
Backup pbs-speaktrue-ca-daily, 04:35 America/Toronto
First archive vm/2201/2026-07-29T04:43:27Z

The VM starts automatically, binds Gunicorn to localhost, and uses a repository-scoped read-only GitHub deploy key. Its environment contains only application-required values; the Supabase management token was intentionally excluded.

Cloudflare Tunnel Design

Use one site-specific tunnel per pool:

Both tunnels route the canonical Host header to:

http://127.0.0.1:8000

For a tunnel-backed load balancer, the endpoint address is the tunnel target:

<TUNNEL_UUID>.cfargotunnel.com

Set the endpoint Host header to:

web.speaktrue.cc

Do not use a public origin hostname as the load-balancer endpoint. A hidden hostname may be added later for direct operator canaries, but it is not needed for the tunnel-backed pool contract and must not appear in marketing, normal OAuth redirects, or application configuration.

Wrong-Zone DNS Cleanup

A CLI preflight attempted to create st-web-ca-origin.speaktrue.cc using an origin certificate scoped to haddadn8n.com. Cloudflare instead created the unused record:

st-web-ca-origin.speaktrue.cc.haddadn8n.com

The tunnel has no matching ingress rule for that hostname, so its catch-all returns HTTP 404 and it cannot reach SpeakTrue. It was deleted from the authenticated Cloudflare dashboard on 2026-07-28.

Cloudflare Load Balancer

The account owner approved and activated the Basic Load Balancing subscription at $5/month. The two included endpoints are allocated to SpeakTrue; EndoNote is not configured in Cloudflare Load Balancing.

Component Live configuration
Load balancer web.speaktrue.cc, public and proxied
Description SpeakTrue web LA primary with Canada failover
Monitor speaktrue-web-health
Monitor contract HTTPS GET /health, port 443, expected 200, Host: web.speaktrue.cc
Monitor timing 60-second interval, 5-second timeout, 2 retries, Western North America
Primary pool speaktrue-web-la
Primary endpoint f0380c66-e9b2-4d95-92f8-571c9adab4e7.cfargotunnel.com
Failover pool speaktrue-web-ca
Failover endpoint 465696ec-f132-4f6f-ae4d-c72b4b55e352.cfargotunnel.com
Endpoint Host header web.speaktrue.cc on both endpoints
Pool order LA first, Canada second
Fallback pool Canada
Traffic steering Off, ordered failover
Session affinity Off
Adaptive routing Off
Final state Load balancer healthy; 2 of 2 pools and 2 of 2 endpoints

The pre-existing proxied Tunnel DNS record for web.speaktrue.cc remains in the zone. Cloudflare Load Balancing takes precedence while enabled; disabling or deleting the load balancer restores behavior based on that existing LA record.

Deployment Path

The current .github/workflows/deploy-speaktrue-homelab.yml is single-target. Canada was bootstrapped manually at the exact production commit.

Before automatic failover, add independent deploy-la and deploy-ca jobs that promote the same SHA and verify:

  1. the requested ref is fetched;
  2. WEB_ASSET_VERSION matches the short SHA;
  3. docker compose config succeeds;
  4. the container becomes healthy;
  5. http://127.0.0.1:8000/health succeeds; and
  6. the deployed SHA is reported without exposing environment values.

A hosted GitHub runner cannot reach Canada’s private Twingate address with the current secrets. Choose and verify a secure management path, such as Cloudflare Access SSH with a service token or a tightly isolated self-hosted runner. Do not expose raw SSH to the Internet to reuse the existing action.

App And Auth Compatibility

Keep the public identity stable:

LEGACY_WEB_PUBLIC_ORIGIN=https://web.speaktrue.cc
LEGACY_AUTH_POST_LOGIN_REDIRECT_URL=https://web.speaktrue.cc/

The normal Supabase redirect allowlist remains:

https://web.speaktrue.cc/**
https://speaktrue.cc/auth/callback
https://speaktrue.cc/auth/callback/

Do not add direct origin hostnames unless intentionally testing OAuth against one.

Activation And Failure Drill

Activation and a controlled failure drill completed on 2026-07-28:

  1. Both tunnel endpoints became healthy under the shared monitor.
  2. The load balancer was deployed with LA first and Canada as fallback.
  3. Baseline /health and / requests returned HTTP 200 and release 15ca9d55.
  4. The LA endpoint was disabled in its pool.
  5. Cloudflare reported the load balancer as degraded with 1 of 2 pools available; the only enabled path was Canada.
  6. During that state, /health and / remained HTTP 200 and the root response still reported release 15ca9d55.
  7. The LA endpoint was restored.
  8. Cloudflare returned to healthy with 2 of 2 pools and 2 of 2 endpoints, and final public health and root checks remained HTTP 200.

For future drills, repeat the same endpoint-disable procedure rather than stopping an entire host, restore LA immediately after proof, and verify the dashboard returns to 2 of 2 before closing the change.

Do not treat a shared Supabase or speech-provider outage as a site failure.

Defaults