SpeakTrue

SpeakTrue LA/Canada Load Balancer Plan

Summary

Build a warm-standby failover setup for web.speaktrue.cc:

• LA remains primary for normal traffic.
• Canada runs the same Docker app as standby.
• Cloudflare Tunnels expose both origins without inbound router ports.
• Cloudflare Load Balancing routes web.speaktrue.cc to LA first and Canada only when LA fails /health.
• GitHub Actions deploys the same commit to both clusters with separate LA and Canada deploy jobs.
• No database migration is required for the load balancer itself; app state remains shared through Supabase Auth, Supabase Storage, Supabase DB, and provider APIs.

Key Changes

Canada Runtime Instance

Provision a Canada Docker VM/container equivalent to the current LA SpeakTrue host.

Required Canada setup:

• Clone Haddadios/SpeakTrue.
• Deploy path: /home/deploy/SpeakTrue/web/python-web-app, unless the Canada host already uses a different deploy convention.
• Copy .env from the LA production shape and adjust only host-local values.
• Keep shared backend values identical:
  • SUPABASE_URL
  • SUPABASE_ANON_KEY
  • SUPABASE_SERVICE_ROLE_KEY, if present
  • DATABASE_URL
  • SUPABASE_STORAGE_*
  • ELEVENLABS_*
  • auth URLs such as LEGACY_WEB_PUBLIC_ORIGIN=https://web.speaktrue.cc
• Run the app locally on Canada as a Docker Compose service bound to 127.0.0.1:8000.
• Verify on the Canada host:
  • docker compose ps
  • curl -fsS http://127.0.0.1:8000/health

Cloudflare Tunnels

Use two site-specific tunnels:

• LA tunnel: speaktrue-web-la
• Canada tunnel: speaktrue-web-ca

Each tunnel routes to the local app:

service = http://127.0.0.1:8000

Create two non-user-facing origin hostnames:

st-web-la-origin.speaktrue.cc
st-web-ca-origin.speaktrue.cc

Tunnel public hostnames:

st-web-la-origin.speaktrue.cc -> http://127.0.0.1:8000
st-web-ca-origin.speaktrue.cc -> http://127.0.0.1:8000

Do not put these origin hostnames in marketing copy, Supabase redirects, or app config. They exist only for Cloudflare Load Balancing health checks and origin routing.

Cloudflare Load Balancer

Create a Cloudflare Load Balancer for:

web.speaktrue.cc

Pool design:

• Pool name: speaktrue-web-prod
• Origin 1:
  • Name: la-primary
  • Address: st-web-la-origin.speaktrue.cc
  • Weight: primary / normal
  • Enabled: true
• Origin 2:
  • Name: ca-standby
  • Address: st-web-ca-origin.speaktrue.cc
  • Weight: standby / failover
  • Enabled: true

Traffic policy:

• Steering: failover priority
• Priority order: LA first, Canada second
• Session affinity: off initially
• Health monitor:
  • Type: HTTPS
  • Path: /health
  • Expected status: 200
  • Interval: 60 seconds
  • Timeout: 5 seconds
  • Retries before unhealthy: 2
  • Retries before healthy: 2

TLS mode:

• Cloudflare edge TLS stays enabled for web.speaktrue.cc.
• Tunnel origins remain plain HTTP locally.
• Do not configure the tunnel service as https://127.0.0.1:8000; Gunicorn is not serving TLS.

GitHub Actions Deploy

Replace the single-target homelab deploy workflow with two deploy jobs.

Secrets:

LA_HOMELAB_SSH_HOST
LA_HOMELAB_SSH_USER
LA_HOMELAB_SSH_PRIVATE_KEY
LA_HOMELAB_DEPLOY_PATH
LA_HOMELAB_SSH_PORT

CA_HOMELAB_SSH_HOST
CA_HOMELAB_SSH_USER
CA_HOMELAB_SSH_PRIVATE_KEY
CA_HOMELAB_DEPLOY_PATH
CA_HOMELAB_SSH_PORT

Workflow behavior:

• Trigger remains pushes to main touching web/python-web-app/**.
• deploy-la and deploy-ca run the same script independently.
• Each job:
  • Checks .env exists.
  • Runs git fetch origin main.
  • Resets deploy checkout to origin/main.
  • Sets WEB_ASSET_VERSION to short commit SHA.
  • Runs docker compose down --remove-orphans.
  • Runs docker compose build --pull.
  • Runs docker compose up -d.
  • Verifies curl -fsS http://127.0.0.1:8000/health.
• deploy-la and deploy-ca should both be required to pass for deployment success.
• Keep jobs separate so a Canada failure does not hide LA deploy status.

App/Auth Compatibility

Keep the public app identity stable:

LEGACY_WEB_PUBLIC_ORIGIN=https://web.speaktrue.cc
LEGACY_AUTH_POST_LOGIN_REDIRECT_URL=https://web.speaktrue.cc/

Do not add origin hostnames to normal auth redirect flows unless direct origin debugging is explicitly needed.

Supabase redirect allowlist should continue to include:

https://web.speaktrue.cc/**
https://speaktrue.cc/auth/callback
https://speaktrue.cc/auth/callback/

Only add origin hostnames to Supabase redirects if intentionally testing OAuth directly against an origin hostname. The default is not to add them.

Monitoring And Verification

Initial rollout checks:

1. Verify LA origin directly:

   curl -fsS https://st-web-la-origin.speaktrue.cc/health
   

2. Verify Canada origin directly:

   curl -fsS https://st-web-ca-origin.speaktrue.cc/health
   

3. Verify the load-balanced hostname:

   curl -fsS https://web.speaktrue.cc/health
   

4. Confirm Cloudflare Load Balancing reports:

   • la-primary healthy
   • ca-standby healthy
   • active routing to LA

Failover drill:

1. Disable the LA tunnel route or stop the LA app container.
2. Wait for Cloudflare to mark LA unhealthy.
3. Confirm:
   • https://web.speaktrue.cc/health remains 200
   • Cloudflare routes to Canada
   • login page loads
   • authenticated session still works
   • STT/TTS basic workflow still works
4. Restore LA.
5. Confirm traffic returns to LA after the healthy threshold.

Operational monitors to add or update:

• Dashboard variables should include both hosts:
  • hl-dockervmla2-speaktrue
  • Canada SpeakTrue Docker host name, once known
• Add or clone alert rules for Canada logs.
• Add an alert for load balancer origin unhealthy if Cloudflare notifications are available.

Test Plan

Before enabling the load balancer:

• LA current production health passes.
• Canada direct origin health passes.
• Canada can complete:
  • app load
  • login redirect flow if tested through web.speaktrue.cc
  • TTS generation
  • STT transcription
  • Soundboard read/write smoke, if safe with a test account
• Both GitHub Actions deploy jobs pass on the same commit.

After enabling the load balancer:

• https://web.speaktrue.cc/health returns 200.
• https://web.speaktrue.cc/ loads the app.
• OAuth start/callback still lands on web.speaktrue.cc.
• Failover drill proves Canada serves traffic when LA is unhealthy.
• Recovery drill proves LA resumes primary role when healthy.

Assumptions And Defaults

• Use LA primary / Canada warm standby, not active-active.
• Use Cloudflare Tunnels for both origins.
• Use two GitHub Actions deploy jobs.
• Keep web.speaktrue.cc as the only user-facing Python web hostname.
• Use hidden origin hostnames only for Cloudflare Load Balancing origin routing and health checks.
• Do not introduce a new database, Redis, or shared filesystem for v1.
• Do not run supabase db push; no DB schema change is required.
• If Cloudflare Load Balancing is not available on the current plan, fallback is manual DNS/tunnel cutover, but that is not the target implementation.