SpeakTrue

Realtime STS Workflow Plan

Status

Implemented in the current working tree. This document records the implemented plan and the verification gates used for the realtime STS checkpoint.

Provider contract checked on 2026-06-23 against the official ElevenLabs API docs:

Summary

Add an account-synced Realtime Speech-to-Speech settings toggle across iOS, Android, and web with two modes:

This is not only a UI switch. It requires a shared settings contract, Supabase session and finalization functions, client streaming code, artifact compatibility, usage accounting, and automatic fallback to the existing batch workflow.

Product Behavior

Architecture Decisions

Data Contract

Settings

Add:

alter table public.user_settings
  add column if not exists sts_workflow_mode text not null default 'batch';

alter table public.user_settings
  drop constraint if exists user_settings_sts_workflow_mode_valid;

alter table public.user_settings
  add constraint user_settings_sts_workflow_mode_valid
  check (sts_workflow_mode in ('batch', 'realtime'));

Update backend/supabase/migrations/001_core_schema.sql if the bootstrap schema snapshot is still being kept current.

user-settings-get and user-settings-update must return and accept:

{
  "settings": {
    "sts_workflow_mode": "batch"
  }
}

Invalid or missing values normalize to batch.

Realtime Session

Create POST /functions/v1/sts-realtime-session.

Request body:

{
  "client_platform": "ios",
  "expected_duration_ms": 30000,
  "audio_format": "pcm_16000",
  "language_code": "en"
}

Response body:

{
  "session_id": "uuid",
  "provider": "elevenlabs",
  "token": "sutkn_...",
  "token_type": "realtime_scribe",
  "model_id": "scribe_v2_realtime",
  "websocket_url": "wss://api.elevenlabs.io/v1/speech-to-text/realtime",
  "expires_in_seconds": 900,
  "commit_strategy": "manual",
  "audio_format": "pcm_16000"
}

Server behavior:

Realtime Finalization

Create POST /functions/v1/sts-realtime-finalize.

Request body:

{
  "session_id": "uuid",
  "committed_transcript": "Hello world.",
  "audio_bytes_sent": 960000,
  "duration_ms": 30000,
  "tts": {
    "voice_id": "CwhRBWXzGAHq8TQ4Fs17",
    "model_id": "eleven_multilingual_v2",
    "language_code": "en",
    "stability": 0.9,
    "similarity": 0.85,
    "speed": 0.75,
    "style_exaggeration": 0.65,
    "speaker_boost": true,
    "response_mode": "artifact"
  }
}

Response body must match the current canonical artifact shape:

{
  "success": true,
  "artifact_id": "uuid",
  "artifact_path": "users/<user-id>/generated/<artifact-id>.mp3",
  "audio_path": "https://signed-url",
  "download_path": "https://signed-url",
  "transcribed_text": "Hello world.",
  "clip_metadata": {
    "source": "sts-realtime",
    "stt_model": "scribe_v2_realtime",
    "tts_model": "eleven_multilingual_v2"
  }
}

Server behavior:

Session Persistence

Add a migration for public.sts_realtime_sessions:

Enable RLS. Users may select their own session rows if needed for diagnostics, but inserts/updates should be service-role only through Edge Functions.

Implementation Sequence

  1. Add the shared settings contract.
    • Create backend/supabase/migrations/<timestamp>_add_sts_workflow_mode.sql.
    • Add public.user_settings.sts_workflow_mode text not null default 'batch'.
    • Add a check constraint for batch and realtime.
    • Update backend/supabase/migrations/001_core_schema.sql if the bootstrap schema remains current.
    • Extend user-settings-get and user-settings-update handler, index, and tests.
  2. Add realtime session persistence.
    • Create backend/supabase/migrations/<timestamp>_create_sts_realtime_sessions.sql.
    • Add RLS and service-role grants matching the repo’s existing migration style.
    • Add helper tests or migration verification coverage if the existing migration gates support it.
  3. Add the Supabase realtime session endpoint.
    • Create backend/supabase/functions/sts-realtime-session/index.ts.
    • Create backend/supabase/functions/sts-realtime-session/handler.ts.
    • Create backend/supabase/functions/sts-realtime-session/handler_test.ts.
    • Register the function in backend/supabase/config.toml.
    • Validate method, auth, platform, expected duration, audio format, and language.
    • Resolve ElevenLabs key server-side and call POST /v1/single-use-token/realtime_scribe.
  4. Add realtime completion and artifact finalization.
    • Create backend/supabase/functions/sts-realtime-finalize/index.ts.
    • Create backend/supabase/functions/sts-realtime-finalize/handler.ts.
    • Create backend/supabase/functions/sts-realtime-finalize/handler_test.ts.
    • Share or extract TTS generation/artifact persistence logic rather than duplicating quota, voice authorization, BYOK, pronunciation, and artifact code.
    • Return the canonical artifact shape used by STS and Soundboard save flows.
  5. Correct Supabase TTS streaming endpoint selection.
    • Edit backend/supabase/functions/tts-generate/index.ts.
    • When responseMode === "stream", call POST /v1/text-to-speech/:voice_id/stream.
    • Preserve existing full, buffered, and artifact behavior.
    • Keep existing quota checks and stream event logging.
  6. Wire iOS settings and realtime STS.
    • Edit ios/SpeakTrue/UserSettings.swift to add STSWorkflowMode, default batch, a UserDefaults key, and a published property.
    • Edit ios/SpeakTrue/UserSettingsSyncService.swift to decode, upsert, apply, default, and missing-column diagnose sts_workflow_mode.
    • Edit ios/SpeakTrue/SettingsTabView.swift to add a Realtime Speech-to-Speech toggle in the STT/STS area.
    • Edit ios/SpeakTrue/AIProxyService.swift to add sts-realtime-session, sts-realtime-finalize, and streaming/finalization helpers.
    • Edit ios/SpeakTrue/STSViewModel.swift to branch on userSettings.stsWorkflowMode; leave the existing batch path intact.
    • Create ios/SpeakTrue/RealtimeSTTService.swift using AVAudioEngine plus URLSessionWebSocketTask to send base64 PCM chunks, receive partial/committed transcript messages, and commit on Stop.
    • Reuse current playback first. Add a dedicated streaming player path only if perceived playback latency is not acceptable.
    • Bump CURRENT_PROJECT_VERSION for concrete iOS app code changes.
  7. Wire Android settings and realtime STS.
    • Edit android/app/src/main/java/com/speaktrue/features/settings/data/SettingsRepository.kt to add stsWorkflowMode, serializer mapping, sanitizer, dirty-field tracking, sync patching, and synced response application.
    • Edit android/app/src/main/java/com/speaktrue/features/settings/ui/SettingsScreen.kt or SettingsSections.kt to add the workflow toggle.
    • Edit android/app/src/main/java/com/speaktrue/features/sts/presentation/STSViewModel.kt to branch between batch and realtime.
    • Edit android/app/src/main/java/com/speaktrue/features/sts/presentation/STSViewModel.kt to expose realtime progress, fallback messages, and artifact state.
    • Create android/app/src/main/java/com/speaktrue/features/sts/data/RealtimeSTSRepository.kt for session token retrieval, WebSocket STT streaming, TTS stream/artifact request, and finalization.
    • Reuse ExoPlayer/temp-file playback first. Add true progressive streaming only if the first pass does not meet latency goals.
  8. Wire web settings and realtime STS.
    • Edit web/python-web-app/templates/partials/index_slow/settings.html to add the workflow control in the Speech-to-Text settings panel, with Speech-to-Speech wording in the label.
    • Edit web/python-web-app/static/js/index_settings.js to load and save sts_workflow_mode.
    • Edit web/python-web-app/src/services/user_settings_service.py to include sts_workflow_mode in strict edge allowlists, edge payload mapping, and local fallback behavior.
    • Edit web/python-web-app/static/js/index_core.js so submitSTSForm() chooses batch or realtime.
    • Keep /api/speech-to-speech as the batch path and preserve its current artifact response contract.
    • Add Flask wrapper routes in web/python-web-app/src/routes/stt.py for realtime session and finalize so legacy web remains behind the existing Flask runtime/auth envelope.
  9. Add rollout controls and diagnostics.
    • Keep batch as the database and local default.
    • Add STS_REALTIME_ENABLED=false as a server-side kill switch for disabling realtime session/finalize functions without a database rollback.
    • Log only redacted session IDs, platform, status, provider status class, and diagnostic labels.
    • Never log transcripts, raw audio chunks, tokens, provider keys, Supabase JWTs, or signed artifact URLs.

Files Expected to Change

Backend

iOS

Android

Web

Verification Plan

Use focused checks per slice, then run broader gates only when crossing contracts.

Backend

npx -y deno@latest test \
  backend/supabase/functions/user-settings-get/handler_test.ts \
  backend/supabase/functions/user-settings-update/handler_test.ts \
  backend/supabase/functions/sts-realtime-session/handler_test.ts \
  backend/supabase/functions/sts-realtime-finalize/handler_test.ts \
  backend/supabase/functions/tts-generate/handler_test.ts

bash scripts/coverage_backend_contracts.sh
bash scripts/regression_migration_paths.sh

iOS

xcodebuild test -project ios/SpeakTrue.xcodeproj -scheme SpeakTrue -destination 'platform=iOS Simulator,name=iPhone 17' -only-testing:SpeakTrueTests/SettingsContractsTests
xcodebuild test -project ios/SpeakTrue.xcodeproj -scheme SpeakTrue -destination 'platform=iOS Simulator,name=iPhone 17' -only-testing:SpeakTrueTests/STTAndSTSViewModelTests

Android

./android/gradlew -p android :app:testDebugUnitTest --tests 'com.speaktrue.features.settings.data.SettingsRepositoryTest' --tests 'com.speaktrue.features.sts.presentation.STSViewModelTest'
./android/gradlew -p android :app:lintDebug

Web

web/python-web-app/venv/bin/pytest \
  web/python-web-app/tests/test_user_settings_service.py \
  web/python-web-app/tests/api/test_stt.py -q

Add or update tests so they prove:

Final Hygiene

git diff --check
git status --short

Rollout Checklist

Explicit Non-Goals

References