SpeakTrue

Unified Speech Library Plan

Status

Proposed implementation plan to make reusable speech discoverable across categories and generation workflows without breaking the existing Soundboard.

Product Thesis

Soundboard is already SpeakTrue’s durable audio store, but its primary navigation model is “remember the folder.” A Unified Speech Library should add global retrieval—recent, search, favorites, tags, and source filters—while keeping categories, ordering, offline behavior, and existing save contracts intact.

User Outcome

A signed-in user can search all of their clips, filter to favorites or a generation source, play a result, see where it lives, and reuse it without first navigating to its category.

Current-State Evidence

• public.soundboard_clips already stores owner, category, file name, stable storage path, transcript, generation metadata, duration, order, offline availability, and timestamps.
• Android decodes generation provenance in SoundboardDtos.kt; iOS CloudClipRecord does not yet select or decode metadata.
• iOS and Android query Soundboard tables directly through RLS-backed PostgREST paths.
• Legacy web loads strict-mode clips through supabase_soundboard_categories_service.py and folder-oriented routes.
• There is no first-class favorite column, tag relation, global paginated query, or consistent cross-platform search contract.

Scope

MVP

• global recent view;
• transcript and file-name search;
• favorites;
• user-defined tags;
• filters for category and generation source;
• cursor pagination and lazy signed-URL resolution;
• web, iOS, and Android parity;
• preservation of categories and current Soundboard workflows.

Later

• heterogeneous library items such as text-only drafts and Quick Speak phrases;
• semantic/vector search;
• shared team libraries;
• automatic AI tagging;
• waveform indexing and content-derived audio features;
• bulk import/export redesign.

Non-Goals

• replacing Soundboard categories;
• moving audio objects during the initial rollout;
• persisting signed URLs;
• storing mutable favorites/tags inside generation metadata JSON;
• server-persisting strict-mode web drafts as part of this plan;
• requiring every old client to migrate in one release.

UX Contract

Add Library as a retrieval view, not a second storage system.

• Default view: recent clips across all categories, newest first.
• Search matches display/file name and transcript; result snippets highlight locally without injecting markup from server content.
• Filter chips: Favorites, Category, Source, Offline, and Tags.
• Every result shows name, category, duration when known, favorite state, source label, and offline state.
• Play resolves a signed URL on demand or uses an existing local cache.
• Moving, deleting, or reordering a result uses existing Soundboard mutations and refreshes both views.
• Empty states distinguish no library content, no search matches, offline-only mismatch, and recoverable network failure.
• Search input is debounced, cancellable, and never blocks category browsing.

Data Model

soundboard_clips additions

• is_favorite boolean not null default false
• index (user_id, is_favorite, updated_at desc) where favorite is true
• index (user_id, created_at desc) for recents

Do not put favorite state in metadata; metadata remains immutable generation provenance.

soundboard_tags

Field	Type	Notes
id	uuid	Primary key
user_id	uuid	Owner
name	text	Display name, bounded and trimmed
normalized_name	text	Unique per owner, case-insensitive
color_token	text nullable	Controlled design token, not raw CSS
timestamps	timestamptz	UTC

soundboard_clip_tags

• clip_id, tag_id, and user_id;
• primary key (clip_id, tag_id);
• ownership trigger/RPC ensures clip and tag share the same user;
• cascade when clip or tag is deleted;
• RLS restricts all operations to the owner.

Search strategy

Start with indexed owner/category/favorite/source filters and bounded case-insensitive matching over file_name and transcript. Measure scale and query plans before adding generated full-text vectors. Do not add semantic search until content-processing privacy, cost, and relevance are separately approved.

API Contract

Add soundboard-library-query as the normalized cross-platform read boundary.

Request:

{
  "query": "appointment",
  "favorite_only": false,
  "category_ids": [],
  "tag_ids": [],
  "sources": ["tts", "sts-live-interpreter"],
  "offline_only": false,
  "cursor": null,
  "limit": 30
}

Response:

{
  "items": [
    {
      "id": "uuid",
      "category": { "id": "uuid", "name": "Saved TTS" },
      "file_name": "Appointment reminder.mp3",
      "storage_path": "users/.../soundboard/...",
      "transcript": "Appointment reminder",
      "metadata": { "source": "tts" },
      "duration_ms": 1800,
      "is_offline_available": true,
      "is_favorite": true,
      "tags": [],
      "created_at": "2026-07-14T00:00:00Z"
    }
  ],
  "next_cursor": null
}

The query response does not include a signed audio URL by default. Add soundboard-library-resolve or reuse an owner-scoped existing resolver to return a short-lived URL for one selected item.

Mutations:

• soundboard-library-favorite with {clip_id, is_favorite};
• soundboard-tag-create, soundboard-tag-update, and soundboard-tag-delete;
• soundboard-clip-tags-update with the complete desired owned tag ID set.

All mutations are idempotent and return the resulting resource state. Direct-table compatibility may remain temporarily, but new clients should converge on the normalized contract.

File Ownership Map

Backend

• Add migrations under backend/supabase/migrations/, then refresh 001_core_schema.sql after validation.
• Add backend/supabase/functions/soundboard-library-*/ and tag handler tests.
• Preserve backend/supabase/functions/soundboard-save-generated/ response compatibility.

Web

• Extend web/python-web-app/static/js/index_soundboard.js and the current Soundboard partial/template.
• Add narrow Flask wrappers under web/python-web-app/src/routes/soundboard.py and service boundaries under web/python-web-app/src/services/.
• Extend test_soundboard_main_tab_surface.py, strict/Supabase service tests, and a focused JS runtime check.

iOS

• Extend CloudClipRecord and Soundboard models in ios/SpeakTrue/CloudStorageService.swift and ios/SpeakTrue/SoundboardAudioClips.swift.
• Add library orchestration beside ios/SpeakTrue/SoundboardViewModel.swift; avoid duplicating cache/player logic.
• Add the Library surface from ios/SpeakTrue/SoundboardView.swift or a dedicated view linked from the same navigation shell.

Android

• Extend SoundboardDtos.kt, mappers, and SoundboardRepository.kt.
• Add library state to SoundboardViewModel.kt and UI to SoundboardScreen.kt or a dedicated nested route.
• Extend existing soundboard repository/view-model/parity tests.

Vertical Slices

• L01: Ship global recent and search on web — Risk: high; Depends: query limits and search semantics decision; Mode: HITL.

  Add the read-only query edge function, owner-scoped pagination, no-URL result envelope, web Library view, play-time URL resolution, and focused query/UI tests. No schema mutation beyond indexes.

  After this: a web user can find and play any owned clip without opening its category.

• L02: Add favorite state end to end on web — Risk: medium; Depends: L01; Mode: AFK.

  Add the column/index, idempotent mutation, result field, optimistic web control with rollback, old-client fixture, and migration regression coverage.

  After this: a web user can favorite a clip and filter the library to favorites.

• L03: Add user-defined tags on web — Risk: high; Depends: L01; Mode: HITL.

  Add tag tables/RLS/ownership checks, tag CRUD and clip association contracts, web assignment/filter UI, duplicate-name handling, and delete-cascade tests.

  After this: a web user can organize clips across categories without moving them.

• L04: Add iOS library search and favorites — Risk: medium; Depends: L01-L02; Mode: AFK.

  Decode metadata/favorite/timestamps, add paginated search/recent/favorite state, reuse cache playback, handle expired URLs, and add focused coding/view-model tests.

  After this: an iOS user has the same global retrieval and favorite behavior as web.

• L05: Add Android library search and favorites — Risk: medium; Depends: L01-L02; Mode: AFK.

  Extend DTOs/repository/state/UI, preserve storage-only combined clip handling, add focused tests, and run the complete Android change gate.

  After this: an Android user has the same global retrieval and favorite behavior as web and iOS.

• L06: Add native tag parity — Risk: medium; Depends: L03-L05; Mode: HITL.

  Add native tag assignment/filtering with accessible controls, conflict refresh, and cross-device sync tests.

  After this: tags created on any surface appear and filter correctly on the others.

• L07: Connect generation provenance and conversations — Risk: medium; Depends: L01 and Conversation C08; Mode: AFK.

  Normalize source filters from clip_metadata, ensure saved conversation turns retain provenance, and add a one-action path from turn to library result.

  After this: a user can retrieve speech by how it was created and trace saved turns back to their workflow type.

• L08: Harden cache, migration, and staged rollout — Risk: high; Depends: L04-L07; Mode: HITL.

  Test pagination under mutation, URL expiry, offline cache preference, old clients, query plans, RLS, rollback, and allowlisted production smoke before enabling the Library by default.

  After this: Unified Speech Library is releasable without weakening Soundboard compatibility.

Verification

• Query tests cover ownership, pagination stability, empty query, special characters, combined clips, and bounded limits.
• Mutation tests cover idempotency, cross-user denial, tag duplicate normalization, delete cascade, and old rows receiving defaults.
• Web tests cover debounce/cancellation, signed-URL refresh, optimistic rollback, and XSS-safe snippets.
• Native tests cover additive decoding, local-cache playback, auth expiry, and cross-device refresh.
• Android slices run bash scripts/verify_android_ci_local.sh before commit/push.
• Schema slices run migration regression gates and inspect query plans against a representative dataset.
• Every slice runs git diff --check and accounts for git status --short.

Acceptance Gate

• Search never returns another user’s clip.
• A favorite or tag change is visible on another surface after refresh.
• Existing category detail, ordering, move, delete, combined-file, and offline flows still work.
• Results do not carry stale signed URLs.
• Search is paginated and bounded; rapid query changes cancel obsolete requests.
• Generation metadata remains provenance, not a container for mutable library state.
• No strict-mode local draft is uploaded as a side effect.

Rollback

Disable speech_library_enabled and return clients to category-first Soundboard views. Keep additive columns/tables and dual-compatible decoding in place. Query and mutation functions can remain deployed but unreachable while investigation occurs. Do not remove favorite/tag data during rollback.

Open Decisions

1. Whether Library is a top-level destination or a default subview of Soundboard.
2. Maximum tag count per user and per clip.
3. Search normalization for punctuation, diacritics, and multilingual transcripts.
4. Whether source labels are user-facing or only filters.
5. Dataset size threshold that justifies Postgres full-text search.