Live Translation Subtitles

Real-time speech-to-text and translation system that lets guests follow conversations and events in their own language on their phone. A venue microphone captures speech, a server transcribes and translates, and each guest's device displays rolling subtitles in their chosen language.

Use cases

• Multilingual events — speaker in English, subtitles in Polish/Spanish/etc. for members and guests
• Venue tours and house rules walkthrough — new members get real-time translation on their phones
• Casual conversation — members who don't share a language can follow along
• Workshops and classes — cooking, gardening, or wellness sessions with international participants

Target languages: English, Polish, Spanish, French, German, Portuguese, Italian, Hindi, Arabic

Architecture

┌─────────────────┐
│  Venue Mic       │  USB mic, Jabra Speak, or conference mic
└────────┬────────┘
         │ audio stream (WebSocket or chunked PCM)
         ▼
┌─────────────────────────────────────────────────┐
│   ALPUCA — primary (M4 Mac Mini, Poland venue)   │
│   M4 Pro 10-core · Neural Engine · on-prem       │
│                                                   │
│  ┌───────────┐   ┌──────────────────────┐         │
│  │  STT      │──▶│  Translation Fan-Out │         │
│  │ Deepgram  │   │  DeepL / Helsinki-NLP│         │
│  │ EU / local│   │  (per requested lang) │         │
│  └───────────┘   └──────────┬───────────┘         │
│                             │                     │
│  ┌──────────────────────────▼────────────┐        │
│  │  WebSocket Server                     │        │
│  │  /ws/subtitles?lang=pl                │        │
│  │  /ws/subtitles?lang=en (transcript)   │        │
│  └───────────────────────────────────────┘        │
└─────────────────────────────────────────────────┘
         │ WebSocket push (~1 Kbps per client — text only)
         ▼
┌─────────────────────────┐
│  Guest Phones / Tablets  │
│  (Web app or native)     │
│                          │
│  ┌────────────────────┐  │
│  │ Language picker     │  │
│  │ Rolling subtitles   │  │
│  │ Font size control   │  │
│  └────────────────────┘  │
└─────────────────────────┘

Fallback: Oracle Phoenix (Arizona) when ALPUCA is unavailable

Scenario 1 — LAN (venue WiFi only, no internet required)

Guests connect to ALPUCA directly on the venue's local network. Oracle is not in the data path.

Scenario 2 — Poland, internet-connected guests

Guests have internet access and connect over the public web. ALPUCA sits at the venue in Poland. Oracle Phoenix is in Arizona (~150 ms RTT from Poland).

Decision guide

Components

1. Mic input capture

Up to 8 mics, not all speaking simultaneously. Each active mic is one independent audio stream to Deepgram (~64 Kbps Opus each). Total inbound at 8 simultaneous: ~512 Kbps — well within Starlink Mini's upload.

Multi-mic setup: Deepgram supports multichannel audio — each mic gets its own channel, enabling per-speaker transcription and optional speaker labels. Run one Deepgram WebSocket connection per active mic, or use a multichannel USB audio interface to send all channels in one stream.

Software: PyAudio or arecord capturing 16kHz mono PCM per channel, streamed to ALPUCA via local pipe or WebSocket.

2. Speech-to-text (STT) engine

Option A — Self-hosted on ALPUCA (free, ~3s latency):

• Whisper.cpp with medium or large-v3 model
• M4 Mac Mini handles real-time transcription easily
• ~1.5 GB RAM for medium, ~3 GB for large
• Chunked processing: 5-second audio windows with 1-second overlap
• Output: timestamped text segments

Option B — Cloud STT (cheap, ~1.5s latency):

• Deepgram Nova-2 streaming API ($0.0043/min)
• Native WebSocket streaming — no chunking needed
• Better accuracy, especially for accented speech
• Polish language support included

3. Translation engine

Option A — Cloud translation (best quality):

Option B — Self-hosted on ALPUCA (free):

• Helsinki-NLP/opus-mt models (one per language pair)
• ~500 MB per model, fast inference on M4
• Quality: good for common phrases, weaker on idioms
• Run via Ollama or direct HuggingFace transformers

Translation caching: Cache translated segments by source hash + target language. If 3 guests want Polish, translate once, broadcast to all. Real-time speech is unique, so the cache mainly helps with repeated phrases and greetings.

4. WebSocket server

Tech: Node.js with ws library (matches existing edge function patterns) or Python FastAPI with websockets.

Endpoints:

# Primary (Oracle Phoenix — datacenter)
wss://subtitles.sponicgardens.com/subtitles?lang=en    # Original transcript
wss://subtitles.sponicgardens.com/subtitles?lang=pl    # Polish translation
wss://subtitles.sponicgardens.com/subtitles?lang=es    # Spanish translation

# Fallback (ALPUCA — LAN only)
ws://alpuca.local:8910/subtitles?lang=en
ws://alpuca.local:8910/subtitles?lang=pl

Message format:

{
  "id": "seg_001",
  "text": "Witamy w Sponic Gardens",
  "lang": "pl",
  "source_lang": "en",
  "source_text": "Welcome to Sponic Gardens",
  "timestamp": 1711800000,
  "is_partial": false
}

• is_partial: true for interim results (updates in-place on client)
• is_partial: false for finalized segments (appended to history)

Connection management:

• Track connected clients per language
• Only translate to languages with active listeners
• Heartbeat ping every 30s, auto-reconnect on client side

Guest UI — subtitle view

New screen accessible via the Sponic Gardens web app (progressive web app) and native apps (iOS + Android).

UI layout

┌──────────────────────────────┐
│  Live Subtitles    [EN ▼] A  │  ← language picker + font size
│──────────────────────────────│
│                              │
│  Welcome to Sponic Gardens   │
│                              │
│  The WiFi password is on the │
│  card in your room           │
│                              │
│  ░░░░░░░░░░░░░░░            │  ← partial/incoming text (dimmed)
│                              │
│                              │
└──────────────────────────────┘

Features

• Language picker: Dropdown at top, defaults to phone locale
• Font size: A button cycles Small → Medium → Large → Extra Large
• Auto-scroll: New text appears at bottom, auto-scrolls (pause on manual scroll-up)
• Partial results: Show interim STT in gray, replace with final in white
• Dark mode: Dark background by default (easier to read in event settings)
• History: Keep last 50 segments, scrollable
• Connection status: live reconnecting disconnected

Native app integration

Reconnect with exponential backoff (1s, 2s, 4s, max 30s). Entry point: new tab/button on the app home screen — only visible when the subtitle server is broadcasting (check via GET /subtitles/status).

Mobile app (Android)

Native Kotlin + Jetpack Compose app (com.sponicgardens.sponic). Connects to the ALPUCA subtitle backend via OkHttp WebSocket. Built on ALPUCA, published to Cloudflare R2, registered in Supabase mobile_builds.

Current version

Features

• Google Sign-In — Credential Manager API with GCP Sponica OAuth; falls back to manual name entry
• View Subtitles — WebSocket connection to ALPUCA, real-time translated subtitles with original text shown below
• Speak + View — dual-path capture: Android SpeechRecognizer for live text injection + AudioRecord/MediaCodec for local M4A recording simultaneously
• Settings screen — language picker (9 languages, defaults from device locale), microphone device selector, input gain slider, sample rate (16/44.1/48 kHz), noise suppression, local recording toggle, cloud sync toggle
• Recordings — browse, play, share, and delete local M4A recordings with JSON sidecar metadata
• Audio device management — hot-swap detection via AudioDeviceCallback; auto-prefer USB > Wired > Bluetooth > Built-in; deduplicated built-in mics

Audio recording architecture

The app uses a dual-path approach to capture audio while maintaining SpeechRecognizer access:

┌──────────────────────────────────┐
│  AudioRecord (VOICE_RECOGNITION) │  ← coexists with SpeechRecognizer
│  PCM 16-bit → gain applied       │
│       │                          │
│       ├─> MediaCodec (AAC-LC)    │  → .m4a file on device
│       │   └─> MediaMuxer (MP4)   │
│       │                          │
│       └─> RMS level (VU meter)   │  → animated bar in UI
│                                  │
│  SpeechRecognizer (parallel)     │  → POST /subtitles/inject
└──────────────────────────────────┘

• AudioSource: VOICE_RECOGNITION — chosen specifically because it coexists with SpeechRecognizer on the same device (unlike MIC or CAMCORDER)
• Encoder flush: uses a @Volatile keepRecording flag + join() with 2s timeout instead of coroutine cancellation, ensuring EOS is sent and the encoder drains completely
• Metadata sidecar: each recording writes a JSON file with gain, sample rate, channels, device info, noise suppression state, duration, file size
• Minimum file size: recordings under 1 KB are auto-discarded (empty/corrupt encoder output)

Key files

Build & publish

# One-step build + publish from M4:
cd apps/mobile
./publish.sh "changelog description"

# Publishes to:
#   R2:  mobile/sponic-{version}-debug.apk  (versioned)
#   R2:  mobile/sponic-debug.apk            (latest)
#   Supabase: mobile_builds row with metadata

Supported languages

Language defaults from the device's system locale at first launch. Users can change it anytime in Settings, and the preference persists across sessions.

Planned (not yet implemented)

• R2 recordings bucket — dedicated sponic-recordings bucket for audio uploads, with Supabase metadata table for recording catalog
• Auto-sync on WiFi — background upload of recordings when cloud sync is enabled and device is on WiFi
• Server-controlled auto-gain — server analyzes uploaded recordings and suggests optimal gain level per mic/venue
• iOS app — Swift/SwiftUI native (OAuth credentials ready in GCP, app not yet built)

Network & connectivity

Primary: ALPUCA (Poland venue, Starlink Mini)

Venue internet: Starlink Mini Roaming — portable, brought to every event.

• Guests on Starlink WiFi: connect to ALPUCA as a LAN device — WebSocket latency <5 ms, same as pure LAN mode
• Deepgram EU (Frankfurt): ~25–40 ms via Starlink EU ground stations
• DeepL (Germany): ~20–35 ms via Starlink
• Full end-to-end subtitle latency (cloud STT + translation): ~1.8–2.0 s

Fallback: Oracle Phoenix (Arizona)

• Guests connect to wss://subtitles.sponicgardens.com via Cloudflare Workers proxy or directly to 144.24.51.48:8910
• Use when ALPUCA is down (power failure, ISP outage, hardware issue)
• Adds ~300 ms transatlantic overhead but is always-on with datacenter reliability

Cost analysis

Per-hour cost (50% speaking time = 30 min audio, ~27K chars)

Monthly estimates

One-time costs

• Conference mic: $0–250 (may already have one)
• No additional server hardware — Oracle Phoenix (Always Free tier) handles primary; ALPUCA M4 available for LAN fallback

Implementation phases

Operations — starting ALPUCA server components

Subtitle backend (required)

The subtitle backend runs on ALPUCA at port 8910. It handles WebSocket subtitle broadcasting and speech injection.

# SSH to ALPUCA
ssh [email protected]

# Start in mock mode (no mic, broadcasts test sentences):
cd /Users/alpuca/Documents/codingprojects/alpacapps/live-subtitles
node server.js --mock

# Start in production mode (requires GEMINI_API_KEY for translation):
GEMINI_API_KEY="your-key" node server.js

# Start with Polish source language mock:
node server.js --mock-pl

Verify the backend is running

# From any machine on the network:
curl http://Alpuca.local:8910/subtitles/status

# Expected response:
# {"active":true,"mock":true,"listeners":0,"languages":[],"supported_languages":["en","pl","es","fr","de","pt","it","hi","ar"]}

Endpoints

Running as a persistent service (launchd)

A launchd plist exists at ~/Library/LaunchAgents/com.alpacapps.live-subtitles.plist but requires valid TLS certs at /tmp/subtitle-cert.pem and /tmp/subtitle-key.pem (which don't persist across reboots). For now, start the server manually via SSH. To fix auto-start, either generate persistent self-signed certs outside /tmp or remove the TLS env vars from the plist to run HTTP-only.

Android app (Sponic)

The native Kotlin app connects to the ALPUCA backend. Build on ALPUCA:

ssh [email protected]
cd /Users/alpuca/Documents/codingprojects/sponic/apps/mobile
source ~/.zshrc
./gradlew assembleDebug
# APK: app/build/outputs/apk/debug/app-arm64-v8a-debug.apk

Google Sign-In (OAuth 2.0)

The app uses Android Credential Manager API with Google Sign-In. OAuth credentials are in GCP project Sponica (801803827261), owned by [email protected].

Stored in Bitwarden: GCP Sponica OAuth Clients (2edd05a6) in DevOps-sponicgarden collection. Console: APIs & Services → Credentials.

Dependencies & API keys

Risk & mitigation