Sight Assist — A calm, honest voice for blind and low-vision users of Mentra Live glasses

The spoken answer comes above all else. No work may precede or delay first audio. UI publishing, history persistence, memory accumulation, place recognition, diagnostics — none of it runs on the path between “the answer is ready” and “audio starts playing.” An os_signpost “assist” interval emits audio-armed → first-audio → first-sentence → record-kicked → answer events so the ordering is checkable in Instruments, not just hoped for. For someone standing in a doorway waiting to know if the path is clear, latency is the product.

Apple Vision is the default for OCR, scene, faces, and places. Cloud vision (OpenAI / Gemini / Anthropic / custom HTTP) is opt-in and used to enrich, never as a hard dependency for core reading and safety output. The app is fully useful with no keys and no network beyond the local Wi-Fi link to the glasses. It guarantees the floor.

Face embeddings, place feature-prints, and area memory live in the app’s Application Support directory and never leave the device unless the user explicitly enables a cloud provider for a given task. Recognition is limited to people the user personally enrolls — this is deliberately not a surveillance or identity-lookup tool. Making privacy structural (data simply isn’t sent) rather than configurable (a toggle you could get wrong) is the engineering-for-outcomes move for a user base disproportionately harmed by data leakage.

The app never guarantees safety or a clear path, and never asserts a person’s identity with certainty. Conservative distance thresholds and hedged phrasing (“this looks like it might be…”) are intentional design, not placeholders. Empty or failed results return an actionable hint (“try improving lighting or moving the camera”) instead of silence or invention. A confident wrong answer to a blind user is worse than no answer — they can’t visually cross-check it. Wrong-name-at-high-confidence is the gating quality metric, not accuracy.

Every outcome is designed to be heard, not seen. Spoken output uses body-relative directions (“on your left”, “ahead” — never “left of the photo”), short TTS-friendly sentences, and leads with safety. Visual UI exists for setup and sighted helpers; it’s a convenience, not the product.

Direct measurement of the Mentra Live camera server revealed constraints no cleverness removes: /api/latest-photo lags a capture by 15–19 seconds; the gallery listing shows it at ~3.7 seconds. Overlapping capture triggers queue and compound in firmware. There’s no non-persisting capture path — every photo writes a gallery file. Rather than paper over these facts with a fake-fast UI, the app tells the truth (“scanning about every 5 seconds”) and re-architects around the constraint. The measured facts are pinned in code comments so nobody optimizes against a limit that isn’t real.

A ~3.7-second still-capture floor makes walk mode miserable. The solution: a dependency-free, in-app RTMP ingest server (Network.framework + VideoToolbox, no third-party libraries) that asks the glasses to publish 960×540 H.264 over shared Wi-Fi. Apple Vision samples the newest frame ~2×/second. Hazards get spoken within a second or two instead of a 5-second cycle. The engineering maturity is in the honest fallback: streaming drains battery faster (stated up front), the firmware refuses to stream below 15% battery, and on any failure the app speaks one honest line and drops back to snapshot polling.

Captures are task-aware. Read-type tasks (label, date, price) demand the full-resolution still; describe-type tasks are served from a live stream frame where 960×540 is plenty. Because a head-mounted camera with slow auto-exposure produces motion blur, stream captures pick the sharpest of the last ~1 second of frames using variance-of-Laplacian scoring. A clearly blurred read-task photo triggers one coached retake (“hold still a moment — taking another photo”); if both stay blurry, the answer opens honestly (“the photo is a bit blurry — here’s what I can make out”). The blur threshold was calibrated against real head-camera frames, not guessed.

Buffered TTS waits for the whole audio file before playing (~7 seconds in the worst case). The app instead streams synthesized PCM chunks so audio starts on the first chunk — measured ~400–530 ms sooner for OpenAI tts-1 and ~110–460 ms sooner for Deepgram Aura, with buffered playback as the universal fallback. Cloud answers also stream sentence-by-sentence, so time-to-first-speech is ~1.3 seconds (OpenAI) or ~0.8 seconds (Anthropic) versus ~7 seconds fully buffered. Measurement was within-stream, first-chunk-vs-whole-clip, because cross-call comparison is network-noisy and flips run to run. Adopt only what you can prove helps.

Reading text “as it comes into view” is trivial to prototype and infuriating in practice: pan across a room and it spews fragments; hold a label in view and it re-reads forever. Two pure, heavily unit-tested policies make it livable. A stability gate requires a text block to appear in roughly the same place across several consecutive samples and be large and centered enough to be intentional before it’s spoken. A dedupe tracker speaks each block once, then fuzzily suppresses it for a ~30-second recency window; it re-arms when the text leaves and returns. When nothing readable is in view, the app stays silent — no “no text found” spam.

Captures are folded, fire-and-forget, into per-area memory: the app recognizes the current area from a whole-image feature-print (or creates a new one), then merges landmarks, hazards, anchor layout, labels, and text with frequency-and-recency weighting. Over repeated visits an area accumulates a stabilized, queryable summary the user can name. Growth is bounded by construction — 50 areas with LRU eviction, capped landmarks and hazards per area — so the on-disk store can’t grow without limit. The accumulation adds zero milliseconds to the speak path because it runs after speech begins.