Accessibility Case Study

Luna Glass

A local-first screen-reading and voice-assistance prototype for accessible computer use without paid cloud services.

Pointer Reading

Reads the screen region around the pointer with a visible high-contrast box and local OCR fallback.

Voice Commands

Uses Ctrl+Alt+M for bounded push-to-talk commands through local Vosk speech recognition.

Offline Speech

Speaks results locally with Piper TTS and supports immediate stop/cancel behavior.

Privacy Boundary

No paid APIs, no cloud OCR, no continuous microphone, and no autonomous actions.

Problem

Standard screen readers and magnification tools do not always match low-vision workflows where the user needs quick context around the pointer, whole-screen summaries, and spoken help without sending screen content to a cloud service.

Solution

Luna Glass combines pointer-aware capture, Linux accessibility APIs, local OCR, local speech recognition, and offline text-to-speech into a bounded assistant workflow. The current accepted strength is dependable reading; contextual explanation is being hardened so Luna can report what it actually captured instead of guessing.

InputHotkey or push-to-talk voice command.

TargetPointer region, current monitor, or focused app context.

ReadAT-SPI semantic text first, OCR fallback when needed.

SpeakLocal Piper voice output with stop/cancel support.

CleanupTemporary captures and audio are cleaned by default.

Architecture

The project is built as a local Electron integration with small native helpers. The architecture favors reliability, privacy, and immediate user control over autonomous behavior.

ElectronGlobal shortcuts, screen capture, overlays, and audio orchestration.

AT-SPISemantic app/control inspection for accessible text.

TesseractLocal OCR when semantic text is missing or incomplete.

VoskOffline speech-to-text for push-to-talk commands.

PiperOffline text-to-speech for spoken responses.

Current Capabilities

Ctrl+Alt+P: read around pointer.
Ctrl+Alt+M: voice command / push-to-talk.
Ctrl+Alt+R: repeat last reading.
Ctrl+Alt+X: stop or cancel.
Full-screen monitor reading by spoken request.
Visible high-contrast region box for predictable pointer reading.

Engineering Decisions

Local first

Screen content, microphone audio, OCR, and speech stay on the machine by default.

Human controlled

The system reads and explains; autonomous clicking or system changes remain out of scope.

Fallback based

Semantic accessibility text is preferred, but OCR keeps the workflow useful when app accessibility is incomplete.

Technology Stack

Electron React AT-SPI Tesseract OCR Piper TTS Vosk STT Linux audio Local-first accessibility

Current Status

Implemented

Pointer reading
Full-monitor reading
Push-to-talk command path
Offline speech output
Stop/cancel flow

In Progress

Highlighted-text reliability
Explanation source diagnostics
Representative app testing
More natural voice behavior

Planned

Local visual explanation audit
Spreadsheet error assistance
Approval-gated corrections

Future Direction

The next milestone is to audit the live capture pipeline for explanation commands. If text sources remain insufficient, the project will evaluate a free local image-understanding adapter before accepting broad visual explanation as reliable.