Vocal Expression

Development biography · earlier archive

Development profile compiled from project source and retained records in October 2026. Dates and test results describe those records; they do not imply a new build, live acceptance test or public release.

PROFILE 1 of 3 — Vocal Expression Response (VXR / "model1")

Biography

Most voice assistants convert speech to text, discard how something was said, and reply in sentences. model1 is built on the opposite premise: the paralinguistic layer — pitch contour, intensity, steadiness, tempo — is the message, and the most honest, most private reply is one made of the same material. Vocal Expression Response takes a Tuned Assistance Event (the arousal / stability / coherence / tempo / confidence the app has already derived from a person's vocalisation, plus tuned support "knobs") and answers with one bounded prosodic utterance: a short vocal gesture with pitch glides and an intensity envelope, rendered in the same metric space the user submitted. The public entry point is a single call, VXRVocalResponder.respond(to:).

The design corrects a specific earlier failure. An initial version routed replies through a continuous MIDI‑2.0 note‑grid "conductor," which produced constant "boop‑boop" output. That path is now an optional, off‑by‑default DAW harness (emitsMIDI2Harness); it must never drive the audible reply. The audible reply is always one utterance — synthesised, spoken, done — with the live mic suppressed while model1 speaks so its own output can't be mistaken for a new turn.

Intended audience. People for whom words are the wrong channel: assistive and regulation contexts where a calm, wordless acknowledgement lands better than a sentence; privacy‑sensitive settings where nothing should be transcribed or uploaded; and researchers studying expressive, turn‑taking voice interaction. The project's house style is explicitly non‑diagnostic and non‑overclaiming.

Distinctive ideas. First, reply in kind: the answer lives in the same prosodic space as the prompt, so acknowledgement is felt rather than read. Second, a minted personal vocabulary: recurring expressions are clustered on‑device into named "bio‑vocal instruments" (e.g. "calm·clear·slow") that can be replayed and composed into phrases — storage is the mint, the codebook is coinage, usage is value, composition is sentences. Third, privacy‑preserving sharing: two people can have their personal codebooks aligned and a meaning "translated" into the other's vernacular, with only quantised, PII‑free digests ever leaving a device.

What it does, end to end. A Tuned Assistance Event becomes a VXRSupportIntent, then a VXRVocalUtterance via VXRVocalSynthesizer, then audible speech via VXRExpressiveVoice.speakUtterance. The voice is a Klatt‑style source–filter model (VXRFormantVoicer.renderPCM): a Rosenberg glottal pulse through a five‑resonator cascade with coarticulated vowels, lip radiation, vibrato, jitter, shimmer, breath and declination — tunable live. When listening, model1 taps its own microphone , responds after ~100 ms of silence following speech, and from the first ~200–350 ms of a turn infers coarse tone and starts a low‑commitment hum ("mmm/mmehh") that morphs into the committed reply. If the user talks over a reply, VXRInterjectionPolicy decides yield / inflate / inject with a fast‑attack, slow‑release duck. An O(1) response‑spine dictionary and a trained topic head (over Apple sentence embeddings) condition and speed up replies.

A concrete user journey. Douglas opens Panels → Sonic Dialogue Test Bed. He clicks Sample and hears a short, warm "mm‑hmm"‑like gesture; the thread shows the turn and the Phrase tab draws its waveform and mel/PCEN spectrogram. He switches to the live mic and speaks a tense, clipped sentence; model1 primes a quick rising hum, then answers with a settling downward glide. He clicks Barge‑in to interrupt and watches the Log show "yield — user took the floor." Over several turns the Library tab accrues named instruments from his recurring expressions; in the Shared tab he seeds a second speaker profile, aligns the two codebooks, and "translates" his last expression into the other speaker's voice — all without any audio leaving the Mac.

Why it matters. model1 demonstrates that an assistant can acknowledge a person in their own vocal terms, on‑device, without words — and that such expressions can become a shared, learnable vocabulary between people rather than a vendor‑defined one. The honest frontier it names is the cross‑user "universal" codebook; the personal codebook is already tractable and built.

Modules & features (authored vs dependencies)

• Authored: Reply engine (VXRTunedAssistanceEvent→VXRSupportIntent→VXRVocalSynthesizer→VXRVocalUtterance→VXRExpressiveVoice); Klatt voice + live knobs (VXRFormantVoicer, VXRVoicerConfig); live listening/priming/turn‑taking (VXRMicInput, VXROpeningTone, VXRDialogueOnset, VXRInterjectionPolicy); personal library & phrases (Library/VXRExpressionCodebook, BioVocalInstrumentLibrary, VXRPhraseGrammar, VocalExpressionLibraryStore); shared lexicon & federation (VXRSharedLexicon, VXRFederation, VXRFederationPrivacy, VXREncryptedRelayTransport, MultipeerFederationTransport); topic conditioning & spine dictionary (VXRTopic, VXRExpressionDictionary, VXRSpineText). Visitor can:* try Sample/Send‑metrics/live‑mic, hear one spoken gesture, watch waveform+spectrogram, drive the Voice sliders, browse the Library, align/translate in Shared.
• Dependencies (not authored): Apple AVFoundation, NaturalLanguage (NLEmbedding), CoreML (vocoder seam), Combine, SwiftUI, MultipeerConnectivity, CryptoKit, Accelerate/vDSP; local package Kalman.

Technical foundations & local packages

Source–filter (Klatt) synthesis; prosody as the carrier of paralinguistic meaning; on‑device streaming k‑means vector quantisation; federated learning with differential privacy (Laplace, seedable) and Sybil‑resistant median aggregation; Apple NaturalLanguage embeddings + an in‑Swift softmax topic head. Local package: Kalman (the project archive — KalmanProsody, scalar estimators). Neural vocoder seam targets a Vocos‑style CoreML model (22050 Hz / 80‑mel / hop 256), no weights shipped. Package versions not inspected — unknown.

Maturity (dated evidence; unknown marked)

Reply engine, Klatt voice, live mic, priming, interjection, library, shared lexicon, topic head: source‑implemented, verified by standalone swiftc self‑tests + two XCTest suites ; Klatt verified standalone (voiced autocorr ~147 Hz, F1 ≈727 Hz). Federation A/C/D: verified standalone. Federation Phase B (MultipeerConnectivity) & live‑LAN: BUILT, UNTESTED (needs two Macs). Neural vocoder: seam only, no model. Dated build/release/runtime evidence: none in‑repo (unknown).

Roadmap, limitations, FAQs

Roadmap: ship a real CoreML neural vocoder behind the seam; test live‑LAN federation on real devices; richer baked spine text via Foundation Models. Limitations: answers in sound, not words (by design); vocoder seam has no model (Klatt only); Phase B / live‑LAN UNTESTED; coarse novel‑phrasing topic accuracy (~4–5/6, mitigated); no audio on non‑macOS; heavy full build, duplicate … 2.swift files can break compilation. FAQs: (1) Speaks words? No — affect as a vocal gesture. (2) Voice uploaded? No — on‑device; federation shares only quantised PII‑free digests. (3) Why once "boop"? The continuous MIDI‑2 grid; now off‑by‑default. (4) Translate two people's expressions? Yes — my unit → shared unit → your unit. (5) iPhone? Full audio path is macOS‑only today.

PROFILE 2 of 3 — Musical Expression Response (MXR)

Biography

Music‑and‑emotion tools routinely blur two things psychology keeps apart: what a piece expresses (a property of the music, inferable from the signal) and what a listener feels (a property of the person, in context). MXR makes that distinction load‑bearing — it lives in the type system as MXRAffectTrajectory.kind == .perceived | .induced. The engine produces perceived emotion today; induced emotion is reserved for a measured signal from the app's MultimodalRegulation sensors, never inferred from a waveform. Predicted‑versus‑measured is the honest, novel core, and nothing in the module claims to read a brain from audio.

Origin/motivation (documented). The module's README and project memory record the ask plainly: extend VXR to tell what a song means by tonal morphology, link it to NeuroAtlas, relate it to how music makes a listener feel, and emit a thesis — built musicology‑first, with the "perceived ≠ induced" honesty maintained throughout. The through‑line is "a song is a voice too."

What it does. MXRTonalMorphologyEngine.analyze computes tension in three registers — tonal (line‑of‑fifths distance + chord dissonance + chromaticism), sensory (roughness), rhythmic (syncopation + density) — plus a spectromorphological gesture reading (swell/decay/spectral open‑close, after Smalley) and an expectation profile (Meyer/Huron ITPRA and Narmour; a deceptive cadence falls naturally out of the dominant→tonic schema). MXRAffectModel maps these to a nine‑factor GEMS vector plus perceived valence/arousal/tension. MXRMechanismEngine attaches cited induction mechanisms — Juslin & Västfjäll's BRECVEMA and the Salimpoor/Zatorre auditory–reward loop — naming (never fabricating) the listener‑dependent ones. MXRThesisGenerator renders the whole as a Markdown thesis: abstract, tension architecture, gestures, an expectation list, GEMS/valence‑arousal, an explicitly pending "induced" slot, and method/caveats.

How a real song gets in. MXRAudioImport.analyze(fileURL:progress:) is a pure assembly of the app's existing DSP stack — no new DSP. It decodes and runs core MIR via AudioScoreEngine.analyzeDetailed (key, tempo, melody, onsets), derives per‑frame timbre and dynamics from the VoiceMemo transient analyzer plus a global harmonic confidence, extracts a 12‑bin chroma via HarmonicContentExtractor, estimates a chord per bar with the one new pure piece, MXRChromaChord.estimate, maps everything into [MXRMusicalFrame] through MXRAudioAdapters, runs the engine, and publishes a ResearchEvidencePacket through MXRResearchBridge. It streams progress ("Decoding" 0.08, "Timbre & transients" 0.42, …) and returns (result, mechanisms, thesisMarkdown).

Intended audience. Musicians, music‑psychology researchers and curious listeners who want a transparent, source‑anchored reading of why a passage feels the way it does — and a tool scrupulous about the difference between an expressive reading and a clinical claim.

A concrete user journey. Douglas opens Creative Research, clicks "Import audio file…", and picks a track; a finding appears as a card with a tension‑arc sparkline. He opens the detail sheet to read the claims, cited mechanisms, open questions and citations, then clicks "Generate thesis → Notes" to drop the write‑up into the app's Notes. Alternatively he imports the same file from the NeuroAtlas Thesis tab, watches a determinate progress bar, and the thesis loads into the workbench to "Digest" like any text source; the Musical Expression subtab shows the analysis as a staged walkthrough with mechanisms mapped (illustratively) to brain circuitry.

Why it matters. MXR is a working, musicology‑first account of musical meaning that refuses the easy overclaim. By separating expressed from felt — and wiring "felt" only to real sensor measurement (not yet connected) — it turns "how does this music make you feel?" into a testable, honest research question rather than a marketing line.

Modules & features (authored vs dependencies)

Authored: MXRMusicalFrame, MXRMorphology, MXRAffect, MXRMechanisms, MXRThesis, MXRAdapters, MXRResearchPacket, MXRResearchBridge, MXRNeuroStaging, MXRNeuroStagePanel, MXRChromaChord, MXRAudioImport, MXRAudioAdapters, MXRSelfTests. Visitor can: import an audio file or text, read the generated thesis, see the tension‑arc sparkline, browse cited mechanisms and citations. Dependencies (reused, not authored by MXR): local DSP packages AudioScore/SemanticCompute (AudioScoreEngine, STFT/SpectralKalman kernels), VoiceMemo (transient analyzer, HarmonicContentExtractor), Imager (stereo FFT Scope), SpectralProcessor, Kalman; Apple Foundation/SwiftUI.

Technical foundations & local packages

Theory anchors : Lerdahl (tonal pitch space), Meyer & Huron (ITPRA), Narmour (melodic implication), Smalley (spectromorphology), Gabrielsson & Lindström and Eerola & Vuoskoski (feature→emotion), Russell (valence–arousal), Zentner/Grandjean/Scherer (GEMS), Juslin & Västfjäll (BRECVEMA), Salimpoor/Zatorre (reward loop). Local packages (reuse stack, per memory "LINKED+confirmed"): AudioScore, SemanticCompute, VoiceMemo, Imager, SpectralProcessor, Kalman (all under the project archive). Versions unknown.

Maturity (dated evidence)

README auto‑assessed 2026‑07‑31: 🟣 In Progress, 90% — 14 files, 70/70 deterministic swiftc checks green (core 26 · adapters 16 · packet 9 · staging 9 · bridge 4 · audio 6). Slices 1–5b implemented and verified against the real app types; live SwiftUI render + the one NeuroResourceRegistry.all edit "confirm on your Xcode build." Slice 6 (measured closed loop) NOT built (needs sensors + audio). Released/deployed: none recorded (unknown).

Roadmap, limitations, FAQs

Roadmap: slice 6 closed loop (overlay measured MultimodalRegulationState vs predicted affect); v2 per‑frame brightness/spectral centroid ; section‑novelty. Limitations: reports perceived emotion only; felt emotion never inferred from audio and not yet sensor‑wired; affect weights are transparent hand‑set priors, uncalibrated; binaural beats explicitly NOT supported (impossible in a mono mixdown; would need per‑ear detection + a contested entrainment mechanism); HSEQProcessor not offline‑drivable (skipped); live SwiftUI render unverified; non‑diagnostic. FAQs: (1) Does it tell me how a song makes me feel? It predicts what the music expresses; felt emotion is a separate, measured quantity (roadmap). (2) Brain‑region links? Illustrative, literature‑anchored — never a brain‑activity claim. (3) Run headless? Yes — pure core self‑tests with swiftc . (4) Any song? macOS import decodes standard files; rights are the user's responsibility. (5) Binaural beats? No — see limitations.

PROFILE 3 of 3 — Supporting research surfaces (3 modules) + scope note

3a · Creative Research gallery

Identity/category: full‑UI Creative panel; research‑evidence browser; macOS; entertrainment portfolio; related to MXR, Research Evidence Graph, Academic Research. Tagline (18 words): Every analysis the system runs, gathered into one browsable gallery of findings, claims, cited mechanisms and evidence. What a visitor can do: browse a stats dashboard (findings/creative/citations/claims/mechanisms/avg‑nodes tiles + a most‑cited‑mechanisms bar chart), filter All/Creative, open finding cards with an MXR tension‑arc sparkline, read a detail sheet (claims/mechanisms/open questions/citations/provenance), "Import MXR analysis", and "Generate thesis → Notes". Foundations/authored: CreativeResearchGalleryModel.swift (pure importer, swiftc 12/12), CreativeResearchGalleryPanel.swift (SwiftUI), CreativeResearchReport.swift; imports from ResearchPacketStore.shared. Entry: Panels → Creative Research (⌃⌥⇧G), window panel.creativeResearchGallery; gallery tile in "Rehab, Research & Media" (pink, sparkles.rectangle.stack). Maturity: README auto‑assessed 2026‑09‑24, 🟢⚪ Complete·Improvable, 85% (model 12/12 swiftc; panel typecheck‑clean; full build verifies in Xcode). Media: none captured; screen‑record the panel. Public links: none verified. Limits: verified by swiftc/typecheck, not by a live app run here; non‑diagnostic. Appendix: AssistantApp/Sources/Creative/CreativeResearch/ .

3b · NeuroAtlas "Musical Expression" evidence subtab

Identity: an Evidence‑lane subtab inside NeuroAtlas (NeuroAtlasTab.musicalExpression) rendering MXR findings — per‑finding thesis + cited induction mechanisms mapped (illustratively) to associated brain circuitry + tension‑arc sparkline, from the same ResearchPacketStore creative findings; a one‑click "Open in NeuroAtlas" handoff bridges the gallery and atlas. Authored: AssistantApp/Sources/Creative/NeuroAtlas/NeuroAtlasMXREvidencePanel.swift . Maturity: view typechecks; NeuroAtlas nav edits "verify on Xcode" (per memory) — live render unverified here. Media: none captured. Public links: none verified. Limits: mechanism→region mapping is illustrative/literature‑anchored, never a brain‑activity claim. Appendix: memory creative-research-gallery (NeuroAtlas integration section).

3c · Vocal Expression WASM web modules (the site's runnable demos)

Identity: a GoDaddy/static‑hosting package of the vocal‑expression modules for the web. What exists: runnable modules docs/WASM-GoDaddy/modules-src/{vocal_expression_core,vocal_expression_regulation,vocal_expression_scene,expressive_vocal_typeface,…}.wasm, a packaging script, a localhost HTTPS server script, and a fixture builder; the scripts generate a scientific demo page (index.html+app.js) with capture/reasoning/typeface/diplopia panels. Maturity: README auto‑assessed 2026‑07‑21, 🟢⚪ Complete·Improvable, 80%. Critical honest limit: the checked‑in .wasm are deterministic JSON fixtures, NOT compiled SwiftWasm yet ("replace with compiled SwiftWasm outputs when the full project wrapper build is available"); index.html/app.js are generated into /tmp, not checked in. Media: the .wasm modules are the real runnable demo assets; the demo page is generated — screenshot it after running the server. Public links: README uses a YOUR_DOMAIN placeholder — no live URL verified. Appendix: docs/WASM-GoDaddy/README.md (FINDER‑STATUS 2026‑07‑21), docs/WASM-GoDaddy/{serve_wasm_localhost.sh,package_wasm_for_godaddy.sh,build_static_wasm_modules.py,modules-src/*.wasm}.