e↗entertrainment
← All project stories
Creative tools / PROJECT STORY

Clouds

Navigate connected ideas in a spatial semantic workspace. Explore graph interaction, local reasoning and the evolving relationship between context and interface.

Semantic workspace research23 MIN READDEVELOPMENT ARCHIVE · OCT 2026
PROJECT REVIEW · 18 October 2026

Inside the project

Prepared with the project’s development agent from source and retained evidence. This is a project-owner assessment, not an independent customer review. Dated checks were not rerun for this website.

This family is the thinking surface of the AssistantApp: three ways of seeing the same local knowledge, none of which needs a network. Clouds is the most immediately striking — a GPU-rendered weather map where an on-device topic modeler clusters tasks, notes, memory and research into drifting cloud fields whose weather states (storm, fog, lightning, pressure) are computed from real signals like contradiction and urgency, with an offline intent engine ranking what you probably need right now. NodeMap is the family's workhorse: an infinite-zoom graph with semantic level-of-detail, per-node voice capture, and a reference-frame layer that completes analogies by vector arithmetic and can materialise the missing concept as a new node. Underneath both, the offline reasoning stack supplies the intelligence: Apple's on-device Foundation Models behind deterministic fallbacks, an author-trained intent classifier, CMAC habit learning, and a measured staged plan-draft-critique pipeline. As an owner review: the strongest working material is the Clouds rendering/intent architecture (full app build passed 2026-06-12; actively extended through October) and the measured staged-reasoning results; the honest weakness is that none of the three surfaces has recorded runtime evidence or a single genuine capture, several strands live only on feature branches, and the staged pipeline is installed but not yet wired to the chat path. It is a deep, coherent, verifiable work-in-progress — not a shipped product.

What stands out

  • Entirely on-device: topic modeling, rendering, intent prediction and reasoning run with no network; the dev host's SSL-blocked network enforces this continuously
  • Real architecture for responsiveness: detached graph builds, debounced stored filters, Metal/MPS instanced rendering with bloom, relevance-budgeted labels
  • Uncertainty is a UI feature: topic confidence, needs-review ratios, evidence spans, critique states and forecast reasons are all surfaced to the user
  • Measured before believed: the staged reasoning pipeline was evaluated on the real on-device model against two baselines before any wiring
  • Cross-pollinating foundations: GraphTransform, zoom-band hysteresis, VXR embeddings and CMAC are shared across all three surfaces

Current limits

  • No runtime verification or genuine captures exist for any surface; all maturity evidence is source/build/test-level
  • Clouds sources are untracked in git per the 2026-08 note (unconfirmed since); NodeMap's futurisation pass is branch-only and never app-built
  • Staged reasoning is installed (2026-09-20) but unwired — feature flag OFF, AIService edits unapplied, planner patch uncommitted; state since 20 Sep unknown
  • Evaluation is a single n=34 run — directional only; trick questions failed in every arm
  • macOS-only throughout; no tests for the Clouds UI layer

Features & evidence

GPU semantic weather map

source

Pan/zoom a Metal+MPS map of topic clouds; weather states from nine-component signals; fbm cloud shader, bloom, animated backdrop

Recorded evidence

Full xcodebuild exit 0 2026-06-12; MSL standalone-compiled same day; files on disk 2026-10-18 (CloudsMetalField.swift)

Intent forecast + mode preemption

source

'Now' strip ranks clouds by blended heuristics + CMAC habit model; one-click suggested-mode switch; per-cloud relevance dims fields and budgets labels

Recorded evidence

In the 2026-06-12 build; CloudsIntentEngine.swift structure re-verified unchanged 2026-10-18

Clouds pane (⌘5), three surfaces

source

Middle-pane destination switching Notes / Tasks / Semantic surfaces, zoom state preserved; window scene kept for deep links

Recorded evidence

CloudsPane.swift verified 2026-10-17; 2026-08-17 build note (SWIFT_EMIT_LOC_STRINGS=NO)

Notes as semantic source + typed seed edges

source

Note→note / note→task relations feed the cloud graph as real edges

Recorded evidence

CloudsWorkspaceModel.swift:86–96 verified 2026-10-18

Cognitive HUD context (NEW since bio)

source

Clouds publishes a situational-awareness summary (clouds/nodes/edges/status) to the app-wide double-shift Cognitive HUD spine

Recorded evidence

CloudsWorkspaceModel.swift:256–275, observed 2026-10-18; not present in the 17 Oct biography

NodeMap infinite zoom (L0–L4 + submaps)

source

Semantic LOD with hysteresis; deep-zoom past L4 opens the focused node's submap recursively

Recorded evidence

NodeMapZoomLevels.swift:20–43 verified 2026-10-17

NMRF reference frame + analogy gestures

component-tested

Right-click: nearest concepts with consensus vote; set A→B, complete A:B::C:? and connect or create the answer node

Recorded evidence

Core self-test 15/15, Swift-6 clean (2026-08-03); on-canvas gestures stub-typechecked only

Per-node vocal expression mic

source

Hold mic on a node, speak, release; node absorbs regulation signals + assistance tuning, popover shows support data

Recorded evidence

NodeVocalExpressionController.swift / NodeMicButton.swift on disk 2026-10-17; project note ~2026-06; runtime unknown

Map wizards + task-scoped map

source

Generate maps from code structure or research material; embedded per-task map

Recorded evidence

CodeNodeMapWizard / ResearchNodeMapAutoWizard / TaskScopedNodeMapView on disk 2026-10-17

NodeMap futurisation pass (branch only)

source

Multimedia edges, instanced renderer, SemanticCompute GPU layouts (spring/Taubin/spectral/PCA), kNN inferred edges, media playback

Recorded evidence

Branch claude/nodemap-multimedia-edges-shading-bd16d6; adapter self-tests 12/12 on hardware (2026-09-01); verified absent from main 2026-10-17; never app-built

Staged reasoning pipeline

component-tested

Triage→plan→gather→draft→critique→revise→finalize with line formats and explicit review states; flag assistantapp.ai.stagedReasoning

Recorded evidence

Harness 126/126 + TSan; real-model n=34 single run 2026-09-20 (staged 32 / direct 28 / loop 24; 5.4 s vs 11.0 s median); installed to main 2026-09-20, unwired

Guide assistant + intent classifier

source

Offline-first contextual help with PII redaction, schema validation, rate limiting; Create ML help-intent classifier

Recorded evidence

AppAssistantHelper.swift header verified 2026-10-17; classifier trained 2026-03-05 (Metadata.json); runtime unknown

Offline topic/transformer explanation sidecars

source

Token attribution, logit lens, attention summaries for topic assignments via local Python sidecars pinned offline

Recorded evidence

Tools/SemanticTopicModeling/{bertopic,ecco,transformer_lens}_sidecar.py verified on disk 2026-10-18; TransformerTracing.swift

Review updates & corrections
  • NEW: Cognitive HUD spine integration in CloudsWorkspaceModel (hudSignature/hudContext/handleHUDAction, lines 256–275) — Clouds now reports situational context to the app-wide double-shift HUD; observed 2026-10-18
  • NEW: Research and Exploration mode filters now admit note nodes (novelty-gated for Research) — CloudsWorkspaceModel.swift:222–224, 246
  • CORRECTION: the biography named Tools/HybridReasonHarness/Evaluation/ as the one safe on-disk demo artefact; that path does not exist in the main checkout (worktree removed). Evaluation numbers remain documented only in dated private notes
  • CloudsIntentEngine.swift was touched on disk since 17 Oct but its public structure is unchanged (same types and functions); no material change identified
  • No new media appeared: repo Docs/ re-checked 2026-10-18 (same three out-of-scope images)

Workflow walkthrough

A guide to the native workflow. These steps describe the app; only the recordings and images embedded on this page are available as website demonstrations.

  1. Press ⌘5 — the Clouds pane opens in the middle pane; choose the Semantic surface
  2. The map fits to your topic clouds; read the weather: a storm glow marks a cluster where contradiction and urgency are high
  3. Click the top chip in the 'Now' strip — the map centres and selects the cloud it predicts you need; hover shows why ('urgency rising', 'matches your routine')
  4. A yellow chip suggests 'Switch to Review'; one click adopts the mode and the map refilters to tensions and open questions
  5. Double-tap the storm cloud — its nodes fan out with typed edges; the inspector shows the topic's evidence spans, confidence and provenance
  6. Zoom far out to return to the all-clouds overview; labels thin out by predicted relevance while every cloud stays visible as a GPU field
Planned captures · not yet available
  • Clouds Semantic surface at overview zoom: weather-tinted cloud fields, Now forecast strip, mode picker
  • 20-second zoom dive: all clouds → one storm cloud's nodes → a node's relations, ending on the inspector's evidence panel
  • NodeMap analogy advisory: A:B::C:? completed geometrically, with the one-click 'connect edge' action
  • Staged-reasoning evaluation table (n=34, one run, 2026-09-20): staged 32 / direct 28 / planner loop 24; median latency 5.4 s vs 11.0 s

Development biography · earlier archive

Development profile compiled from project source and retained records in October 2026. Dates and test results describe those records; they do not imply a new build, live acceptance test or public release.

Clouds (Semantic Weather Workspace)

Biography

Clouds began as an answer to a familiar failure of productivity tools: lists and folders show what you wrote down, not what is going on. Across the AssistantApp — tasks, notes, memory chunks, imported ChatGPT history, node maps, a semantic archive, neuro-theory and academic-research handoffs — the same themes recur, collide and drift, but no single surface showed that weather. Clouds is that surface: a semantic weather map where topics are clouds, and the state of your thinking is literally the climate.

The pipeline is entirely on-device. CloudsSourceAdapters converts eight local sources into CloudsNodes . CloudsGraphBuilder then fits the app's native SemanticTopicModeler over every node, groups nodes into clouds per topic, aggregates nine-component signals (relevance, energy, urgency, confidence, novelty, unresolved, contradiction, ambiguity, drift), and assigns a weather state via CloudsWeatherEngine: contradiction or urgency becomes a storm, ambiguity becomes fog, unresolved pressure, novelty lightning, low energy cold. Since August 2026 notes are a first-class source, and note→note / note→task relations from NoteAnalysis are passed through as typed seed edges (CloudsGraphBuilder.build(nodes:seedEdges:)), so the map draws real relationships, not decoration.

Rendering is the distinctive part. The map is an MTKView underlay : instanced cloud-field quads shaded with a procedural fbm-noise "puff" fragment shader, drifting and pulsing with each cloud's energy and urgency; real cross-cloud edges as feathered instanced quads; an animated nebula backdrop; and an MPSImageGaussianBlur bloom pass composited over the drawable. The world-to-screen transform runs in the vertex shader from shared GraphTransform uniforms, so the GPU layer tracks SwiftUI pan/zoom exactly — interactive labels stay in SwiftUI above it. Heavy work never touches the UI thread: graph building runs detached, filtering is debounced into stored published arrays, and interactive labels are budgeted per zoom band (10/16/24/48 at L0–L3+), ranked by predicted relevance, while unlabelled clouds stay visible as pure GPU fields.

The second distinctive idea is intent preemption, fully offline. CloudsIntentEngine blends deterministic semantic signals (urgency, unresolved threads, contradiction, topic emergence, recency, mode fit, query match) with a learned habit model — the app's own CMAC tile-coded function approximator, trained contrastively on the user's actual cloud selections (chosen cloud → 1.0, passed-over clouds → 0.0) over hour-of-day, weekday and mode features, persisted with the last 400 selection events to Application Support/AssistantApp/CloudsIntentMemory.json. The result is a ranked "Now" strip of forecast chips with human-readable reasons ("urgency rising", "you tend to open this around now"), per-cloud relevance that dims less-relevant fields, and a mode-preemption chip ("Switch to Review — tension is building across your top clouds").

A concrete journey: press ⌘5 (or the sixth axis-toolbar icon) and the Clouds pane opens in the middle pane — since August 2026 it is a workspace destination, not a popup, so zoom and focus persist while you work. Pick the Semantic surface. The map fits to your clouds; a storm cloud glows over "release planning" because two notes contradict a task. The Now strip's top chip says "matches your routine"; clicking it centres and selects the cloud. You double-tap to focus: the cloud's nodes fan out with typed edges to the centre. The inspector shows the topic's evidence spans — why the modeler grouped these items — plus signal bars and provenance. A yellow chip suggests switching from Research to Review; one click adopts it, and the map refilters to contradictions and open questions. Three surfaces share the same infinite-zoom principle: Notes (clusters by meta-tag), Tasks (by workflow stage), and Semantic (topic clouds over everything).

Clouds is honest software: topic quality, assignment confidence and needs-review ratios are displayed, not hidden; every cloud can explain itself.

Features (all authored; what a visitor sees)

  • GPU weather map: drifting procedural cloud fields, bloom, animated backdrop (Metal/MPS, 30 fps)
  • Nine weather states driven by nine-component semantic signals
  • Six working modes (Research / Build / Review / Recovery / Planning / Explore) that refilter the map
  • L0–L4 zoom bands with hysteresis; double-tap deep-zoom: all clouds → one cloud's nodes → one node's relations
  • "Now" intent forecast strip + suggested-mode preemption chip (offline, learns your habits)
  • Relevance-based label decluttering; dimming of low-relevance clouds
  • Sidebar search (debounced), source filters, cloud list with weather badges; inspector with topic evidence ("Why this topic"), signal bars, tags, links, provenance
  • Three-surface pane (Notes / Tasks / Semantic) at ⌘5; separate full window for App Intents/deep links
  • Glass visual system : sky gradient atmosphere, weather-glass strokes, spring selection motion

Technical foundations

SwiftUI + Combine (debounced filtering, stored published arrays); Metal + MetalKit + MetalPerformanceShaders (instanced quads, embedded MSL validated standalone with xcrun metal); SemanticTopicModeler (native, no network); CMACWeightStore (tile-coded sparse approximator with eligibility traces — shared with the Axis cognitive layer); GraphTransform shared with NodeMap. No external dependencies; no network at any point.

Maturity (dated evidence; aspirations are in §8)

  • Source implemented: 8 files in AssistantApp/Sources/Clouds/ + Sources/Shell/Axis/CloudsPane.swift (verified on disk 2026-10-17)
  • Built: full xcodebuild -scheme AssistantApp -destination platform=macOS completed exit 0 on 2026-06-12 (session evidence; shader also compiled standalone). Post-August additions: project note (2026-08-17) records the main-checkout build verified with SWIFT_EMIT_LOC_STRINGS=NO; I did not rebuild today (read-only).
  • Tests: none exist for Clouds. Unknown.
  • Runtime verified: no recorded screenshot or live-run evidence on disk. Unknown / not verified.
  • Released: no. Version-control caveat: per the 2026-08 project note, Sources/Clouds/ is untracked in git in the main checkout only — it exists on disk but on no branch (unconfirmed today; I could not run git read-only here).

Roadmap + limitations

Roadmap (documented in session notes, not speculation): commit the untracked Clouds sources; capture real screenshots; extend seed-edge sources beyond notes; broaden CMAC features. Limitations: macOS-only; intent engine starts heuristic-only and needs selection history to become habit-aware; topic modeling is lexical (no neural embeddings in the cluster step); no tests; GPU instance structs are layout-fragile against the embedded MSL (documented); map label culling caps interactive labels, so very large corpora rely on zoom to reach minor clouds.

FAQs

  • Does anything leave the machine? No. Topic modeling, rendering and the intent engine are entirely on-device; intent memory is a local JSON file.
  • What makes a cloud "stormy"? Aggregated node signals: contradiction > 0.42 or urgency > 0.78 → storm; drift, ambiguity, unresolved, energy and novelty map to drift/fog/pressure/heat/lightning .
  • Is the forecast an LLM? No — a deterministic heuristic blend plus a CMAC habit model trained on your own clicks; its influence grows only as its prediction confidence rises.
  • Why Metal rather than SwiftUI Canvas? The original Canvas implementation redrew every field per frame on the CPU; the Metal field renders hundreds of animated cloud instances with bloom at 30 fps and keeps the main thread for interaction.
  • Can I trust the clusters? The inspector shows per-topic confidence, needs-review ratio and the evidence spans behind each assignment — the UI is designed to expose model uncertainty.

NodeMap (Infinite Semantic Graph Navigation)

Biography

NodeMap answers a question mind-mapping tools don't: what should a knowledge graph do beyond sit there? The project's thesis, developed across 2026, is that a map earns its keep in three stages — first by surviving scale (semantic zoom), then by sensing its owner (classification and voice), and finally by becoming machinery for thought itself (a geometric reference frame where relations are vectors).

Navigation first. The live canvas renders node cards over a CALayer grid backdrop with axis lanes and a focus vignette. Zoom is semantic, not just optical: five bands L0–L4 with hysteresis thresholds . At overview bands, cluster cards summarise regions ; as you push past L4's deep-zoom trigger, the map opens the focused node's submap — the "infinite zoom" that Clouds later borrowed. A context rail carries the selected node's editor and related content; a semantic trail records where your attention has travelled; deep links let other app surfaces land on a specific node. Two wizards generate whole maps: CodeNodeMapWizard from code structure and ResearchNodeMapAutoWizard from research material, plus a task-scoped variant embedded in task detail.

Sensing second. Every node is semantically classified — classifyAllNodes inherits a parent task's first-pass axes (intent/temporal/domain/priority/workflow) so no node is ever unclassified, and classification runs on creation. The vocal layer is NodeMap's most personal feature: hold the mic button beside a node (200 ms arm delay so ordinary clicks don't record), speak while a live waveform meter runs, release, and a sweeping halo plays while the vocal-expression pipeline analyses the recording; the node then carries derived regulation signals, assistance-tuning values and support suggestions, inspectable in a popover .

Thinking third — the distinctive idea. ReferenceFrame/NodeMapReferenceFrame.swift (built 2026-08-03, pure Foundation, prefix NMRF) reframes the map per Hawkins' Thousand Brains theory: a graph stores topology, but a cortical column stores a reference frame — a metric space where concepts have locations. Each node's sentence embedding (via the app's VXREmbeddingTopicClassifier, Apple NaturalLanguage, offline) becomes its coordinate, and the frame then supports: nearest (perception — predict what's at a location), displacement (an edge's meaning as a vector), path-integration move (thinking as movement), between (interpolate two ideas), vote (deterministic dimension-subset "columns" each predict, then agree — the Thousand-Brains consensus with an explicit agreement/uncertainty score), trajectory (a default-mode walk through idea-space), and the flagship, analogy(a:b::c:?) = c + (b − a). On-canvas gestures wire this in: right-click a node for "Reference-frame neighbours"; set A, set B, then "Complete A:B::this:?" on a third node — the discovered answer arrives as an advisory with a one-click "connect the edge" action, and if the answer's confidence is below 0.72 the frame concludes the concept doesn't exist yet and offers to create the missing node. The map stops being a diagram and becomes externalised working memory that can propose its own edges.

A concrete journey: open a project map, scroll out to L0 — clusters only. Dive toward "gait analysis"; cluster cards give way to full node cards at L3. Right-click the "cadence" node → Reference-frame neighbours → an advisory lists the five geometrically nearest concepts with a consensus score. Set "monarchy"→"democracy" as A→B, right-click "capitalism", complete the analogy — the frame's demo derivation of "socialism" is the documented toy example. Hold the mic on a blocked node and talk through the problem; the node absorbs your vocal signals and adjusts its assistance tuning. Double-click into its submap and keep going — the map never ends.

Features

In the main checkout (visible today): infinite-zoom canvas with L0–L4 semantic LOD + hysteresis; deep-zoom submaps; cluster cards; typed edges with trails overlay; context rail editor; semantic trail; node search/connectivity; deep linking; code + research map wizards; task-scoped map; universal node classification; per-node vocal-expression mic with waveform/halo/popover; NMRF reference frame with neighbours/analogy/gap-creation gestures (advisory-driven). Worktree-only (branch claude/nodemap-multimedia-edges-shading-bd16d6, 2026-08-03 → 2026-09-01, NOT in main): multimedia edges (strength/media/colour + edge editor), semantic shading palette, viewport governor ("never lose the map"), viewport culling, 3D media edges with GraphCamera parallax, Catmull-Rom trail ribbons, single-Canvas instanced renderer with unified interaction overlay + spatial-grid hit-testing (P0–P3), SemanticCompute-accelerated layouts (spring-electrical incl. GPU, Taubin tidy, spectral/Fiedler, PCA depth), embedding-kNN inferred edges, k-means semantic clustering, inline node media playback with VoiceMemo handoff.

Technical foundations

SwiftUI + Canvas + CALayer backdrop; GraphTransform world↔screen maths (shared with Clouds); VXREmbeddingTopicClassifier (NaturalLanguage sentence embeddings, on-device) for reference-frame coordinates; ActionableAdvisory platform for NMRF results; SemanticCompute local SPM package (the project archive) — already a linked product of the app target; the worktree pass exploits 9 of its families (CSR adjacency, spring-electrical with Metal GPU parity, Taubin, spatial grid, SpMV, pairwise distance, matmul, transpose, labelwise reduction). All offline.

Maturity (dated; unknowns marked)

  • Main-checkout features: source present 2026-10-17. NMRF core self-test 15/15 green + Swift 6 strict-concurrency clean (2026-08-03); NMRF bridge/gestures stub-typechecked against real signatures, "Xcode-gated" — no recorded full app build exercising them. Vocal-expression mic: recorded as wired ~2026-06; no runtime capture on disk. Runtime verification for all of it: unknown.
  • Worktree pass: parse-verified + adversarially reviewed (0 compile findings, 6→7 logic bugs found and fixed, per 2026-09-01 note); SemanticCompute adapter self-tests 12/12 on real hardware incl. GPU==CPU parity; full app build was blocked in that worktree by the absent local firebase package — never app-built, and absent from main.
  • Tests: NMRF 15/15 self-test; SemanticCompute package 570/570 CLI (package-side). No NodeMap UI tests.
  • Released: no.

Roadmap + limitations

Roadmap (documented): land the worktree futurisation pass in main once the firebase-package build blocker is resolved; real embedding vectors in EmbeddingRegistry to unlock kNN/PCA/cluster features; instanced-renderer completion (edge media handles, inertia); AUv3-hosting node objects; L0–L2 cluster hulls from cluster: tags; off-screen VoiceOver reachability. Limitations: macOS-only; edges in the main checkout are rendered on a non-hit-testable Canvas (edge selection/editing is worktree-only); NMRF semantic quality is bounded by sentence-embedding quality and it organises/verifies rather than invents (documented honestly in the project notes); two dead monolith files on disk are intentional reference copies — not dead weight to "clean up".

FAQs

  • Is the "infinite zoom" real? Functionally yes: zoom bands change representation, and pushing past the deepest band opens the focused node's submap, recursively.
  • What does the reference frame actually add over the graph? Operators: nearest-neighbour perception, relation vectors, interpolation, consensus voting with an uncertainty score, and completed analogies that can materialise as new edges or new placeholder concepts.
  • Does voice data leave the machine? No — the vocal-expression pipeline is on-device; results are stored on the node and as a local memory chunk.
  • Why are some advertised features missing when I build main? The multimedia-edge/3D/instanced-render/SemanticCompute pass lives on a feature branch worktree and has never been merged; the site should not present those as shipped.
  • Can it generate new ideas? The frame composes and verifies; generation still needs an LLM — that boundary is deliberate and documented.

Offline Reasoning Surfaces (final part; scope complete)

Biography

The stack grew from a constraint and a conviction. The constraint: the development host's outbound network is SSL-blocked by design, so anything that mattered had to work offline. The conviction, visible across the 2026 notes: an assistant you trust with tasks, notes and voice must be able to think locally, show its uncertainty, and never silently substitute imagination for evidence.

The foundation layer is provider plumbing. AppleIntelligenceOpenAICompat wraps Apple's on-device Foundation Models behind the same chat-completion contract as remote providers, so the central AIService router can treat "local" as just another source with capability flags (vision, structured output, must-stay-on-device). Beside it sit a deliberately boring set of safety pieces: FoundationModelRefusal (handling the model's documented quirk of refusing certain constrained-list guides), PlannerReplyRecovery (a tolerant reader for planner turns the strict decoder rejects — written after measurement showed the main agent loop failed to decode its first planner turn in 31 of 34 held-out questions and leaked raw JSON to the user), and OfflineOrchestrationSeam for bridging orchestration outcomes.

The perception layer is small and fast. HelpIntentClassifier is a Create ML text classifier trained by the author on 2026-03-05 ("Model for Palm Assistant"). The VXR library adds sentence-embedding topic classification (Apple NaturalLanguage, mean-centred prototypes, cosine matching) plus a trained softmax topic head. CMACWeightStore — a tile-coded sparse function approximator with eligibility traces, momentum and prompt-guidance output — learns continuous mappings from interaction, and is consumed by both the Axis cognitive layer and the Clouds intent engine. OfflineCodeReasoningLogicModel and TaskOfflineReasoningSupport provide deterministic intent and cognitive-load/assistance-mode hints for code and task contexts with no model call at all.

The flagship is the staged reasoning pipeline (2026-09-20): triage (no model call) → optional plan → relevance-gated context gathering → draft → critique → optional revise → finalize. Its design rebellion is format: line-oriented protocols (STEPS: / NEED: / VERDICT: / ISSUES:), never JSON, because strict JSON is exactly where small on-device models fail. Critique produces an explicit ReasoningReview state — passed, issues open, revised-but-unverified, unchecked — rather than a confident answer. And unusually for a personal project, it was measured before being believed: on the real on-device model over a held-out n=34 set (one run), staged scored 32 versus direct prompting's 28 and the existing planner loop's 24, at roughly half the loop's median latency (5.4 s vs 11.0 s); the notes record, with equal honesty, that score gaps of 2–4 are noise, that arithmetic self-consistency voting bought nothing, and that trick questions failed in every arm — a model-capacity limit assigned to a later phase.

The surfaces a user actually meets: the in-app Guide assistant (AppAssistantHelper — offline-first contextual help with PII redaction, strict schema validation, caching, rate limiting, and the rule that it never drives the UI directly); task detail's offline reasoning hints; the Clouds "Now" forecast and mode preemption (profile 1); the NodeMap reference-frame advisories (profile 2); and the transformer-trace inspector, which can explain a topic assignment down to token attributions, logit-lens layer rankings and attention summaries via local Python sidecars pinned to TRANSFORMERS_OFFLINE=1.

A concrete journey: you ask the assistant a planning question with Wi-Fi off. Triage decides it needs a plan; the gatherer pulls only memory chunks that pass a relevance gate; the draft is critiqued, one issue is fixed in revision, and the answer arrives with its review state visible. You open a task — the offline support layer has already estimated cognitive load and suggested an assistance mode. In Clouds, the forecast strip has pre-ranked tonight's likely work. Nothing left the machine; every layer that guessed said so.

A six-phase roadmap governs what comes next: provider symmetry, a context/memory substrate, a @Generable-based planner/verifier now that the macOS 26 floor guarantees structured generation, a downloaded System-2 reasoner (weights destined for an external drive), and evaluation/observability. The stack's character is its honesty: three divergent copies of the reasoning contracts are documented as a hazard, a degenerate standalone extraction is explicitly flagged "do NOT mistake it for the real loop", and a false "12/12 compiled" claim is recorded and corrected in the notes themselves.

Features / surfaces

  • Guide assistant: offline-first contextual help, PII redaction, schema validation, caching, rate limiting, deterministic fallbacks
  • Staged reasoning pipeline with explicit critique states and a feature flag (assistantapp.ai.stagedReasoning, default OFF)
  • Planner-reply recovery for strict-decoder failures (measured: 29/34 raw-JSON leaks → 0)
  • Create ML help-intent classifier (author-trained, 2026-03-05)
  • CMAC prompt guidance: confidence, novelty, suggested mode, ask-for-clarification flag
  • Task cognitive-load / assistance-mode hints; offline code-assist intent inference
  • Clouds intent forecast + NodeMap reference-frame advisories (cross-references, profiles 1–2)
  • Transformer-trace explanations (token attribution, logit lens, attention/MLP summaries) via offline sidecars

Technical foundations

FoundationModels (Apple Intelligence, on-device) behind an OpenAI-compatible seam; Create ML / Core ML; NaturalLanguage sentence embeddings; CMAC (tile coding, TD(λ)); line-format protocols instead of JSON for small-model robustness; local Python sidecars (transformer-lens/BERTopic) forced offline by environment pins. Local packages: Tools/AssistantOfflineAIModelKit (standalone SPM extraction — builds/tests headlessly with a deterministic heuristic engine, but its orchestration loop is a degenerate stand-in, 0 app imports) and Tools/HybridReasonHarness (evaluation harness, 126/126 + TSan).

Maturity (dated; unknowns marked)

  • Source implemented: Offline/ files verified on disk 2026-10-17 (AppAssistantHelper, OfflineCodeReasoningLogicModel, TaskOfflineReasoningSupport, StagedReasoningBridge, PlannerReplyRecovery, OfflineOrchestrationSeam, FoundationModelRefusal, AppleIntelligenceOpenAICompat, CognitiveReasoningFrame, CognitiveObservationRequests, GuidedIntakeModels, TODO/{OrchestrationLoop, LocalProvider, ContextRetrieval}, HelpIntentClassifier.mlpackage)
  • Tested/measured: staged pipeline harness 126/126 + TSan; real-model held-out n=34 eval, one run (2026-09-20 — treat small gaps as noise, per the notes). NMRF/CMAC consumers have their own self-tests (profiles 1–2).
  • Installed but NOT wired (critical honesty): the 8 staged-reasoning files were installed into main by a user-run SHA-pinned script on 2026-09-20, remain untracked, and have no caller — the three AIService edits were never applied; the feature flag is OFF. The planner-recovery patch was applied to main's OrchestrationLoop on 2026-09-20 22:12 . Phase 1 routing work is uncommitted on the working tree; a prepared pathspec commit script went stale and needs regenerating. Full Xcode build of the wired pipeline: never run. Current wiring state as of today: unknown beyond those notes.
  • Released: no.

Roadmap + limitations

Roadmap (documented, 2026-09-09/20): wire staged reasoning into AIService behind its flag; commit the planner patch pair; Phase 2 context substrate; Phase 3 @Generable planner/verifier + native Tool catalog; Phase 4 downloaded System-2 reasoner (Qwen3-Thinking / gpt-oss-20b class, weights to the external drive); Phase 5 eval/observability. Limitations: single-run evaluation (n=34) — directional, not statistical; trick questions fail in every arm; three divergent contract copies until consolidation; much of the stack is uncommitted/untracked; small on-device models remain the capability ceiling until Phase 4.

FAQs

  • Is this a wrapper around a cloud LLM? No — Apple's on-device Foundation Models plus author-trained classifiers and deterministic heuristics; the dev host's network is SSL-blocked by design, so offline operation is continuously exercised.
  • What's actually proven? The pipeline beat direct prompting and the agent loop on one held-out run and halved median latency; the harness is green at 126/126. Wiring into the main chat path is the documented next step, not a shipped fact.
  • Why line formats instead of JSON? Measured: the strict-JSON planner failed to decode in 31/34 cases on the real model; line protocols plus a tolerant recovery reader eliminated the user-visible JSON leaks.
  • What happens when the model is unavailable? Every surface has a deterministic fallback — heuristic intent hints, cached guide answers, neutral forecasts — and says so rather than fabricating.
  • Where do big model weights go? the project archive (external drive), a 2026-09-09 user decision; nothing heavyweight lives in the repo.
  • Part 1 — Clouds semantic workspace: m-mvcztzni-c1b804
  • Part 2 — NodeMap / graph navigation: m-mvczvkmf-cc3663
  • Part 3 — Offline reasoning surfaces: m-mvczx5k2-a2fd18

Full-scope synthesis (for the site's section lead): The three surfaces are one coherent story — an assistant that thinks locally and shows its work. A native topic modeler and signal system turn tasks, notes, memory and research into a GPU-rendered semantic weather map (Clouds); an infinite-zoom graph gives ideas places and turns topology into geometry for analogical reasoning (NodeMap); and an offline reasoning stack — Foundation Models behind deterministic fallbacks, an author-trained classifier, CMAC habit learning, a measured staged-critique pipeline — powers the forecasts, advisories and help those surfaces show. Shared DNA: GraphTransform world↔screen maths, L0–L4 zoom-band hysteresis, VXR sentence embeddings, CMAC, and a house style of exposing model uncertainty in the UI.

Consolidated uncertainty register (publish nothing that contradicts these):

  • Runtime verification: none, for all three surfaces. No recorded live-run evidence or screenshots exist anywhere on disk; native window capture is documented to return blanks. All maturity claims are source/build/test-level, not runtime.
  • Media: zero publishable assets. None invented. Only safe demo artefact: the Tools/HybridReasonHarness/Evaluation/ results table (synthetic Q&A). Real screenshots require an app run the user must do or authorise.
  • Clouds git status: untracked in the main checkout per the 2026-08 project note — on disk but on no branch. Unconfirmed on 2026-10-17 (I could not run git in this session).
  • Clouds build evidence is dated: full xcodebuild exit 0 on 2026-06-12; the August additions are covered by a 2026-08-17 build note; not rebuilt today.
  • NodeMap's futurisation pass (multimedia edges, instanced renderer, SemanticCompute layouts) exists only on branch claude/nodemap-multimedia-edges-shading-bd16d6; verified absent from main today; never app-built (firebase package blocker). Do not present as shipped.
  • NMRF reference-frame gestures: core self-test 15/15 (2026-08-03), but the on-canvas wiring is stub-typechecked only.
  • Staged reasoning pipeline: installed into main 2026-09-20 but unwired (flag OFF, AIService edits never applied; planner patch applied-but-uncommitted at 2026-09-20 22:12). Wiring state after that date: unknown.
  • Evaluation numbers (staged 32 / direct 28 / loop 24 on n=34; 5.4 s vs 11.0 s median) are a single run — directional only; gaps of 2–4 are noise per the project's own notes.
  • Public links: none exist or were verified; none invented. Website URL is the owner-private one from your brief.

Explore the connections.

Browse all project stories ↗