Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,18 @@ Four reflexes before you touch anything:

### The two layers

| Layer | What it is |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Sources/BlurtEngine/` | Swift package (`swift-tools-version:6.2`, `platforms: [.macOS(.v15), .iOS(.v18)]`) owning the pipeline. Pure logic behind protocol seams, no AppKit-shell deps, **no external SPM deps** — Foundation/Security/AVFoundation/CoreAudio plus toolchain modules like Synchronization, with AppKit types only at the seams. The macOS-only files (Accessibility reads, clipboard paste, CoreAudio route reads, TCC and code-signing helpers) are fenced behind `#if os(macOS)`; `check.yml`'s `ios-build` job builds the library for the iOS simulator so a symbol that escapes its fence fails the PR. |
| `App/Blurt/` | AppKit/SwiftUI shell (Xcode project generated by XcodeGen) that wires the engine to an overlay window, the main window, a Settings scene, a menu bar item, and the trigger key. Its only package is the local `BlurtEngine` (declared in `App/Blurt/project.yml`). |
| `App/BlurtiOS/` | The iPhone app (SwiftUI) and the keyboard it embeds, generated by XcodeGen from its own `project.yml` and built by `check.yml`'s `ios-build` job. The app owns the engine's pipeline over an always-on capture; the keyboard is a remote control that inserts the words. Their logic is `BlurtiOSCore`, a framework apart from the UI the way the engine is apart from the Mac shell. `App/BlurtiOS/README.md` is its guide. |
| Layer | What it is |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Sources/BlurtEngine/` | Swift package (`swift-tools-version:6.2`, `platforms: [.macOS(.v15), .iOS(.v18)]`) owning the pipeline. Pure logic behind protocol seams, no AppKit-shell deps, **no external SPM deps** — Foundation/Security/AVFoundation/CoreAudio plus toolchain modules like Synchronization and Observation (`APIKeyModel`, the one `@Observable` type both shells share), with AppKit types only at the seams. The macOS-only files (Accessibility reads, clipboard paste, CoreAudio route reads, TCC and code-signing helpers) are fenced behind `#if os(macOS)`; `check.yml`'s `ios-build` job builds the library for the iOS simulator so a symbol that escapes its fence fails the PR. |
| `App/Blurt/` | AppKit/SwiftUI shell (Xcode project generated by XcodeGen) that wires the engine to an overlay window, the main window, a Settings scene, a menu bar item, and the trigger key. Its only package is the local `BlurtEngine` (declared in `App/Blurt/project.yml`). |
| `App/BlurtiOS/` | The iPhone app (SwiftUI) and the keyboard it embeds, generated by XcodeGen from its own `project.yml` and built by `check.yml`'s `ios-build` job. The app owns the engine's pipeline over an always-on capture; the keyboard is a remote control that inserts the words. Their logic is `BlurtiOSCore`, a framework apart from the UI the way the engine is apart from the Mac shell. `App/BlurtiOS/README.md` is its guide. |

The package's second library, **`AssemblyAI`** (`Sources/AssemblyAI/`), is the AssemblyAI API client the
engine is built on: the `HTTPTransport` seam and its streamed upload body, the dictation wire types and
multipart framing, `APIKeyValidator`, and dictation vocabulary — `KeyTerms` (parsing a list and fitting
it to `keyterms_prompt`'s caps) and `TextShortcut` with its on-device `TextShortcutExpander`. It holds no Blurt policy (no prompts, styles, key storage or
`User-Agent`; `APIKeyValidator.blurt()` is where Blurt's agent joins it), so it can become a standalone
Swift SDK. It is part of the same local package, so the dependency-free rule is unchanged.
it to `keyterms_prompt`'s caps). It holds no Blurt policy (no prompts, styles, key storage,
`User-Agent` or on-device text processing; `APIKeyValidator.blurt()` is where Blurt's agent joins it,
and text shortcuts live in the engine), so it can become a standalone Swift SDK. It is part of the same local package, so the dependency-free rule is unchanged.

The engine's dependency-free rule is a **rule**. The app carries one remote package — Sparkle, the
in-place updater (see [Updates](#updates)) — allowlisted by URL in `check.sh`'s dependency guard, and
Expand All @@ -60,7 +60,7 @@ workflow and the _why_ behind the design; the engine's README covers the _what_
```text
Sources/AssemblyAI/ the AssemblyAI API client library (README.md there): HTTPTransport +
ChunkedRequestBody, Dictation/ (wire types, multipart framing,
KeyTerms, TextShortcuts), APIKeyValidator. No Blurt policy;
KeyTerms), APIKeyValidator. No Blurt policy;
BlurtEngine depends on it
Sources/BlurtEngine/ the engine (dependency-free Swift package)
README.md the engine's developer guide (quick start, seams, error table)
Expand All @@ -86,7 +86,8 @@ Sources/BlurtEngine/ the engine (dependency-free Swift package)
Permissions/ PermissionStatus + SetupReadiness (every platform);
PermissionsChecker (mic + Accessibility), PermissionsReset,
SigningIdentity (macOS only)
Pipeline/ DictationSession (actor) + phases, UI projections, geometry, log
Pipeline/ DictationSession (actor) + phases, UI projections, geometry, log,
TextShortcuts (the on-device phrase expansion)
STT/ AssemblyAITranscriber, Conversation/TranscriptionContext,
CleanupInstruction, SyncSTTLimits
TTS/ experimental read-aloud: SelectionSpeaker, AssemblyAISpeechSynthesizer
Expand Down Expand Up @@ -651,7 +652,7 @@ verbatim transcript, all inside the same request, and it rides **every** request

**The switch is applied to the response, not the request** (since 2026-09-11). The response
carries both `text` (the verbatim transcript, "never altered by the LLM") and `llm_response`
(the rewrite), so `AssemblyAITranscriber.transcript(from:)` simply picks: with **enhanced
(the rewrite), so `AssemblyAITranscriber.transcript(from:enhanced:)` simply picks: with **enhanced
transcripts** on (`EnhancedTranscriptsStore`, on by default, read per request via the
transcriber's injected `enhancedTranscripts` closure) it prefers the rewrite and falls back to
`text` when `llm_response` is null or blank — the rewrite is best-effort (5 s server-side
Expand Down
8 changes: 4 additions & 4 deletions App/Blurt/Blurt/AppCoordinator.swift
Original file line number Diff line number Diff line change
Expand Up @@ -291,9 +291,9 @@ final class AppCoordinator {
// normal flow. Asked through the router, because while a read-aloud owns the
// gate the reset isn't this dictation's to make (see `SelectionSpeechRouting`).
// It gets the phase itself because an ask about a selection ends in one,
// `.handedBack`, carrying the transcript the router answers.
if phase.isTerminal {
speechRouter?.sessionReachedTerminalPhase(phase)
}
// `.handedBack`, carrying the transcript the router answers — and every
// phase, not just the terminal ones, so it can tell an earlier dictation
// still finishing from the end of a request it is waiting on.
speechRouter?.sessionPhaseChanged(phase)
}
}
4 changes: 2 additions & 2 deletions App/Blurt/Blurt/Hotkey/SelectionSpeechRouter.swift
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ final class SelectionSpeechRouter {
perform(routing.keyLatched())
}

func sessionReachedTerminalPhase(_ phase: PipelinePhase) {
perform(routing.sessionReachedTerminalPhase(phase))
func sessionPhaseChanged(_ phase: PipelinePhase) {
perform(routing.sessionPhaseChanged(phase))
}

private func perform(_ actions: [SelectionSpeechRouting.Action]) {
Expand Down
32 changes: 15 additions & 17 deletions App/Blurt/Blurt/Overlay/OverlayView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ struct OverlayView: View {
// surface progressing rather than three unrelated states.
waitingContent("Transcribing")
.transition(.opacity)
case .error(let message):
case .error(let message, nil):
// "Try again" tells the user what to do; the full failure reason is too
// long for the pill, so expose it on hover. The VoiceOver announcement
// (OverlayWindowController) speaks the same message for non-sighted users.
Expand All @@ -136,27 +136,25 @@ struct OverlayView: View {
StatusLineText("Pasted")
.transition(.opacity)
.help(state.accessibilityLabel)
case .inputSilent:
// The silent-mic notice: the device name is the diagnosis, so it goes on
// the pill itself — the controller has widened the capsule to fit — in the
// same orange as the "Error" word.
case .error(_, let notice?):
// A failure that spells out its reason (the silent mic: the device name is
// the diagnosis), so it goes on the pill itself — the controller has
// widened the capsule to fit — in the same orange as the "Error" word.
//
// Two runs, so a device name too long for the capped pill is cut in its
// middle while "isn't sending any audio" — the part that says what's wrong —
// always shows in full, and neither shrinks.
if let notice = state.pillNotice {
HStack(spacing: 0) {
StatusLineText(
notice.subject + " ", color: BlurtBrand.errorOrange,
truncation: .middle, shrinksToFit: false)
StatusLineText(notice.predicate, color: BlurtBrand.errorOrange, shrinksToFit: false)
.fixedSize()
.layoutPriority(1)
}
.padding(.horizontal, Self.errorTextInset)
.transition(.opacity)
.help(state.accessibilityLabel)
HStack(spacing: 0) {
StatusLineText(
notice.subject + " ", color: BlurtBrand.errorOrange,
truncation: .middle, shrinksToFit: false)
StatusLineText(notice.predicate, color: BlurtBrand.errorOrange, shrinksToFit: false)
.fixedSize()
.layoutPriority(1)
}
.padding(.horizontal, Self.errorTextInset)
.transition(.opacity)
.help(state.accessibilityLabel)
case .noTarget:
// Quiet, informational notice: there was no text field to type into, so
// the transcript went to the clipboard. Styled exactly like "Pasted"
Expand Down
13 changes: 10 additions & 3 deletions App/Blurt/Blurt/Overlay/OverlayWindowController.swift
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ final class OverlayWindowController {
// The revert timer for a transient notice; its dwell comes from the engine
// (`OverlayUIState.noticeDwellSeconds`, unit-tested there).
private var errorRevertTask: Task<Void, Never>?
/// Bumped by every `setVisible`, so a fade-out's completion can tell it was superseded.
private var fadeGeneration = 0

// The pill fades in fast — the appear is tied to the user's keypress, so a snappy
// ramp reads as instant response — but fades out gently. Asymmetric on purpose.
Expand Down Expand Up @@ -254,9 +256,14 @@ final class OverlayWindowController {

/// Drives the pill on/off screen, fading unless Reduce Motion is on. Idempotent
/// and re-entrant: a key-down during the fade-out re-targets alpha back to 1,
/// and the in-flight fade's completion only orders the panel out if it actually
/// reached transparent — so a quick hide→show never strands a hidden panel.
/// and the in-flight fade's completion only orders the panel out if no later
/// call has superseded it — so a quick hide→show never strands a hidden panel.
/// A generation, not the alpha it ended at: `show` zeroes alpha itself to
/// resize under a transparent panel, and that must not read as the fade-out
/// having finished.
private func setVisible(_ visible: Bool) {
fadeGeneration &+= 1
let generation = fadeGeneration
let reduceMotion = NSWorkspace.shared.accessibilityDisplayShouldReduceMotion
if visible {
// Already up and fully opaque: nothing to ramp. Without this, each phase
Expand Down Expand Up @@ -292,7 +299,7 @@ final class OverlayWindowController {
completionHandler: { [weak self] in
// NSAnimationContext completion handlers run on the main thread.
MainActor.assumeIsolated {
guard let self, self.panel.alphaValue < 0.01 else { return }
guard let self, self.fadeGeneration == generation, self.panel.alphaValue < 0.01 else { return }
self.dismissPanel()
}
})
Expand Down
1 change: 0 additions & 1 deletion App/Blurt/Blurt/Wizard/TextShortcutsSection.swift
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import AssemblyAI
import BlurtEngine
import SwiftUI

Expand Down
7 changes: 7 additions & 0 deletions App/BlurtiOS/BlurtiOS/Sources/AudioSource.swift
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,13 @@ extension WindowedAudioSource: nonisolated AVCaptureAudioDataOutputSampleBufferD
func captureOutput(
_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection
) {
// The window keeps the mic open for minutes between utterances; resampling
// buffers nobody will read is wasted. The converter is dropped with them, so
// the next utterance doesn't start on the old audio's resampler tail.
guard feed.isCapturing else {
converter = nil
return
}
if converter == nil { converter = PCMConverter() }
guard let converter, let chunk = converter.convert(sampleBuffer) else { return }
feed.deliver(chunk)
Expand Down
Loading
Loading