Jot Vocabulary on Mac
What the Mac app now does with a user's vocabulary, why, and what we measured. Jot for iPhone shares the rules through the jot-shared package, but several pieces live in the app, and those are listed at the end as work for iOS.
Three rules
A correction applies only when the speech model itself chose the word, a real acoustic score supports it, or a measured spelling distance does. The old path passed invented values (confidence 0.85, margin 0) to the gate; those are gone.
Every applied correction is recorded, can be undone, and can be asked about. Spelling-only matches apply under the full guard set, never as an unseen patch.
No rule is learned when the heard word is an everyday word. Rules like “not → Jot” or “fine → Flink” were being learned this way; they are refused now, and removed once from existing stores.
What runs, per speech model
The vocabulary step runs right after decoding, before text cleanup (fillers, numbers) and before any AI rewrite. There is no time cutoff. On a 30-minute dictation it takes about 4.5 s in a debug build.
| Model | While transcribing | After decoding |
|---|---|---|
| Nemotron multilingual | Decode-time bias toward the user's terms (strength 3.5). The CTC keyword spotter listens to the same audio in parallel. | VocabularyPass: spelling matches first, then spotter hits placed on the word Nemotron wrote at that moment. |
| Nemotron English | CTC spotter in parallel. No bias: this model has no hook for it. | VocabularyPass, same as above. |
| Parakeet v2 / v3 | Nothing. | CTC spot + ctcTokenRescore, then VocabularyGate.apply with Parakeet's real token confidences. No spelling corrector on this path. |
| Japanese | Nothing. | Alias substitution, unchanged. |
Sources: DualPipelineTranscriber.swift (Nemotron), Transcriber.swift (Parakeet), VocabularyRescorerHolder.swift (gateDetections, finishRescore), VoiceInputPipeline.swift (streaming spotter), NemotronMultilingualStreamingTranscriber.swift (bias).
Three kinds of evidence, three rules
The model itself produced the term under bias. Nothing to patch afterwards: the word is already in the transcript. Mac, Nemotron multilingual only.
A CTC spotter detection with its real log-prob score. Placed on the transcript word the decoder wrote within 1.0 s of the detection, using the decoder's own token timings (WordTimeline). When word timings can't be aligned to the transcript, the detection is dropped; it is never placed by proportional position.
A spelling match from the model-free corrector (word n-grams up to 8, OSA Damerau-Levenshtein, per-language maximum distance, ambiguous ties dropped). Applies only through the gate's full guard set, and only for languages with a common-word list. Visible and undoable.
In VocabularyPass, spelling matches claim their words first. On the owner's 40 recordings that order got 35 of 48 mentions right with one wrong apply in 4 of 5 runs, against 34–35 with one or two wrong the other way round. The corrector names an exact word window; spotter time spans are coarse and its scores uniformly weak.
The model listens for the terms
In the FluidAudio fork, the Nemotron greedy decoder adds a bonus to the tokens of a user's term. A term may only start where the unbiased decoder would itself start a word. The decoder snapshots its state when bias first flips a choice; if the term doesn't complete, it rewinds to the snapshot, bans that start, and continues. This works across chunk edges and is resolved at finish(), so a term cut off at stop never leaves half a word. Bias turns off for the rest of an utterance after 64 rewinds. The vocabulary is set only between utterances, with canonical spellings (no aliases).
| Strength | Rare-word WER | Other words WER | Owner: names caught | Owner: wrong place | |
|---|---|---|---|---|---|
| off | 14.1% | 2.08% | 14 / 48 | 0 | |
| 2.5 | 8.4% | 2.02% | · | · | |
| 3.0 | 7.9% | 2.00% | · | · | |
| 3.5 | 7.3% | 1.98% | 31 / 48 | 1 | chosen |
| 4.0 | 6.8% | 1.98% | 35 / 48 | 3 | rejected |
| 4.5 | 6.3% | 1.99% | 37 / 48 | 5 | rejected |
WER columns: LibriSpeech test-clean, 2,620 clips, with the standard 100-word rare-word biasing lists (Le et al., 2021). With 1,000-word lists, 3.5 gives 8.3% and 2.03%. The owner's recordings were used only to reject settings, never to tune one: above 3.5 the bias starts writing terms over words that were actually said.
Safety checks that pass at 3.5: a vocabulary of made-up words leaves 40 recordings byte-identical to bias off; live and one-shot decoding agree with bias and blank rescue on or off; stopping inside a term at 262 different points never leaves a fragment.
What Jot learns, and what it refuses
- Guard.
CorrectionStorerefuses to record a confirmation or an “Always replace” grant when the heard phrase contains any common word in the active language (VocabularyGate.isCommonOriginal). Reverts and keeps are still recorded. - One-time cleanup. At launch the Mac calls
dropCommonOriginalRules()once (MacVocabCore.migrateCommonOriginalRulesIfNeeded, fromAppDelegate). It is marked done only when the call returns a count, notnil. - Asks follow the same predicate. A correction whose heard word is common is never offered as an ask.
How the Mac asks “Did you mean…?”
The paste is held and the pill asks one question at a time, then delivers once (AppDelegate.handleDeliveryBridge).
- What can be asked: an applied correction (“keep it?”), a blocked near-miss whose heard word is not common (“did you mean X?”), and the merge teach (“sri ram” → “Sriram”), which is offered once ever per phrase.
- What is skipped: pairs the user already rejected or told Jot to stop asking about, pairs granted “Always replace”, and any correction the AI transform reworded away.
- Order and limit: weakest evidence first (spelling before acoustic), then how often the pair was confirmed; at most 3 per recording. If a normal ask is present, the merge teach waits for another recording.
The ranking and limit mirror AskPolicy in jot-shared, but the Mac applies them in its own code (MacVocabGate in VocabCoreAdapters.swift), because the Mac pill works on its own correction type.
What the user sees on Mac
- The “Enable vocabulary boosting” switch is still there.
- Aliases stay in the vocabulary file (
Term: alias1, alias2); there is no inline editor. - Warnings under a term as it is typed: “Too short” for two characters or fewer, and the shared
VocabularyHygienewarning when the term's first word is an everyday word or it opens with one (“And…” in “Andalamma”). - No limit on list size; the help text advises under 100 terms. The help text still describes the old matching paths and needs updating.
Where the iOS summary is right, and where it describes Windows
1. Matching engine. Windows nudges Parakeet while transcribing and always runs the spelling corrector; iOS runs the corrector only when its acoustic model finds nothing; jot-shared adds one combined pass but can't nudge.
right- The nudge lives in the speech engine, not in jot-shared. On Mac it's in the FluidAudio fork and only for Nemotron multilingual. Jot for iPhone runs Parakeet, so it has no nudge today; Parakeet bias is planned as shared work for Mac and iPhone.
- The Mac's Nemotron paths run both kinds of evidence on every recording through
VocabularyPass, like Windows. iOS running the corrector only as a fallback is the part to change. - One gap on Mac: its Parakeet path doesn't use the spelling corrector yet.
2. What you see. Windows has no on/off switch and no “Sounds like” field, learns alternate spellings only from corrections, caps the list at 200, and warns on bad entries; iOS still has the switch and an editable “Sounds like”.
Windows, not Mac- The Mac also still has the switch, keeps aliases, and has no cap. Those removals are Windows decisions and weren't made on Mac.
- The bad-entry warnings are shared and on Mac. iOS should show them where terms are added.
3. What jot-shared doesn't cover. Vocabulary as the very last step with a 4-second cutoff; Auto-detect and unsupported languages always off; asking only about applied corrections (at most 3); the UI removals.
mostly Windows- Last step, 4 s cutoff: not on Mac. The Mac runs vocabulary right after decoding, before cleanup and AI transform, with no cutoff. The ask then checks the final text, so a correction the transform reworded away is dropped.
- Languages: on Mac the spelling corrector runs only for the 17 languages that ship a common-word list, and fails closed for Slovenian and Croatian. Other languages get acoustic matches only. The Mac has no auto-detect language setting.
- Asks: at most 3, yes. But the Mac also asks about blocked near-misses whose heard word isn't common, and offers the merge teach once, so asks aren't limited to applied corrections.
What iOS should do
- Run
VocabularyPasson every recording, with spelling matches and acoustic matches together, instead of the corrector only as a fallback.jot-shared: VocabularyPass.run - Give it word timings. Build them from Parakeet's token timings with
WordTimeline.wordTimes. Without them, acoustic detections are dropped by design.jot-shared: WordTimeline - Mark real spotter hits as acoustic, with their real scores. Never pass a synthesized confidence or margin.VocabularyGate.Detection evidence
- Run the one-time learned-rule cleanup at launch and mark it done only on a non-nil result. Use
isCommonOriginalfor ask admission too.CorrectionStore.dropCommonOriginalRules · VocabularyGate.isCommonOriginal - Show the hygiene warning wherever a term is added.VocabularyHygiene.warning
- Keep asks to 3, weakest evidence first, with the same suppression and one-time merge teach.AskPolicy
- Don't pull the FluidAudio fork. Its change is Nemotron-only and iPhone runs Parakeet.
- Don't edit the existing golden fixtures in jot-shared; Windows is locked to them. Add new fixtures only, and run
swift test(38 vocabulary + 8 text-pipeline tests pass today).
Known limits
- Two similar-sounding terms can pull a misheard word to the wrong one, as in “Vineet Flink”. Not fixed; any fix must be general, not a special case.
- A term caught by bias keeps the model's capitalization (“KubeCTL” for kubectl). Accepted.
- Words applied by bias aren't written to correction history yet, so they can't be asked about or undone the way other corrections can.