Dictation into any field, and the secretary

Until this week the app listened to other people’s speech. Now it also listens to yours. You speak Russian, and the text lands where your cursor is, either as spoken or translated into English. Getting there took three steps.

Step 1: Russian and English at once (August 3)

The first job was listening to mixed English and Russian speech for the live subtitles. Apple’s English recognizer has no Russian at all. We checked the list of languages it supports on a Mac: 30 of them, and Russian is not among them. Russian lives in a different recognizer from the same framework, built for dictation. So we run both at the same time on the same sound, and for each phrase we pick the one that is more confident about what it heard. A Russian phrase is kept as it is and not translated; an English one is translated into Russian. A single phrase gets one language: the recognizer cannot switch language in the middle, so a foreign word inside a sentence is written down by ear in the winning language.

The Russian recognizer has a flaw we measured the same day: it puts no punctuation, even when asked. A sentence arrives as one line without full stops. That is bad for reading, and worse for translation: a request “перепроверь” came out as “I’ll double-check”, a promise instead of a request. So dictation does not use it: Russian dictation runs on a separate model, Parakeet, which puts punctuation in by itself. The recognizer pair above serves the live subtitles.

Step 2: dictation by hotkey (August 4)

Two key combinations. Option-Space puts what you said into the field under the cursor as it was said. Option-Control-Space translates it into English first. The text is pasted as one piece when you finish. The paste has guards: if something is selected in the field, or a password field is active, it does not blindly overwrite anything. Spoken punctuation works too: say the name of a mark and you get the mark.

Step 3: the secretary (August 5)

The third combination, Option-Command-Space, starts a long session. You talk for as long as you need, with pauses and distractions, and each recognized piece is pasted straight into the field where you began. You can switch to another app meanwhile; the text keeps going to the original place.

That promise needs care. Before every paste the app checks that the target application is still running, that the focus is still in the same field (not another field of the same app, such as a search box), that nothing is selected there, and that the system is not in password entry. If the focus moved, the piece waits and the panel says so, rather than pasting somewhere else. The session ends with the same key, with a cross on the panel, or after 15 minutes.

We deliberately did not capture the Escape key for the whole session. Capturing it would stop Escape from working in every other app while you dictate, and a session lasts minutes. The cross on the panel ends it instead.

One bug found on August 5: pasting silently failed in chat apps built on Electron when the keyboard layout was not Latin. We now switch to a Latin layout for the instant of the paste.

Today: both ways are now one dictation. Hold a key for a phrase, or tap it briefly for the secretary, and both are free.