We measured cloud recognizers and stayed local
Two big cloud services now offer speech recognition that lists Russian and Ukrainian: OpenAI’s GPT-Transcribe and Google’s Gemini 3.5 Transcribe. We wanted to know how much better they are than the model that runs on your Mac. So we measured.
The test
We used our own set of phrases, read aloud by four people: 380 phrases per voice, 294 of them for Ukrainian. The services got the audio files with no hints and no term lists, and we compared the result with Parakeet, our on-device model, under the same conditions. We counted the share of words that came out wrong, ignoring punctuation. This test is the one case where recordings were sent to those services. The app itself does not do this.
Share of wrong words, all voices together:
- Russian: Parakeet 13.5%, Gemini 10.0%, GPT-Transcribe 9.8%
- Ukrainian: Parakeet 18.0%, Gemini 20.5%, GPT-Transcribe 14.6%
- English: Parakeet 13.7%, Gemini 13.1%, GPT-Transcribe 10.6%
What we saw
GPT-Transcribe was 3 to 4 points better in every language, and the gap was clear, not a fluke of the sample. Gemini was close to Parakeet in Russian and English. In Ukrainian, with no language hint, it was worse than Parakeet: it finished short phrases in Russian. It also writes numbers as digits.
One thing did not improve. On phrases with English terms inside Russian speech, the cloud services still got 17 to 26% of words wrong against Parakeet’s 32%. “Антропик”, “докер” and “мигрейт” came out in Cyrillic from all of them. So a term dictionary would still be needed on top of any cloud recognizer.
Why we said no
A cloud tier would have cost us real money for each minute of speech, a server to hand out access tokens, two outside integrations to keep working. The larger cost is the promise itself. Today speech never leaves your Mac, and that sentence is short and true. Three or four points of accuracy are not worth giving it up.
There is a caution about the numbers, too. Our phrases were read from a sheet, not spoken freely. We have been burned by this before: an earlier model looked 1 to 2 points better on read-aloud phrases and then did worse on live dictation, with “uh” in the text and sentences cut in half. We removed it. A gain on a test sheet does not always carry over to real speech.
What we do instead
We try to win those 3 to 4 points on the device: other local models, measured right away on real dictation recordings, processing the audio before recognition, and further training Parakeet on our own recordings. We will report what works. If you want to see exactly what leaves your Mac today, it is listed on what leaves your Mac.