Why we removed a recognizer that scored better
For a while the app had a second Russian recognizer, GigaAM. It re-checked every Russian piece of dictation after the main recognizer, Parakeet. On paper it was better. In real use it was worse, so we removed it completely.
What the numbers said
We keep a fixed set of test phrases that people read aloud, and we count the mistakes the recognizer makes on them. On that set GigaAM gave one to two points fewer mistakes per hundred words than Parakeet alone. That is a clear win, and it is why we added it.
What live speech showed
A test set of read phrases is tidy. People read a sentence from a sheet, speak evenly and do not stumble. Real dictation is not like that. You think while you talk, you pause, you restart.
We looked at the text GigaAM produced from our own everyday dictation, 557 re-checked pieces in a single day. It was bad in four ways:
- it wrote “э-э” and “и-и-и” into the text, the hesitation sounds that nobody wants on the page;
- it cut sentences with a full stop and a capital letter in the middle of a phrase;
- it dropped the last word where one piece of audio met the next;
- in four pieces out of ten it changed only the punctuation, and made it worse.
None of this showed on the test set, because the test set has no hesitations and no half-finished sentences.
What we decided
We could have tried to patch the check so that the test set caught these problems. We chose not to. The person who dictates every day saw the problem in the text, and that is the test that counts. We removed GigaAM from the app, from the settings and from the list of components. We also took the model files off our download server. The app deletes any leftover copy from the Mac at launch.
The lesson is simple. A number from read phrases does not carry over to spontaneous speech. If a new candidate for Russian appears, we will test it on real dictation first and on read phrases second. Parakeet remains our recognizer for Russian.
If you want to know what we use today and how the models are installed, see languages and models.