We fine-tuned the recognizer on our own recordings
The recognizer in the app is Parakeet from NVIDIA. It is good out of the box, but it makes mistakes on exactly the things we dictate: product names, English words inside Russian speech, numbers. So we trained it a bit more on recordings we made ourselves. This is what we learned.
What we did
Fine-tuning means taking the finished model and continuing its training on new material. We did it on a Mac, with no rented graphics cards. The model has about 627 million parameters, and one pass over roughly 80 minutes of audio takes about 12 minutes.
The recordings are ours: a few people read hundreds of phrases aloud, including phrases built to trip the model. Examples are “сафари-парк”, which the model likes to write as “Safari Park”, and sentences full of numbers.
Which recipe won
We compared variants on two people the model had never heard, 100 new phrases each. Mistakes per hundred words, lower is better:
| Variant | Mistakes |
|---|---|
| Parakeet as NVIDIA ships it | 40.1 |
| Our previous version | 22.4 |
| Mixed with the base model at 0.4 | 25.0 |
| Mixed with the base model at 0.6 | 22.8 |
| No mixing | 19.5 |
| Distorted audio, mixed at 0.6 | 21.7 |
| Distorted audio, no mixing | 18.2 |
The last row is the winner: 4.2 points better than the previous version.
Two ideas deserve a plain explanation. “Distorted audio” means that while training, we add room echo and other damage to the recordings, so the model learns to cope with real rooms. “Mixing” means averaging the tuned model with the original, to keep it from forgetting what it knew. We used mixing at 0.6 for a while, and it turned out to hurt. It had been chosen on one voice and a first set of phrases that had few tricky cases. Once we added a second set with twice as many traps and numbers, the mixing only took away the gain.
What did not help
More data of the same kind. Three versions trained on more and more of it scored the same, within noise. New phrases from the same people did not help either, nor did stronger distortions, which were worse than mild ones. What helps is new voices and the right recipe.
How we measure
Only on voices the model never heard, and on phrases it never saw. If a voice was in the training, the number looks about a third better than it will be for a real person. So we train a separate model without each test voice and measure on that.
The tuned model is served from cdn.verbaide.com, and the app picks it up from there. Today: since 24 September the app asks before installing a new version. See languages and models.