A smaller judge for your dictionary terms

Speech recognizers write unusual words the way they sound. Say “Redis” in a Russian sentence and you may get “редис”. VerbaIDE has dictionaries of terms for this: a list of names and how they tend to be misheard. This week we trained a small model to make the final call, and it turned out nearly as good as one more than twice its size.

What the judge does

When the recognized text contains something that looks like a term from your dictionary, the app has to decide: is that really the term, or an ordinary word? “Redis” and a Russian word that sounds the same are different things, and a blind replacement breaks sentences that were right.

So the app asks a small language model, the judge. It reads the phrase and the candidate term and answers yes or no. We learned earlier that asking about every candidate is safer than trusting a hand-made list of risky ones. The catch is time: the question is asked while you wait for your text.

What we measured

Four voices read a set of test phrases that contain terms. For each model we counted two things: phrases the judge fixed (the term went in where it belonged) and phrases it broke (a right word was replaced). The score is fixed minus broken. “+63” means 63 more phrases got better than got worse.

• The 4-billion-parameter model: 70 fixed, 5 broken, +65.
• Our fine-tuned 1.7-billion model: 68 fixed, 5 broken, +63.
• The same 1.7-billion model before our training: 77 fixed, 23 broken, +54. It answered “yes” to almost everything.
• A fine-tuned 0.6-billion model: +58.

All five breakages of the fine-tuned model came from one disputed phrase, a trap we left in on purpose. We kept the trap words out of the training, so this measures words the model has not seen. On a held-out list of 25 unseen terms, the share of correct “no” answers went from 20% to 96%.

On speed, one question takes about 0.07 seconds on the 1.7-billion model against about 0.20 on the 4-billion one. The file is 1.7 GB against 3.3 GB. That is why we like it.

For the training we generated example phrases where the right answer is known from how they were built. The “+63” is a measurement on our own test set, not a promise for your speech.

Where it stands

It is not the default. The app can now point the judge at its own model through a hidden setting, separate from the style model. We turned it on on one of our own Macs on October 7. We have not measured it on live dictation yet. Until we do, everyone else keeps the current setup, and we will say so here when that changes.

More about the models is on the page languages and models.