Term dictionaries, and the hint we hid

When you say “pull request” in a Russian sentence, the recognizer writes “пул реквест”. It writes “Антропик” for Anthropic and “Гитхаб экшн” for GitHub Actions. We spent a stretch of work on fixing that, and the result is not the one we expected.

Three ways to fix terms

The recognizer works on sound, so the first idea was to tell it which terms to listen for. We keep a small second model, 98 MB, that listens for the terms in the audio and rewrites the text where it hears one. We call this the hint. The second idea is a plain replacement list applied to the finished text, with no model. The third is a judge: a small language model that reads the sentence and decides whether a term really belongs in it.

We tested all three on four voices and 1,434 phrases, about 57 settings in total. For each way we counted phrases it fixed and phrases it broke.

• The hint: fixed 86, broke 253. It also took 2.26 seconds per phrase against 0.33 seconds without it.
• The replacement list alone: fixed 69, broke 16, instantly.
• The judge: fixed 79, broke 10. It costs almost no time, because the model only wakes up when a term candidate actually appears in the text.

The hint broke three times more than it fixed. Its typical failure is a similar-sounding word: “посмотрел” turning into “Postgres”. We tried trimming the term list to remove risky words and got the same three-to-one ratio. So we turned the hint off and hid its switch in settings. The code stays, with a comment saying why it is off.

What the judge does now

The judge used to ask the model only about variants we had marked as risky. We tested asking about every candidate instead. The net gain was the same, but it broke 6 phrases instead of 10, and it removes a hand-written list that could drift out of date. Now the judge asks about every candidate. Size matters here: a 1.7-billion-parameter model broke 24 phrases against 10 for a 4-billion one, with the same shift in word errors. Counting only errors per word would not have shown that.

Term lists for everyone

The built-in lists for programmers, designers and managers are open to everyone: you can pick them, edit them and make your own. We found a public list of 165 developer terms under the MIT licence, and nothing at all for designers or managers, so those lists we wrote ourselves. We also found no ready-made data on how a Russian speaker pronounces English terms, so that half is our own work too.

What not to trust in these numbers

Our test set has few terms: the 259 replacements touch only 16 to 28 phrases out of 380. The seconds measure processing a file, not the delay you feel in live dictation. One voice was an outlier, with a 28% word error rate against 7.5 to 12% for the others. We have not yet measured live-dictation delay with the judge asking about everything. Read the numbers as direction, not as a promise.