Translating the text under your mouse pointer

Subtitles cover speech. Most English we meet is written, though: a page, a chat, a code review. Today the app learned to translate the sentence under your mouse pointer, in any application.

How it works

You hold a key combination and point at English text. By default the combination is Option and Command. A small panel appears next to the pointer with three blocks: the original in small type, the Russian translation under it, and a dictionary entry for the word under the pointer. We look the word up in English, not in its translation, because a definition of the English word is more useful. The system dictionary reduces the word to its base form, so “caching” finds “cache”.

If you have already selected a phrase and the pointer is inside the selection, the app translates the whole selection instead of one sentence. Only inside: an old selection somewhere else in the text would otherwise override whatever you point at later, and you would not understand why the wrong text was translated. A very long selection is cut at 600 characters at a word boundary, because translating a whole document you selected with Command-A is slow and pointless.

The mode is off by default. It needs the Accessibility permission, and we do not ask for a permission from someone who has not asked about the feature.

Reading text without watching the screen

The obvious way to read text under a pointer is to recognize the screen picture. That would put the purple screen-recording indicator in the menu bar permanently. We first checked what the system’s accessibility interface returns. In the browser, the code editor and the chat app we tried, it gives the sentence exactly at the pointer position. So no screen recording was needed.

Browsers were the hard part. For a selection that spans a heading and a paragraph, the accessibility interface returns one flat string, and the heading sticks to the next line. Our first fix read the fonts of the selection and built the structure from font sizes. It worked, but it was a workaround. A better source turned out to be the clipboard: browsers put the full rich text there. So the app sends Command-C to the active app, reads the formatted copy, and puts your previous clipboard content back. It compares the copy with what accessibility returned, so it never shows a text from another window. Headings, lists and bold text survive.

The translator does not keep markers like “##” or list dashes. We take them off, translate the text, and put them back. If the translator returns a different number of lines, we show its answer as it is.

The pinned window

A pin in the corner of the panel turns it into a normal window. It stays, you can resize it, select text with the mouse, and look up a selected word. Each block has its own buttons for smaller and larger type, and the size is remembered. A speaker button reads the block aloud. We choose the voice by alphabet, not by the system’s language detector: it called the word “масштаб” Ukrainian, and there is no Ukrainian voice on the Mac, so the speech would have silently failed. A text with at least one Cyrillic letter is read with the Russian voice.