Building Hushcue

Hushcue is a Windows app that makes subtitles for videos, entirely on your own PC. You drop in a video and get an .srt, a subtitle track in an .mkv, or the text burned into an .mp4. Underneath it runs OpenAI's Whisper.

It went from an empty solution to a released installer in two days. This is how it got built, what broke along the way, and what I learned about making automatic subtitles good.

Where it started

Whisper is very good at turning speech into words. Putting raw Whisper output on screen is another matter. It invents "Thank you for watching" over silence. It gets stuck and repeats the same line twenty times. It spells names three different ways in one video. And it has no idea how long a subtitle should stay up or where a line should break.

I had a Python tool built on faster-whisper that handled most of that, and it worked. But it was a Python install, a virtual environment and a command line. I wanted a proper desktop app: an installer, a window, drag and drop. So the plan was to rewrite it as a native Windows app in C# with WinUI 3, and to end up with at least the same quality, at the same speed.

Picking the engine

The obvious way to run Whisper from C# is Whisper.net, a wrapper around whisper.cpp. Before building anything on it, I ran it against the Python tool on the same four clips, with the same decoding settings and the same clean-up, so the engine was the only difference.

(The reference transcripts were rough drafts, so the absolute numbers are high. The gap is what counts.)

The quantized models were worse still: they fell into repetition loops, and Whisper.net has no repetition penalty to pull them out. The timings mattered most, because everything that builds subtitles depends on knowing when each word starts and ends. So Whisper.net was a no-go on day one.

faster-whisper's speed comes from CTranslate2, a C++ inference library; the Python part is a layer on top. So Hushcue calls CTranslate2 directly, through a small C++ bridge with a plain C interface, and everything faster-whisper does in Python got ported to C#: the audio features, the tokenizer, the decode loop, word timestamps, the hallucination filter, language detection, and later the Silero voice detector.

Porting it exactly

The rule for the port was the same output, not roughly the same output. That made it testable:

• Given the same audio features, the C# decoder produces the same words and timings as faster-whisper on every test clip, in every mode.
• The C# features are 98% bit-identical to numpy's. Python's rounding even got its own helper, because round(x, 2) doesn't behave like .NET's default.
• The subtitle clean-up has a golden test: the C# output must be byte-identical to the Python version on 60 cases.

It paid off for the rest of the project. If a number moved, it was because I changed something, not because the port drifted. End to end, Hushcue came out about 9% faster than the Python tool on the same GPU, with the same engine and less start-up.

The pipeline

Here is what happens to each file.

1. Check the file. Read its length and whether it has sound and a picture (cover art doesn't count), so bad files are flagged before the run, and the batch's time left can be estimated from each file's length.
2. Pull out the sound. ffmpeg decodes the audio straight into memory as 16 kHz mono. On the Thorough setting it is cleaned up first: low rumble cut, steady noise reduced, levels evened out.
3. Skip silence (optional). Silero VAD finds the parts with speech so the model only runs on those. It's off by default, because it can miss quiet or breathy speech.
4. Transcribe. Whisper runs in 30-second windows. Every window gets the user's vocabulary, so names, brands and slang get spelled their way all the way through. Whisper has never seen most stream chat, so without it you get things like "poggers" → "poker's" and "KEKW" → "keck W". Decoding runs at temperature 0 only: Whisper's default fallback re-decodes hard parts with random sampling, which made two identical runs disagree on about 38% of the words.
5. Repair. Looped passages are cut out of the audio and decoded again on their own, with a repetition penalty. Anything that still loops is capped.
6. Build the cues. Words become subtitles with line breaks and timings.
7. Review. A screen that jumps to what's worth checking: invented phrases, loops, long gaps, words Whisper wasn't sure of.
8. Deliver. Save the .srt, add it as a track, or burn it in. Embedding runs alongside transcription, joined by a channel, so file 2 is embedded while file 3 is transcribed.

Measure first

Once the port matched, the question became: where is Whisper actually wrong? Most of its real mistakes aren't random. They're words that sound like other words: their / there, principle / principal, brake / break, weather / whether. Only the words around them can decide which one is meant.

So I built a small test set: four spoken passages of about a minute each, with 93 scored words that have a sound-alike twin, plus six unusual names given in a vocabulary. Each passage also comes in a hard version (faster speech, loud noise, telephone bandwidth) and one with long silent gaps, where Whisper likes to make things up. One command runs the app's whole job on them and prints every miss.

The first run found a real bug. The names were right for the first 25 seconds of each passage and wrong after that. The vocabulary went in as Whisper's initial prompt, and with previous-text context off, that prompt is dropped after the first window. Moving it into every window took the names from 15 of 38 right to 31 of 38, and the overall word error rate from 2.5% to 1.3%. A one-line cause, invisible without a test that ran past 30 seconds.

The same test set also talked me out of an idea. Letting each window see the text before it sounds like it should help, since more context should mean better guesses. The error rate went up instead, because Whisper carried invented text from one window to the next. In one run it filled a pause with "Thank you for watching. Please subscribe to the channel and press the bell icon." So that stays off.

"Not sure"

The sound-alike misses that were left were the same handful of words in every version of the audio, clean or noisy. That means they come from the model's sense of language, not from the sound, and no audio trick fixes them.

So the fix went into the review instead. Whisper gives each word a probability. When a word from one of about 75 sound-alike groups comes in under a threshold, the review marks it not sure with one-click swaps: Whisper wasn't sure of "principal": it could be "principle". On the test set, Balanced flags 28 of 559 sound-alike words and catches 13 of the 14 that are actually wrong.

I also tried flagging every low-probability word. Not worth it: of the words under 0.5, only one in ten is wrong, and nobody reads ten false alarms to find one.

The words are only half of it

A subtitle file has two parts: the words, and the cues, meaning when each subtitle appears, how long it stays and where its lines break. Viewers notice bad cues at least as much as a wrong word, and cues are cheap to fix in code.

Broadcasters have worked this out already. Netflix's English style guide asks for at most two lines of 42 characters, a reading speed of up to 20 characters a second, and between 5/6 of a second and 7 seconds on screen. It also says where lines may break: after punctuation or before a conjunction, and never between an article and its noun, a first and last name, or a pronoun and its verb.

The first release just balanced the two lines' lengths and ended each subtitle on its last word. The version I'm working on now scores every possible break by those rules, cuts long subtitles at a clause, and holds each one into the pause after it until it can be read. It also:

• drops short "Thank you" subtitles when the voice detector hears no speech under them, and keeps them when someone really says it
• trims each word to the speech the voice detector hears in it, so a subtitle doesn't appear before anyone talks
• starts every window from a short punctuated sentence, whose style Whisper copies, so later windows keep their capitals and full stops

Shipping it

Getting it onto other people's PCs was its own small project.

• cuDNN turned out not to be needed. The plan was to ship NVIDIA's cuBLAS and cuDNN, about 1.8 GB. Checking how the CTranslate2 wheel is built showed it never loads cuDNN, which cut 1.1 GB. The rest isn't shipped either: the app downloads cuBLAS (about 630 MB) on first run, only on PCs with an NVIDIA GPU. Without one, it runs on the CPU.
• The speech model is downloaded on first run too, about 1.6 GB, pinned to a commit with every file's checksum, and resumed if the connection drops. Both downloads are tested against a local fake server that can cut the connection or stall halfway.
• Published builds crashed where development builds didn't. An unpackaged WinUI publish leaves out the compiled XAML unless MSIX tooling is switched on. The native libraries also need the Visual C++ runtime, which a clean Windows doesn't have, so it ships next to the exe.
• Startup speed: ReadyToRun opens the window about 25% sooner for 44 MB more. Trimming would halve the .NET part, but the trimmed app crashes in XAML at start, so that's for later.

The installer is 66 MB, installs per user without an admin prompt, and adds "Generate subtitles with Hushcue" to the right-click menu of video files. Pushing a version tag builds it on GitHub Actions and publishes a release with checksums, and the app checks for new releases once a day and verifies the download before installing it.

Where it is now

Hushcue 0.1.1 is out. Next is releasing the subtitle timing work above, snapping subtitle edges to shot changes, and code signing, so Windows stops warning about the installer.

The code is MIT licensed and on GitHub.