← back

MalScribe: Malayalam voice notes to text

I get Malayalam voice notes on WhatsApp. I wanted to read them: as Malayalam text, with an English translation beside it, and without listening to each one twice.

MalScribe does that. It is a command-line tool and a small native Mac app. Drop a voice note on it and you get the Malayalam transcript and the English translation, each with a Copy button.

Stock Whisper does not speak Malayalam

OpenAI’s Whisper large-v3 is the obvious starting point, and for Malayalam it is unusable. What worked was a Whisper medium model fine-tuned for Malayalam by Swathanthra Malayalam Computing (SMC), running on Apple’s MLX framework.

Malayalam brought two problems that English does not:

Cleaning up with an LLM

The raw transcript is rough, especially where people mix English words into Malayalam, which is most of the time. A second pass with Claude Sonnet cleans up the Malayalam and writes the English translation.

I also tried a local model through Ollama. It worked, but in testing it got the amounts and the dates wrong. In a voice note about paperwork, the amounts and the dates are the message.

A head-to-head test

I took a 14-second note that mixed Malayalam and English and ran it through several models.

Gemini is now the default. The CLI calls it over plain HTTP, tries a second model when the first is busy, and falls back to local Whisper plus Claude if both fail. On the test clip it came back perfect.

There is a catch. The key I used is on Google’s free tier, which allows 20 requests a day per model. My testing used up a day’s quota. The fix is to turn on billing for the key.

The Mac app

The app is SwiftUI and wraps the CLI. It takes files by drag and drop, from the Dock or from ⌘O, and queues them. Transcription runs at about real time, plus about 20 seconds for the cleanup.

MalScribe with an empty queue: Drop WhatsApp voice notes here An empty queue. The picker at the top right chooses the engine.

MalScribe transcribing and translating a WhatsApp voice note with Gemini A voice note on its way through Gemini.

It has a play button too. It used to hand the .opus file to whatever app macOS thought should open it. On my Mac that was the wrong app. The app now converts the audio and plays it itself.

The AI part

Claude built all of it: the chunking, the model comparison, the Gemini client with its retries and fallbacks, and the Mac app. I supplied the voice notes and judged the output, because I read Malayalam.