Rethinking Speech-To-Text Support in Linux

All major operating systems ship with built-in speech-to-text accessibility features that let a user type into a text field by using their voice instead of a keyboard. Well, almost all of them. Linux distros don't. In fact, Linux distros can't even support that feature. What's worse is that there is not even proper third-party support for this.

Let's take a look at the current state of speech-to-text support in major operating systems:

Current Speech-to-text support in Linux

Some open-source projects try to bring STT functionality to Linux, but none provide an easy-to-use solution that works across all Linux distributions. And this is not their fault. The problem is that Linux doesn't have a single standardized way to implement STT engines and models. Each project has to implement everything from scratch, including the STT engine, model support, and a way to simulate input.

Projects like Nerd Dictation, Voxtype, OpenWhisper, and my own Super STT are all attempts at solving a problem that should not exist in the first place.

Each of these community projects does things differently, but in the end, each is just a hack to get STT working on Linux. Each has its pros and cons, but none of them are a proper solution to the problem.

The Solution

The solution is simple: Linux distributions need to ship with reliable built-in support for speech-to-text.

How we achieve that is not simple at all. At the very least we need to solve the following problems:

• Ship a speech-to-text engine with Linux distributions without bloating them.
• Create a system-level speech-to-text abstraction layer that lets applications interact with the engine.
• Create a native, safe, non-intrusive way to input into text fields across all desktop environments and applications.
• Develop an engine that can support tiny models, as well as powerful models.
• Address model weight licenses, their training data availability, and other legal considerations.
• Have each distro provide a list of models available for download that the STT engine can immediately use without any extra steps or requirements.

Starting from Zero

For the last year, my own project, Super STT, had been using Rust implementations of the Whisper and Voxtral models in order to support a quick and simple installation, without requiring Python, PyTorch, or other heavy system dependencies. With a binary size of only 26MB, the daemon was able to run over 12 different models, and if the user had an NVIDIA GPU with the CUDA Toolkit installed in their system, each of the models could be run on the GPU, offering much higher performance.

I was very happy with what Super STT could already accomplish. After all, Voxtral models were the state-of-the-art models for much of last year.

However, things felt odd. In order to support a new model, new code needed to be added to the binary directly, making the binary increase in size (although very little), and making things more fragile than they should be.

So a few months ago, I decided to rewrite things. I decided to turn Super STT into a proper, standardized, STT engine with the following features and capabilities:

• Ability to run any STT model in any programming language. From tiny, basic models to large, state-of-the-art models that can be GPU-accelerated.
• Small footprint so that it can ship anywhere.
• Must be daemon-based, so it can be enabled/disabled on demand if necessary.
• Must implement a protocol that other system apps can also implement to transcribe things natively. For example, a notes app could use the STT engine to get transcriptions when you press a button within its UI, displaying the real-time transcriptions in a different way from the final transcriptions, showing other app animations, etc.
• Must be security-conscious. Give the user control over which apps can interact with the engine.

The First Step

Let's pause for a moment, because I need to be fully transparent about this. I use LLMs to help me with the development of Super STT, especially in the last couple of months, where I needed to do a lot of refactoring and converting my protocol docs to an actual abstraction layer. Even with that extra (massive) help, Super STT still has a long way to go to be an actual solution to the bigger STT problem. This is a huge problem that, so far, I have mostly been tackling alone, so I will continue to use whatever help I can get.

I try to use LLM tools responsibly, but I know this is a controversial topic, so feel free to skip Super STT if it goes against your values or beliefs. Anything that Super STT does can be used as a reference to implement a better solution in other STT projects like Nerd Dictation, Voxtype, OpenWhisper, etc., if you would like to contribute to any of those awesome projects instead.

Now, let's continue. This week, I shipped v0.2.0 of Super STT, which does everything in that list. I also wanted to make it genuinely good and easy for people to use, so it includes a few extra things. The engine binary remained the same 26MB, which is probably already small enough to ship with a distro, but could be shrunk further with some optimizations.

Backends

Super STT now has the concept of backends. Each backend can add support for many models at once, although ideally one backend should add support for one family of models at most. For example, the Voxtral backend adds support for the voxtral-mini and voxtral-small models and is only 6MB. Likewise, the OpenAI backend adds support only for the OpenAI cloud models and is only 206KB!

Anyone can create their own backend and publish it to the default Super STT registry. Furthermore, any distro or person can replace this registry with their own in order to only list their own approved models for whatever reasons.

Super STT can run the following types of backends:

• WASM binaries, which are good for CPU-only models or online models that just need to connect to an online API.
• Subprocesses, which are good for running models that require more performance or need access to the GPU.

Each backend runs in a sandboxed manner, preventing it from changing system files or connecting to the internet without permission. For example, in order for the OpenAI backend to connect to OpenAI servers, it must first declare that it needs access to api.openai.com in its own configuration file.

As of this writing, the following backends are available in Super STT:

1. Voxtral (Rust Subprocess)
2. Whisper (Rust Subprocess)
3. Qwen-ASR (Python Subprocess)
4. Mistral (WASM - Online)
5. OpenAI (WASM - Online)
6. Deepgram (WASM - Online)

An Open Protocol

The daemon itself includes the STT engine and has everything necessary to download backends, load models, transcribe, and simulate typing into apps. However, in order to actually make the engine human-friendly, you want to be able to interact with it using a client. Super STT now has a full HTTP protocol, so a client can be implemented in any programming language and can even be a website. A client can manage the engine settings, backends, models, start/stop transcriptions, receive transcription updates, and do pretty much anything else you can think of.

The Super STT project right now has three clients:

• The main app (GUI), to manage the engine settings, download backends, load models, etc.
• A CLI app to allow transcriptions to be started/stopped via commands.
• A COSMIC Desktop applet that visualizes the user's voice when a transcription is started.

Anyone can create a client for the engine to extend capabilities or even just to replace the current main app to manage the engine with their own.

Security

Each client must receive scope-based permissions from the user in order for it to connect to the engine. When a client tries to connect to the engine for the first time or after a system restart, the engine will show the user a consent window asking for the user's permission before allowing the client's connection.

For now, the Super STT default clients are excluded from this security requirement and are allowed to connect without explicit consent. This decision was made to improve the user experience, but may be reverted in the future.

What It Accomplished

Super STT has been rebuilt to support a better architecture, getting it closer to being a real solution to the STT problem Linux has. However, the project still has a long way to go. Let's see a very optimistic view of what Super STT could theoretically take care of now that the new version of Super STT has been released.

• Ship a speech-to-text engine with Linux distributions without bloating them.

Yes, the engine itself is only 26MB unoptimized.

• Create a system-level speech-to-text abstraction layer that lets applications interact with the engine.

Yes, the abstraction layer foundation is there now, although it is meaningless if not adopted by anyone.

• Create a native, safe, non-intrusive way to input into text fields across all desktop environments and applications.

No, this hasn't been touched yet.

• Develop an engine that can support tiny models, as well as powerful models.

Yes, the engine can run any model from any framework. For example, it runs backends that use Rust (Candle) and other backends that use Python (PyTorch).

• Address model weight licenses, their training data availability, and other legal considerations.

Maybe? A custom registry (see below) could help, but I need to research this more.

• Have each distro provide a list of models available for download that the STT engine can immediately use without any extra steps or requirements.

Yes, this is possible by swapping out the registry index to one that the distro provides.

What's Next?

I hope my work sparks some conversations about the speech-to-text issues that Linux currently has. Super STT is not meant to be a perfect solution, at least not yet. For now, Super STT is chaotic and bleeding-edge and will continue to push things in a better direction for everyone.

The next step is simply to listen to what others have to say about this issue, the protocol, security, and any other aspects of the project.