cohere-transcribe-service

I was looking through the HuggingFace speech-to-text benchmarks and this model caught my eye since it is supposed to be very fast and accurate but it has an NVFP4 quant that doesn't take too much VRAM as well. This is an experimental, LLM-generated HTTP server which runs this model on a Blackwell GPU.

HTTP microservice that transcribes Opus-encoded audio using CohereLabs/cohere-transcribe-03-2026 in 4-bit NF4 quantization, tuned to coexist with another resident LLM on the same RTX 5090.

Quickstart

docker compose up --build
curl -X POST http://127.0.0.1:8001/v1/audio/transcriptions \
  --data-binary @sample.opus \
  -H "Content-Type: audio/ogg"

API

POST /v1/audio/transcriptions

• Body: raw bytes of an Ogg/Opus audio file.
• Query params (optional): language (en, fr, de, …, ar), punctuation (1/0).
• Response: {"text": "...", "elapsed_s": 0.43, "audio_seconds": 8.5, "language": "en", "punctuation": true}.

GET /health

Returns {"status": "ok", "model": "..."}.

VRAM budget

Target host: RTX 5090 (32 GB) with ~27 GB occupied by another LLM. 4-bit model footprint:

• Weights: ~1.2 GB
• Activations + KV cache: ~1.0–1.5 GB
• Total peak: ~2.5 GB (fits in ~4.5 GB free budget)

Citations

• Model card: https://huggingface.co/CohereLabs/cohere-transcribe-03-2026
• Transformers CohereAsr docs: https://huggingface.co/docs/transformers/model_doc/cohere_asr
• 4-bit quantization (bitsandbytes): https://huggingface.co/docs/transformers/main_classes/quantization
• RTX 5090 / Blackwell Docker setup: https://salad.mintlify.app/container-engine/tutorials/machine-learning/pytorch-rtx5090
• Opus/Ogg in-memory decoding (soundfile): https://python-soundfile.readthedocs.io/en/latest/#in-memory-files