{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "Why I run speech-to-text locally instead of calling a cloud API",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api-59j7",
  "publish_date": "2026-08-18",
  "corpus_stage": "",
  "notes": "",
  "id": "2ea6604f0c9a",
  "collected_at": "2026-10-02T14:03:54",
  "title": "Why I run speech-to-text locally instead of calling a cloud API - DEV Community",
  "body_markdown": "Why I run speech-to-text locally instead of calling a cloud API\nYesterday I wrote about deploying gemma, bge-m3, and whisper on a single server without enough VRAM for all three.\nThis post is about why whisper is one of those three.\nThe problem with cloud STT\nWhen you call a cloud speech-to-text API—OpenAI's Whisper endpoint, Google Speech-to-Text, Amazon Transcribe—the audio travels to their servers. For most use cases that's fine. For mine it isn't.\nThe project I'm building transcribes work calls. Work calls contain client names, project specifics, sometimes pricing discussions. Sending that audio to a vendor's inference endpoint means it travels over the network and gets processed on hardware I don't control.\nRunning whisper locally means the audio file stays on the machine.\nWhat the setup actually looks like\nLocal server: RTX 3060 (12 GB VRAM)\nfaster-whisper\nin\nint8\nquantized mode\nAudio files uploaded from a phone via a PWA, stored on the same machine, processed there\nA transcription call looks like this:\nfrom\nfaster_whisper\nimport\nWhisperModel\nmodel\n=\nWhisperModel\n(\n\"\nmedium\n\"\n,\ndevice\n=\n\"\ncuda\n\"\n,\ncompute_type\n=\n\"\nint8\n\"\n)\nsegments\n,\ninfo\n=\nmodel\n.\ntranscribe\n(\n\"\nrecording.m4a\n\"\n,\nbeam_size\n=\n5\n)\nfor\nsegment\nin\nsegments\n:\nprint\n(\nf\n\"\n[\n{\nsegment\n.\nstart\n:\n.\n1\nf\n}\ns]\n{\nsegment\n.\ntext\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nNo API call. No auth header. No request log on a vendor's side. The audio file doesn't leave the machine.\nThe VRAM situation\nThe 12 GB card is shared with two other models. Whisper\nmedium\nin int8 uses roughly 2.5–3 GB of VRAM when warm. That's workable as long as the models don't run at the same time.\nThe three models load sequentially in my pipeline: transcribe → embed → analyze. Whisper runs first, which means it has the most headroom before the other two have loaded. When gemma is handling a VLM task and the card is under pressure, whisper drops to CPU. Slower—30–60 seconds per minute of audio instead of 8–12—but it completes without crashing.\nI didn't anticipate needing that fallback. It showed up during the first few real recordings when I hadn't fully worked out the load order yet.\nWhat this doesn't solve\nLocal STT doesn't fix transcription accuracy automatically. Whisper\nmedium\nis good but not perfect. Names, domain-specific terms, and cross-talk all degrade it. I haven't run it on enough actual work calls to have a reliable accuracy number. On clean audio, 90%+. On a speakerphone with background noise, noticeably worse.\nIt also doesn't solve the downstream pipeline. Right now I can transcribe a call and get a text file. The next piece—extracting structured information from that transcript, linking it to what came in over text and email—is what I'm working on now. That part isn't done.\nWhy not a hosted Whisper endpoint?\nHosted Whisper endpoints exist and some are cheap. If data residency doesn't matter, they're probably the right call: faster, no hardware to manage, no VRAM budgeting.\nThe asymmetry is the point. A slower local run that keeps audio in-house is worth the tradeoff when the content of the calls is the thing you're trying to protect. The cost of audio leaving is not symmetric with the cost of running it locally.\nWhere this fits\nI'm building a phone assistant that collects what comes in over calls, texts, and emails—and turns it into something useful instead of leaving it scattered across three different apps. Local transcription is the piece that's working. Everything past the transcript is still being built.\nIf you've dealt with local Whisper deployment on shared GPU, curious what you found: do you bother with beam_size tuning, or does the accuracy delta not matter enough at inference time to be worth it?\nPrevious post in this series: Running three AI models on one local server when your VRAM doesn't cover all of them",
  "metadata": {
    "title": "Why I run speech-to-text locally instead of calling a cloud API - DEV Community",
    "description": "Why I run speech-to-text locally instead of calling a cloud API   Yesterday I wrote about... Tagged with ai, selfhosted, llm, python.",
    "keywords": "ai, selfhosted, llm, python, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api-59j7",
    "og:title": "Why I run speech-to-text locally instead of calling a cloud API",
    "og:description": "Why I run speech-to-text locally instead of calling a cloud API   Yesterday I wrote about...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzx5r1a3x652fyact07re.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "Why I run speech-to-text locally instead of calling a cloud API",
    "twitter:description": "Why I run speech-to-text locally instead of calling a cloud API   Yesterday I wrote about...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzx5r1a3x652fyact07re.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2ea6604f0c9a/code/body-000.txt",
      "chars": 249,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://dev.to/hannune/running-three-ai-models-on-one-local-server-when-your-vram-doesnt-cover-all-of-them-b7g",
      "label": "Yesterday I wrote about deploying gemma, bge-m3, and whisper on a single server without enough VRAM for all three.",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:38Z",
  "source_url": "https://dev.to/hannune/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api-59j7",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3840,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2ea6604f0c9a/pages/source.html",
  "local_html": "attachments/2ea6604f0c9a/pages/article.html",
  "body_base_path": "attachments/2ea6604f0c9a/pages/",
  "body_markdown_path": "attachments/2ea6604f0c9a/pages/article.md"
}