{ "source_file": "0930", "source_platform": "Dev.to", "author": "Tae Kim", "title_listed": "Running three AI models on one local server when your VRAM doesn't cover all of them", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://dev.to/hannune/running-three-ai-models-on-one-local-server-when-your-vram-doesnt-cover-all-of-them-b7g", "publish_date": "2026-08-17", "corpus_stage": "", "notes": "", "id": "ff97d60db0c3", "collected_at": "2026-10-02T14:03:54", "title": "Running three AI models on one local server when your VRAM doesn't cover all of them - DEV Community", "body_markdown": "The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it OOMâd immediately. Iâd known this was going to happen, but I tried anyway to see where the ceiling actually was.\nThe machine is a workstation I already had, enough VRAM for any single model but not three simultaneously. I had some options: get more hardware, split across multiple machines, or figure out a sequential loading pattern. I went with sequential because adding hardware or managing separate boxes felt like problems I didn't want to solve yet.\nSequential loading means: load whatever model you need, use it, unload before moving on. Nothing runs in parallel. This was fine for my use case because the workloads don't actually overlap in time â I'vm not running embedding lookups while transcribing a recording. The load time overhead adds a few seconds per task. For Whisper, that's ~4 seconds on top of ~90 seconds of transcription for a 30-minute call, which I don't notice in practice.\nThe bge-m3 decision took the most deliberation\nI started with\nall-MiniLM-L6-v2\nfor embeddings. It's small and fast. The issue was Korean/English mixed documents. I'm processing meeting summaries and reference docs that switch between the two, and MiniLM's cross-lingual similarity scores were unreliable enough that I was getting wrong nearest-neighbor results. bge-m3 handles cross-lingual matching better. It costs more VRAM and I had to drop the batch size from the default to stabilize it, but the accuracy difference on my actual data was clear.\nOne thing I didn't fully account for: when bge-m3 and Whisper are both unloaded, the load time for whichever comes next varies. bge-m3 seems to be slower on first load than subsequent loads in the same session, probably something to do with model weights caching at the OS level. I haven't investigated this properly.\nWhisper was the easy pick\nmedium.en. I ran large once to compare on a client call recording and the accuracy improvement wasn't worth the extra VRAM cost for my use case. Transcription time for a 30-minute recording is under two minutes with medium. That's fast enough that I'm not sitting watching it.\nGemma was a coin flip between a few candidates\nI needed local image analysis for screenshots and scanned documents. I tested gemma, a couple of LLaVA variants, and MiniCPM-V on maybe 20 images from my actual use case. Counted how many each model described correctly. Gemma came out ahead on document-heavy images. That test was informal enough that a different set of images might have given different results, but I had to pick something.\nOne thing I noticed later: gemma's processing time varies a lot by input image resolution. Standard screenshots are fast. A high-res photo from a phone camera takes noticeably longer. I still haven't profiled exactly what's happening there. If you're planning around latency, test with the actual image sizes you'll be using, not benchmarks from smaller inputs.\nWhere things stand\nThe sequential loading setup is stable. Iâve been running it for a few weeks without the OOM errors from the first attempt. The models work individually and I have code that chains them in sequence for a given input set.\nWhat Iâm still building is the layer that makes use of this: a pipeline that processes call recordings, emails, and messages together and produces something more useful than three separate outputs. The individual pieces are working. Connecting them into a coherent pipeline is the current work.\nOriginally posted at\nhannune.ai", "metadata": { "title": "Running three AI models on one local server when your VRAM doesn't cover all of them - DEV Community", "description": "The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it... Tagged with ai, selfhosted, machinelearning, llm.", "keywords": "ai, selfhosted, machinelearning, llm, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/hannune/running-three-ai-models-on-one-local-server-when-your-vram-doesnt-cover-all-of-them-b7g", "og:title": "Running three AI models on one local server when your VRAM doesn't cover all of them", "og:description": "The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1695668548342-c0c1ad479aee%3Fw%3D1200%26h%3D630%26fit%3Dcrop", "twitter:site": "@thepracticaldev", "twitter:creator": "@HannuneTech", "twitter:title": "Running three AI models on one local server when your VRAM doesn't cover all of them", "twitter:description": "The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1695668548342-c0c1ad479aee%3Fw%3D1200%26h%3D630%26fit%3Dcrop" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://hannune.ai/blog/deploying-gemma-bge-m3-whisper-one-server", "label": "hannune.ai", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:49:38Z", "source_url": "https://dev.to/hannune/running-three-ai-models-on-one-local-server-when-your-vram-doesnt-cover-all-of-them-b7g", "papers": [], "actual_video_files": [], "body_chars": 3546, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/ff97d60db0c3/pages/source.html", "local_html": "attachments/ff97d60db0c3/pages/article.html", "body_base_path": "attachments/ff97d60db0c3/pages/", "body_markdown_path": "attachments/ff97d60db0c3/pages/article.md" }