{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "6 LLM Deployment Formats In Production",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/6-llm-deployment-formats-in-production/",
  "publish_date": "2026-08-07",
  "corpus_stage": "",
  "notes": "",
  "id": "87c9bb9407e7",
  "collected_at": "2026-10-02T13:46:26",
  "title": "6 LLM Deployment Formats in Production",
  "body_markdown": "Jul 30, 2026\nLLMOps\n6 LLM Deployment Formats in Production\n...explained visually.\nAvi Chawla\n👉\nGoogle and Anthropic agree on one thing about LLM inference:\nThe faster a model runs, the fewer machines can run it.\nThis sounds counterintuitive, but a model file gets faster by locking in hardware-specific choices into the file.\nAnd those choices only fit the hardware they were made for.\nThat is why they build the same model more than once, one version per chip or runtime it has to run on.\nThe visual below covers the six formats used to run LLMs in production:\n1-2) Pickle and safetensors hold raw weights. Both run anywhere Python runs, so neither is faster than the framework around it.\nA\n.pt\nfile is a list of instructions that\ntorch.load\nexecutes and loading a checkpoint can also run a payload (possibly malicious).\nSafetensors holds plain numbers instead, laid out the way memory needs them, so there is nothing to rebuild when the file opens.\nBoth still need Python, the model code, and a tokenizer alongside.\n3) GGUF is llama.cpp’s format, built so a model can run on a machine with no ML framework installed.\nTo achieve this, weights, tokenizer, and chat template ship in one file, and llama.cpp or Ollama runs it directly.\n4) ONNX is an interchange format, built so a model trained in one framework can run somewhere that framework was never installed.\nIt decides the operations and the order they run in, but leaves the hardware open, so the runtime picks CPU, CUDA, or an NPU when the file loads.\n5) MLX is Apple’s framework for its own silicon, where the CPU and GPU share a single pool of memory.\nIt decides where the weights live, so they are never copied between the two, and that advantage does not help on any other machine.\n6) TensorRT is NVIDIA’s compiler, which turns a model into machine code for one specific GPU.\nEverything is settled at build time, down to instructions tested on the exact card it compiles for, which is why the file will not load on a different GPU architecture.\nThe safe default is to stop as high on the chart as the latency budget allows, because every row down trades speed with the types of hardware it can run on.\nTo dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:\nRead Part 1 on fundamentals of LLMOps here →\nRead Part 2 on understanding the core building blocks of LLMs →\nRead Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →\nRead Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →\nRead Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →\nRead Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →\nRead Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →\nRead Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →\nRead Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →\nRead Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →\nRead Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →\nRead Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →\nRead Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →\nRead Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.\nGood day!\nPublished on Jul 30, 2026\nComments\nShare\nCopy link\nShare to X\nShare to Facebook\nShare to Linkedin\nCopied\nPrevious\n[Hands-on] Rebuilding Claude Code's Harness\nNext\nWhy RAG Latency Is a Prefill Problem, Not a Retrieval Problem",
  "metadata": {
    "title": "6 LLM Deployment Formats in Production",
    "description": "...explained visually.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "6 LLM Deployment Formats in Production",
    "og:description": "...explained visually.",
    "og:url": "https://www.dailydoseofds.com/p/6-llm-deployment-formats-in-production/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w1200/2026/08/f5d9157f-a52d-45dd-9e0c-cb7a6e88c4bd_1418x1350.png",
    "og:image:width": "1200",
    "og:image:height": "1142",
    "twitter:card": "summary_large_image",
    "twitter:title": "6 LLM Deployment Formats in Production",
    "twitter:description": "...explained visually.",
    "twitter:url": "https://www.dailydoseofds.com/p/6-llm-deployment-formats-in-production/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w1200/2026/08/f5d9157f-a52d-45dd-9e0c-cb7a6e88c4bd_1418x1350.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "LLMOps",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-07-29T19:18:00.000Z",
    "article:modified_time": "2026-08-07T19:20:27.000Z",
    "article:tag": "LLMOps"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "bytes": 11344,
      "reused": true,
      "local_path": "attachments/87c9bb9407e7/images/000.jpg",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/6-llm-formats-2.gif",
      "bytes": 1646118,
      "reused": true,
      "local_path": "attachments/87c9bb9407e7/images/001.gif",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/6-llm-formats-2.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!QLDU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60c7adad-7a59-4f1e-ba4d-4b2e0abc36d9_1412x414.gif",
      "bytes": 593552,
      "reused": true,
      "local_path": "attachments/87c9bb9407e7/images/004.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!QLDU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60c7adad-7a59-4f1e-ba4d-4b2e0abc36d9_1412x414.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!6I4q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F168490da-875c-40fc-bc59-a31c28b32df8_1412x206.gif",
      "bytes": 185826,
      "reused": true,
      "local_path": "attachments/87c9bb9407e7/images/005.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!6I4q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F168490da-875c-40fc-bc59-a31c28b32df8_1412x206.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!msDm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faee0b127-3913-43a7-9dfe-d15d88a22994_1412x206.gif",
      "bytes": 262852,
      "reused": true,
      "local_path": "attachments/87c9bb9407e7/images/006.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!msDm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faee0b127-3913-43a7-9dfe-d15d88a22994_1412x206.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!FkdI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d6b48eb-410d-4cc3-bfc9-16787c5ca3c2_1412x206.gif",
      "bytes": 271677,
      "reused": true,
      "local_path": "attachments/87c9bb9407e7/images/007.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!FkdI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d6b48eb-410d-4cc3-bfc9-16787c5ca3c2_1412x206.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!JnLO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2226426-4204-48f3-9546-1542a14dd243_1413x210.gif",
      "bytes": 231278,
      "reused": true,
      "local_path": "attachments/87c9bb9407e7/images/008.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!JnLO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2226426-4204-48f3-9546-1542a14dd243_1413x210.gif",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-1/",
      "label": "Read Part 1 on fundamentals of LLMOps here →",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-2",
      "label": "Read Part 2 on understanding the core building blocks of LLMs →",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-3",
      "label": "Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixt",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-4",
      "label": "Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applic",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-5",
      "label": "Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and differ",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-6",
      "label": "Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-7",
      "label": "Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techn",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-8",
      "label": "Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic contex",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-9",
      "label": "Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong und",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-10",
      "label": "Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for ev",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-11",
      "label": "Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-12/",
      "label": "Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF,",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-13/",
      "label": "Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model ",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-14",
      "label": "Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decision",
      "type": "a"
    },
    {
      "href": "https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2F6-llm-deployment-formats-in-production%2F&text=6%20LLM%20Deployment%20Formats%20in%20Production",
      "label": "Share to X",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2F6-llm-deployment-formats-in-production%2F",
      "label": "Share to Facebook",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2F6-llm-deployment-formats-in-production%2F",
      "label": "Share to Linkedin",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/p/hands-on-rebuilding-claude-codes-harness/",
      "label": "Previous [Hands-on] Rebuilding Claude Code's Harness",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/building-rag-systems-course-part-12-with-implementation/",
      "label": "Next Why RAG Latency Is a Prefill Problem, Not a Retrieval Problem",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:56Z",
  "source_url": "https://www.dailydoseofds.com/p/6-llm-deployment-formats-in-production/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4380,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/87c9bb9407e7/pages/source.html",
  "local_html": "attachments/87c9bb9407e7/pages/article.html",
  "body_base_path": "attachments/87c9bb9407e7/pages/",
  "body_markdown_path": "attachments/87c9bb9407e7/pages/article.md"
}