{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Hamel Husain / Parlance Labs",
  "title_listed": "vLLM & Large Models",
  "keywords": "vLLM,推理,大模型",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://hamel.dev/notes/llm/inference/big_inference.html",
  "publish_date": "2023-10",
  "corpus_stage": "推理部署",
  "notes": "vLLM与大模型推理",
  "id": "f5ba35dd4c45",
  "collected_at": "2026-10-02T14:29:04",
  "title": "vLLM & large models – Hamel’s Blog",
  "body_markdown": "vLLM & large models\nUsing tensor parallelism w/ vLLM & Modal to run Llama 70b\nPublished\nJuly 20, 2026\nWarning\nCorrection\nA previous version of this note suggested that you could run Llama 70b on a single A100. This was incorrect. The Modal container was caching the download of the much smaller 7b model. I have updated the post to reflect this.\nh/t to\nCade Daniel\nfor finding the mistake.\nIntroduction\nLarge models like\nLlama-2-70b\nmay not fit in a single GPU. I previously\nprofiled\nthe smaller 7b model against various inference tools. When a model is too big to fit on a single GPU, we can use\nvarious techniques\nto split the model across multiple GPUs.\nCompute & Reproducibility\nI used\nModal Labs\nfor serverless compute. Modal is very\neconomical\nand built for machine learning use cases. Unlike other clouds, there are plenty of A100s available. They even give you $30 of free credits, which is more than enough to run the experiments in this note. Thanks to Modal, the scripts I reference in this note are reproducible.\nIn this note, I’m using\nmodal client\nversion:\n0.50.2889\nDistributed Inference w/\nvLLM\nvLLM\nsupports tensor parallelism\n, which you can enable by passing the\ntensor_parallel_size\nargument to the\nLLM\nconstructor.\nI modified\nthis example Modal code\nfor\nLlama v2 13b\nto run\nLlama v2 70b\non 4 GPUs with tensor parallelism. Below is a simplified diff with the most important changes:\ndef download_model_to_folder():\nfrom huggingface_hub import snapshot_download\nsnapshot_download(\n-        \"meta-llama/Llama-2-13b-chat-hf\",\n+        \"meta-llama/Llama-2-70b-chat-hf\",\nlocal_dir=\"/model\",\ntoken=os.environ[\"HUGGINGFACE_TOKEN\"],\n)\nimage = (\nImage.from_dockerhub(\"nvcr.io/nvidia/pytorch:22.12-py3\")\n.pip_install(\"torch==2.0.1\", index_url=\"https://download.pytorch.org/whl/cu118\")\n+    # Pin vLLM to 8/2/2023\n+    .pip_install(\"vllm @ git+https://github.com/vllm-project/vllm.git@79af7e96a0e2fc9f340d1939192122c3ae38ff17\")\n-    # Pin vLLM to 07/19/2023\n-    .pip_install(\"vllm @ git+https://github.com/vllm-project/vllm.git@bda41c70ddb124134935a90a0d51304d2ac035e8\")\n# Use the barebones hf-transfer package for maximum download speeds. No progress bar, but expect 700MB/s.\n-    .pip_install(\"hf-transfer~=0.1\")\n+     #Force a rebuild to invalidate the cache (you can remove `force_build=True` after the first time)\n+    .pip_install(\"hf-transfer~=0.1\", force_build=True)\n.run_function(\ndownload_model_to_folder,\nsecret=Secret.from_name(\"huggingface\"),\ntimeout=60 * 20)\n)\n...\n-@stub.cls(gpu=\"A100\", secret=Secret.from_name(\"huggingface\"))\n+# You need a minimum of 4 A100s that are the 40GB version\n+@stub.cls(gpu=gpu.A100(count=4, memory=40), secret=Secret.from_name(\"huggingface\"))\nclass Model:\ndef __enter__(self):\nfrom vllm import LLM\n# Load the model. Tip: MPT models may require `trust_remote_code=true`.\n-       self.llm = LLM(MODEL_DIR)\n+       self.llm = LLM(MODEL_DIR, tensor_parallel_size=4)\n...\nSee\nbig-inference-vllm.py\nfor the actual script I used.\nWarning\nBe Careful To Mind The Cache When Downloading Files\nI found that when I ran the above code and changed the model name, I had to force a rebuild of the image to invalidate the cache. Otherwise, the old version of the model would be used. You can force a rebuild by adding\nforce_build=True\nto the\n.pip_install\ncall.\nWhen I initially wrote this note, I was fooled into believing I could load\nmeta-llama/Llama-2-70b-chat-hf\non a single A100. It was this tricky issue of the container that cached the download of the much smaller\n7b\nmodel. 🤦\nAfter setting the appropriate\nsecrets\nfor HuggingFace and Weights & Biases, You can run this code on Modal with the following command:\nmodal\nrun big-inference-vllm.py\nYou need\nat least 4 A100 GPUs\nto serve Llama v2 70b.\nWhat Happens With Smaller Models?\nEven though distributed inference is interesting for big models that do not fit on a single GPU, interesting things happen when you serve smaller models this way. Below, I test throughput for\nLlama v2 7b\non 1, 2, and 4 GPUs. The throughput is measured by passsing\nthese 59 prompts\nto\nllm.generate\n.\nllm.generate\nis described in the\nvLLM documentation\n:\nCall\nllm.generate\nto generate the outputs. It adds the input prompts to vLLM engine’s waiting queue and executes the vLLM engine to generate the outputs with high throughput.\nHere are the results, averaged over 5 runs for each row:\nYou can see all the individual runs\nhere\n. In my experiments, the 70b model needed a minimum of 4 A100s to run, so that’s why there is only one row for that model (Modal only has instances with 1, 2, or 4 GPUs).\n```{python}\nWarning\nDo Not Compare To Latency Benchmark\nThe tok/sec number you see here is VERY different than the latency benchmark shown on\nthis note\n. This particular benchmark maximizes throughput by running multiple requests in parallel. The previous latency benchmark measures the time it takes to process a single request.\nObservations\nA100s are much faster than A10s, but A10s are significantly cheaper.\n1\nOn A10s, scaling up to more GPUs increases throughput at first, but then seems to diminish. It appears like there is a Goldilocks zone in terms of the right number of GPUs to maximize throughput. I did not explore this in detail, as Modal only has instances with specific numbers of GPUs.\n2\nThe much larger\nLlama v2 70b\nmodel is only ~2x slower than its 7b counterpart.\nAside: Pipeline Parallelism\nIn theory, Pipeline Parallelism (“PP”) is slower than Tensor Parallelism, but tools for PP are compatible with a wider range of models from the HuggingFace Hub. By default, HuggingFace\naccelerate\nwill\nautomatically\nsplit the model across multiple GPUs when you pass\ndevice_map=\"auto\"\n. (Accelerate offers other kinds of parallelism as well, like integrations with\nDeepSpeed\n).\nThis blog post\nand\nthese docs\nare an excellent place to start. I will explore this and other kinds of parallelism in future notes.\nFootnotes\nAs of 8/6/2023 2 A10s costs\n.000612 / sec\non Modal, whereas 1 A100 40GB will cost\n0.001036 / sec\n. See\nthis pricing chart\n↩︎\nFor A10 and A100s you can only get up to 4 GPUs. Furthermore, I ran into an issue with vLLM and llama 70b, where it doesn’t like an odd number of GPUs.\n↩︎",
  "metadata": {
    "title": "vLLM & large models – Hamel’s Blog",
    "description": "Using tensor parallelism w/ vLLM & Modal to run Llama 70b",
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "vLLM & large models – Hamel’s Blog",
    "og:description": "Using tensor parallelism w/ vLLM & Modal to run Llama 70b",
    "og:image": "https://user-images.githubusercontent.com/1483922/208359430-f55d7503-3a98-4875-a35c-16314c9439d0.png",
    "og:site_name": "Hamel’s Blog",
    "og:locale": "en_US",
    "twitter:title": "vLLM & large models – Hamel’s Blog",
    "twitter:description": "Using tensor parallelism w/ vLLM & Modal to run Llama 70b",
    "twitter:image": "https://user-images.githubusercontent.com/1483922/208359430-f55d7503-3a98-4875-a35c-16314c9439d0.png",
    "twitter:creator": "@HamelHusain",
    "twitter:site": "@HamelHusain",
    "twitter:card": "summary_large_image"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/f5ba35dd4c45/code/body-000.txt",
      "chars": 1609,
      "language": "text"
    },
    {
      "file": "attachments/f5ba35dd4c45/code/body-001.txt",
      "chars": 31,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://x.com/edacih",
      "label": "Cade Daniel",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/meta-llama/Llama-2-70b-hf",
      "label": "Llama-2-70b",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/inference/inference.html",
      "label": "profiled",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/transformers/v4.15.0/parallelism",
      "label": "various techniques",
      "type": "a"
    },
    {
      "href": "https://modal.com/",
      "label": "Modal Labs",
      "type": "a"
    },
    {
      "href": "https://modal.com/pricing",
      "label": "economical",
      "type": "a"
    },
    {
      "href": "https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html",
      "label": "supports tensor parallelism",
      "type": "a"
    },
    {
      "href": "https://modal.com/docs/guide/ex/vllm_inference",
      "label": "this example Modal code",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/hamel-site/blob/master/notes/llm/inference/big-inference-vllm.py",
      "label": "big-inference-vllm.py",
      "type": "a"
    },
    {
      "href": "https://modal.com/docs/guide/secrets",
      "label": "secrets",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/hamel-site/blob/master/notes/llm/inference/questions.py",
      "label": "these 59 prompts",
      "type": "a"
    },
    {
      "href": "https://docs.vllm.ai/en/latest/getting_started/quickstart.html",
      "label": "vLLM documentation",
      "type": "a"
    },
    {
      "href": "https://wandb.ai/hamelsmu/llama-inference?workspace=user-hamelsmu",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/accelerate/index",
      "label": "accelerate",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/accelerate/usage_guides/big_modeling",
      "label": "automatically",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/accelerate/usage_guides/deepspeed",
      "label": "DeepSpeed",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/accelerate-large-models",
      "label": "This blog post",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:44Z",
  "source_url": "https://hamel.dev/notes/llm/inference/big_inference.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6180,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/f5ba35dd4c45/pages/source.html",
  "local_html": "attachments/f5ba35dd4c45/pages/article.html",
  "body_base_path": "attachments/f5ba35dd4c45/pages/",
  "body_markdown_path": "attachments/f5ba35dd4c45/pages/article.md"
}