{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Hamel Husain / Parlance Labs",
  "title_listed": "Optimizing LLM latency",
  "keywords": "推理延迟,优化,LLM",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://hamel.dev/notes/llm/inference/inference.html",
  "publish_date": "2023-10",
  "corpus_stage": "推理部署",
  "notes": "LLM推理延迟优化",
  "id": "aef6398af80f",
  "collected_at": "2026-10-02T14:29:05",
  "title": "Optimizing latency – Hamel’s Blog",
  "body_markdown": "Optimizing latency\nAn exploration of ways to optimize on latency.\nPublished\nJuly 20, 2026\nSummary\nBelow is a summary of my findings:\n🏁\nmlc\nis the fastest\n. This is so fast that I’m skeptical and am now motivated to measure quality (if I have time). When checking the outputs manually, they didn’t seem that different than other approaches.\n❤️\nCTranslate2\nis my favorite tool, which is among the fastest but is also the easiest to use\n. The documentation is the best out of all of the solutions I tried. Furthermore, I think that the ergonomics are excellent for the models that they support. Unlike vLLM, CTranslate doesn’t seem to support distributed inference just yet.\n🛠️\nvLLM\nis really fast, but CTranslate can be much faster.\nOn other hand,\nvLLM supports distributed inference\n, which is something you will need for larger models.\nvLLM might be the sweet spot for serving very large models.\n😐\nText Generation Inference\nis an ok option (but nowhere near as fast as\nvLLM\n) if you want to deploy HuggingFace LLMs in a standard way\n. TGI has some nice features like telemetry baked in (\nvia OpenTelemetry\n) and integration with the HF ecosystem like\ninference endpoints\n. One thing to note that as of 7/28/2023, the license for TGI was changed to be more\nrestrictive that may interfere with certain commercial uses\n. I am personally not a fan of the license.\nRough Benchmarks\nThis study focuses on various approaches to optimizing\nlatency\n. Specifically, I want to know which tools are the most effective at optimizing latency for open source LLMs. In order to focus on latency, I hold the following variables constant:\nbatch size of\nn = 1\nfor all prediction requests (holding throughput constant).\n1\nAll experiments were conducted on a\nNvidia A6000\nGPU, unless otherwise noted.\nMax output tokens were always set to\n200\n.\nAll numbers are calculated as an average over a fixed set of 9 prompts.\nThe model used is\nmeta-llama/Llama-2-7b-hf\non the HuggingFace Hub\n2\n.\nIn addition to batch size of\nn = 1\nand using a\nA6000\nGPU (unless noted otherwise), I also made sure I warmed up the model by sending an initial inference request before measuring latency.\nLlama-v2-7b benchmark:\nbatch size = 1, max output tokens = 200\navg tok/sec\navg time (seconds)\navg output token count\nplatform\noptions\ngpu\nCTranslate2\nfloat16 quantization\nA6000\n44.8\n4.5\n200.0\nint8 quantization\nA6000\n62.6\n3.2\n200.0\nHF Hosted Inference Endpoint\n-\nA10G\n30.4\n6.6\n202.0\nHuggingFace Transformers (no server)\n-\nA6000\n24.6\n7.5\n181.4\nnf4 4bit quantization bitsandbytes\nA6000\n24.3\n7.6\n181.4\nTGI\n-\nA6000\n21.1\n9.5\n200.0\nquantized w/ GPTQ\nA6000\n23.6\n8.8\n200.0\nquantized w/ bitsandbytes\nA6000\n1.9\n103.0\n200.0\nmlc\nq4f16\nA6000\n117.1\n1.3\n153.9\ntext-generation-webui\nexllama\nA6000\n77.0\n1.7\n134.0\nvllm\n-\nA100 (on Modal Labs)\n41.5\n3.4\n143.1\nA6000\n46.4\n3.8\n178.0\nIn some cases I did not use an\nA6000\nb/c the platform didn’t have that particular GPU available. You can ignore these rows if you like, but I still think it is valuable information. I had access to a A6000, so I just used what I had.\nI noticed that the output of the LLM was quite different (less tokens) when using\nvLLM\n. I am not sure if I did something wrong here, or it changes the behavior of the LLM.\nFurthermore, the goal was not to be super precise on these benchmarks but rather to get a general sense of how things work and how they might compare to each other out of the box. Some of the tools above are inference servers which perform logging, tracing etc. in addition to optimizing models which effect latency. The idea is to see where there are significant differences between tools. I discussed this more\nhere\n.\nBackground\nOne capability you need to be successful with open source LLMs is the ability to serve models efficiently. There are two categories of tools for model inference:\nInference servers:\nthese help with providing a web server that can provide a REST/grpc or other interface to interact with your model as a service. These inference servers usually have parameters to help you make\ntrade-offs between throughput and latency\n. Additionally, some inference servers come with additional features like telemetry, model versioning and more. You can learn more about this topic the\nserving section\nof these notes. For LLMs, popular inference servers are the\nText Generation Inference (TGI)\nand\nvLLM\n.\nModel Optimization\n: These modify your model to make them faster for inference. Examples include\nquantization\n,\nPaged Attention\n,\nExllama\nand more.\nIt is common to use both\nInference servers\nand\nModel Optimization\ntechniques in conjunction. Some inference servers like\nTGI\nand\nvLLM\neven help you apply optimization techniques.\n3\nNotes On Tools\nOther than benchmarking, an important goal of this study was to understand how to use different platforms & tools.\nmlc\nStart with compiling the model as shown in\nthese docs\nAfter\ninstalling MLC\n, you can compile\nmeta-llama/Llama-2-7b-chat-hf\nlike so:\npython3\n-m\nmlc_llm.build\n\\\n--hf-path meta-llama/Llama-2-7b-chat-hf\n\\\n--target cuda\n--quantization\nq4f16_1\nThe arguments for the compliation are documented\nhere\n. This puts the model in the\n./dist/\nfolder with the name\nLlama-2-7b-chat-hf-q4f16_1\n.\nYou can use\ntheir python client\nto interact with the compiled model:\nfrom\nmlc_chat\nimport\nChatModule, ChatConfig\ncfg\n=\nChatConfig(max_gen_len\n=\n200\n)\ncm\n=\nChatModule(model\n=\n\"Llama-2-7b-chat-hf-q4f16_1\"\n, chat_config\n=\ncfg)\noutput\n=\ncm.generate(prompt\n=\nprompt)\nYou can see the full benchmarking code\nhere\n.\nWarning\nI wasn’t able to get\nmeta-llama/Llama-2-7b-hf\nto run correctly with the supplied python client so I am using the chat variant (\nLlama-2-7b-chat-hf\n) as a proxy. I asked the kind folks who work on the mlc project and they said the python client is currently designed for chat, such that they have\nthis system prompt\nthat is hard coded for llama models:\nconv.system =\n      (\"[INST] <<SYS>>\\n\\nYou are a helpful, respectful and honest assistant. \"\n       \"Always answer as helpfully as possible, while being safe. \"\n       \"Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, \"\n       \"or illegal content. \"\n       \"Please ensure that your responses are socially unbiased and positive in nature.\\n\\n\"\n       \"If a question does not make any sense, or is not factually coherent, explain why instead \"\n       \"of answering something not correct. \"\n       \"If you don't know the answer to a question, please don't share false \"\n       \"information.\\n<</SYS>>\\n\\n \");\nIf you want to fix this, you must edit\nmlc-chat-config.json\n, changing\nconv_template\nto\nLM\n.\nThese docs\nsay more about the\nconfig.json\n.\nThe config file is located in\n./dist/<model-name>/params/mlc-chat-config.json\n. For example:\n> cat ./dist/Llama-2-7b-hf-q4f16_1/params/mlc-chat-config.json\n\n{\n    \"model_lib\": \"Llama-2-7b-hf-q4f16_1\",\n    \"local_id\": \"Llama-2-7b-hf-q4f16_1\",\n    \"conv_template\": \"llama-2\",\n    \"temperature\": 0.7,\n    \"repetition_penalty\": 1.0,\n    \"top_p\": 0.95,\n    \"mean_gen_len\": 128,\n    \"max_gen_len\": 512,\n    \"shift_fill_factor\": 0.3,\n    \"tokenizer_files\": [\n        \"tokenizer.json\",\n        \"tokenizer.model\"\n    ],\n    \"model_category\": \"llama\",\n    \"model_name\": \"Llama-2-7b-hf\"\n}\nCTranslate2\nCTranslate2\nis an optimization tool that can make models ridiculously fast. h/t to\nAnton\n. The documentation for CTranslate2 contains\nspecific instructions for llama models\n.\nTo optimize\nllama v2\n, we first need to quantize the model. This can be done like so:\nct2-transformers-converter\n--model\nmeta-llama/Llama-2-7b-hf\n--quantization\nint8\n--output_dir\nllama-2-7b-ct2\n--force\nmeta-llama/Llama-2-7b-hf\nrefers to the\nHuggingFace repo for this model\n. The benchmarking code is as follows (can also be found\nhere\n):\nimport\ntime\nimport\nctranslate2\nimport\ntransformers\nimport\nsys\nsys.path.append(\n'../common/'\n)\nfrom\nquestions\nimport\nquestions\nimport\npandas\nas\npd\ngenerator\n=\nctranslate2.Generator(\n\"llama-2-7b-ct2\"\n, device\n=\n\"cuda\"\n)\ntokenizer\n=\ntransformers.AutoTokenizer.from_pretrained(\n\"meta-llama/Llama-2-7b-hf\"\n)\ndef\npredict(prompt:\nstr\n):\n\"Generate text give a prompt\"\nstart\n=\ntime.perf_counter()\ntokens\n=\ntokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))\nresults\n=\ngenerator.generate_batch([tokens], sampling_topk\n=\n1\n, max_length\n=\n200\n, include_prompt_in_result\n=\nFalse\n)\ntokens\n=\nresults[\n0\n].sequences_ids[\n0\n]\noutput\n=\ntokenizer.decode(tokens)\nrequest_time\n=\ntime.perf_counter()\n-\nstart\nreturn\n{\n'tok_count'\n:\nlen\n(tokens),\n'time'\n: request_time,\n'question'\n: prompt,\n'answer'\n: output,\n'note'\n:\n'CTranslate2 int8 quantization'\n}\nif\n__name__\n==\n'__main__'\n:\ncounter\n=\n1\nresponses\n=\n[]\nfor\nq\nin\nquestions:\nif\ncounter\n>=\n2\n: responses.append(predict(q))\ncounter\n+=\n1\ndf\n=\npd.DataFrame(responses)\ndf.to_csv(\n'bench-ctranslate-int8.csv'\n, index\n=\nFalse\n)\nText Generation Inference (TGI)\nWarning\nLicense Restrictions\nThe license for TGI was\nrecently changed\naway from Apache 2.0 to be more restrictive. Be careful when using TGI in commercial applications.\nText generation inference\nwhich is often referred to as “TGI” was easy to use without any optimization. You can run it like this:\n“start_server.sh”\n#!/bin/bash\nif\n[\n-z\n\"\n$HUGGING_FACE_HUB_TOKEN\n\"\n]\nthen\necho\n\"HUGGING_FACE_HUB_TOKEN is not set. Please set it before running this script.\"\nexit\n1\nfi\nmodel\n=\n\"TheBloke/Llama-2-7B-GPTQ\"\nvolume\n=\n$PWD\n/data\ndocker\nrun\n--gpus\nall\n\\\n-e\nHUGGING_FACE_HUB_TOKEN=\n$HUGGING_FACE_HUB_TOKEN\n\\\n-e\nGPTQ_BITS=4\n-e\nGPTQ_GROUPSIZE=128\n\\\n--shm-size\n5g\n-p\n8081:80\n\\\n-v\n$volume\n:/data ghcr.io/huggingface/text-generation-inference\n\\\n--max-best-of\n1\n\"\n$@\n\"\nWe can then run the server with this command:\nbash\nstart_server.sh\n--model-id\n\"meta-llama/Llama-2-7b-hf\"\nNote\nHelp\nYou can see all the options for the TGI container with the help flag like so:\ndocker\nrun ghcr.io/huggingface/text-generation-inference\n--help\n|\nless\nQuantization\nQuantization was very difficult to get working. There is a\n—quantize\nflag with accepts\nbitsandbytes\nand\ngptq\n. The\nbitsandbytes\napproach makes inference\nmuch\nslower, which\nothers have reported\n.\nTo make\ngptq\nwork for llama v2 models requires a bunch of work, you have to\ninstall the text-generation-server\nwhich can take a while and is very brittle to get right. I had to step through the\nMakefile\ncarefully. After that you have to download the weights with:\ntext-generation-server\ndownload-weights meta-llama/Llama-2-7b-hf\nYou can run the following command to perform the quantization (the last argument is the destination directory where the weights are stored).\ntext-generation-server\nquantize\n\"meta-llama/Llama-2-7b-hf\"\ndata/quantized/\nHowever, this step is not needed for the most popular models, as someone will likely already have quantized and uploaded them to the Hub.\nPre-Quantized Models\nAlternatively, you can use a pre-quantized model that has been uploaded to the Hub.\nTheBloke/Llama-2-7B-GPTQ\nis a good example of one. To get this to work, you have to be careful to set the\nGPTQ_BITS\nand\nGPTQ_GROUPSIZE\nenvironment variables to match the config. For example\nThis config\nnecessitates setting\nGPTQ_BITS=4\nand\nGPTQ_GROUPSIZE=128\nThese are already set in\nstart_server.sh\nshown above.\nThis PR\nwill eventually fix that.\nTo use the\nTheBloke/Llama-2-7B-GPTQ\nwith TGI, I can use the same bash script with the following arguments:\nbash\nstart_server.sh\n--model-id\nTheBloke/Llama-2-7B-GPTQ\n--quantize\ngptq\nComparison Without TGI Server\nWhen I first drafted this study I got the following response on twitter:\nBased on your code (\nhttps://t.co/hSYaPTsEaK\n) it seems like you measure the full HTTP request, which is like comparing trees to an apple.\n— Philipp Schmid (\n@_philschmid\n)\nJuly 29, 2023\nPhillip certainly has a point! I am indeed testing both! I’m looking for big differences in tools here, and since some inference servers have optimization tools, and some optimization tools do not have an inference server I cannot do a true apples to apples comparison. However, I think its still useful to try different things as advertised to see what is possible, and also take note of really significant gaps in latency between tools.\nTherefore, I ran the following tests to perform the similar optimizations as TGI, but without the server to see what happened:\nHuggingFace Transformers\nI was able to get slightly better performance without the TGI server as predicted by Phillip,\nbut it did not account for the the massive gap between some tools\n(which is exactly the kind of thing I was looking for).\nTo benchmark quantization with bitsandbytes, I\nfollowed this blog post\nand wrote\nthis benchmarking code\n. I quantized the model by loading it like this:\nmodel_id\n=\n\"meta-llama/Llama-2-7b-hf\"\ntokenizer\n=\nAutoTokenizer.from_pretrained(model_id)\nnf4_config\n=\nBitsAndBytesConfig(\nload_in_4bit\n=\nTrue\n,\nbnb_4bit_quant_type\n=\n\"nf4\"\n,\nbnb_4bit_compute_dtype\n=\ntorch.bfloat16\n)\nmodel_nf4\n=\nAutoModelForCausalLM.from_pretrained(model_id, quantization_config\n=\nnf4_config)\nUnlike TGI, I was able to get bitsandbytes to work properly here, but just like TGI it didn’t speed anything up for me with respect to inference latency. As reflected in the benchmark table, I got nearly the same results with transformers\nwithout any optimizations\n.\nGPTQ\nI also quantized the model using\nAutoGPTQ\nwithout an inference server to compare against TGI. The code for that is\nhere\n.\nThe results were so bad ~ 5 tok/sec that I decided not to put this in the table, because it seemed quite off to me.\nText Generation WebUI\nAman\nlet me know about\ntext-generation-web-ui\n, and also\nthese instructions\nfor quickly experimenting with\nExLlama\nand\nggml\n. I wasn’t able to get the\nggml\nvariant to work properly, unfortunately. If you are really serious about using exllama, I recommend trying to use it without the text generation UI and look at the\nexllama\nrepo, specifically at\ntest_benchmark_inference.py\n. (I didn’t have time for this, but if I was going to use exllama for anything serious I would go this route).\nFrom the root of the\ntext-generation-web-ui\nrepo, you can run the following commands to start an inference server optimized with\nExLlama\n:\npython3\ndownload-model.py TheBloke/Llama-2-7B-GPTQ\npython3\nserver.py\n--listen\n--extensions\nopenai\n--loader\nexllama_hf\n--model\nTheBloke_Llama-2-7B-GPTQ\nAfter the server was started, I used\nthis code\nto conduct the benchmark.\nOverall, I didn’t like this particular piece of software much. It’s bit bloated because its trying to do too many things at once (An inference server, Web UIs, and other optimizations). That being said, the documentation is good and it is easy to use.\nI don’t think there is any particular reason to use this unless you want an end-to-end solution that also comes with a web user-interface (which many people want!).\nvLLM\nvLLM\nonly works with CUDA 11.8, which I configured using\nthis approach\n. After configuring CUDA and installing the right version of PyTorch, you need to install the bleeding edge from git:\npip\ninstall\n-U\ngit+https://github.com/vllm-project/vllm.git\nA good recipe to use for vLLM can be find on\nthese Modal docs\n. Surprisingly, I had much lower latency when running on a local\nA6000\nvs. a hosted\nA100\non Modal Labs. It’s possible that I did something wrong here. Currently,\nvLLM\nis the fastest solution for when you need distributed inference (i.e. when your model doesn’t fit on a single GPU).\n.\nvLLM\noffers a server\n, but I benchmarked the model locally using their tools instead. The code for the benchmarking can be\nfound here\n:\nfrom\nvllm\nimport\nSamplingParams, LLM\n#from https://modal.com/docs/guide/ex/vllm_inference\nquestions\n=\n[\n# Coding questions\n\"Implement a Python function to compute the Fibonacci numbers.\"\n,\n\"Write a Rust function that performs binary exponentiation.\"\n,\n\"What are the differences between Javascript and Python?\"\n,\n# Literature\n\"Write a story in the style of James Joyce about a trip to the Australian outback in 2083, to see robots in the beautiful desert.\"\n,\n\"Who does Harry turn into a balloon?\"\n,\n\"Write a tale about a time-traveling historian who's determined to witness the most significant events in human history.\"\n,\n# Math\n\"What is the product of 9 and 8?\"\n,\n\"If a train travels 120 kilometers in 2 hours, what is its average speed?\"\n,\n\"Think through this step by step. If the sequence a_n is defined by a_1 = 3, a_2 = 5, and a_n = a_(n-1) + a_(n-2) for n > 2, find a_6.\"\n,\n]\nMODEL_DIR\n=\n\"/home/ubuntu/hamel-drive/vllm-models\"\ndef\ndownload_model_to_folder():\nfrom\nhuggingface_hub\nimport\nsnapshot_download\nimport\nos\nsnapshot_download(\n\"meta-llama/Llama-2-7b-hf\"\n,\nlocal_dir\n=\nMODEL_DIR,\ntoken\n=\nos.environ[\n\"HUGGING_FACE_HUB_TOKEN\"\n],\n)\nreturn\nLLM(MODEL_DIR)\ndef\ngenerate(question, llm, note\n=\nNone\n):\nresponse\n=\n{\n'question'\n: question,\n'note'\n: note}\nsampling_params\n=\nSamplingParams(\ntemperature\n=\n1.0\n,\ntop_p\n=\n1\n,\nmax_tokens\n=\n200\n,\n)\nstart\n=\ntime.perf_counter()\nresult\n=\nllm.generate(question, sampling_params)\nrequest_time\n=\ntime.perf_counter()\n-\nstart\nfor\noutput\nin\nresult:\nresponse[\n'tok_count'\n]\n=\nlen\n(output.outputs[\n0\n].token_ids)\nresponse[\n'time'\n]\n=\nrequest_time\nresponse[\n'answer'\n]\n=\noutput.outputs[\n0\n].text\nreturn\nresponse\nif\n__name__\n==\n'__main__'\n:\nllm\n=\ndownload_model_to_folder()\ncounter\n=\n1\nresponses\n=\n[]\nfor\nq\nin\nquestions:\nresponse\n=\ngenerate(question\n=\nq, llm\n=\nllm, note\n=\n'vLLM'\n)\nif\ncounter\n>=\n2\n:\nresponses.append(response)\ncounter\n+=\n1\ndf\n=\npd.DataFrame(responses)\ndf.to_csv(\n'bench-vllm.csv'\n, index\n=\nFalse\n)\nHuggingFace Inference Endpoint\nI deployed an\ninference endpoint\non HuggingFace for\nmeta-llama/Llama-2-7b-hf\n, on a\nNvidia A10G\nGPU. I didn’t try to turn on any optimizations like quantization and wanted to see what the default performance would be like.\nThe documentation for these interfaces can be found\nhere\n. There is also\na python client\n.\nTheir documentation says they are using TGI under the hood. However, my latency was significantly faster on their hosted inference platform than using TGI locally. This could be due to the fact that I used a\nA10G\nwith them but only a\nA6000\nlocally. It’s worth looking into why this discrepancy exists further.\nThe code for this benchmark can be found\nhere\n.\nFootnotes\nIt is common to explore the inference vs throughput frontier when conducting inference benchmarks. I did not do this, since I was most interested in latency.\nHere is an example\nof how to conduct inference benchmarks that consider both throughput and latency.\n↩︎\nFor\nLlama v2 models\n, you must be careful to use the models ending in\n-hf\nas those are the ones that are compatible with the transformers library.\n↩︎\nThe Modular Inference Engine\nis another example of an inference server that also applies optimization techniques. At the time of this writing, this is proprietary technology, but its worth keeping an eye on this in the future.\n↩︎",
  "metadata": {
    "title": "Optimizing latency – Hamel’s Blog",
    "description": "An exploration of ways to optimize on latency.",
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "Optimizing latency – Hamel’s Blog",
    "og:description": "An exploration of ways to optimize on latency.",
    "og:image": "https://hamel.dev/notes/llm/bench_dark5.png",
    "og:site_name": "Hamel’s Blog",
    "og:locale": "en_US",
    "og:image:height": "552",
    "og:image:width": "1185",
    "twitter:title": "Optimizing latency – Hamel’s Blog",
    "twitter:description": "An exploration of ways to optimize on latency.",
    "twitter:image": "https://hamel.dev/notes/llm/bench_dark5.png",
    "twitter:creator": "@HamelHusain",
    "twitter:site": "@HamelHusain",
    "twitter:card": "summary_large_image",
    "twitter:image-height": "552",
    "twitter:image-width": "1185"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/aef6398af80f/code/body-000.txt",
      "chars": 105,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-001.txt",
      "chars": 182,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-002.txt",
      "chars": 649,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-003.txt",
      "chars": 484,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-004.txt",
      "chars": 115,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-005.txt",
      "chars": 1137,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-006.txt",
      "chars": 17,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-007.txt",
      "chars": 436,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-008.txt",
      "chars": 58,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-009.txt",
      "chars": 70,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-010.txt",
      "chars": 64,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-011.txt",
      "chars": 74,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-012.txt",
      "chars": 72,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-013.txt",
      "chars": 308,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-014.txt",
      "chars": 150,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-015.txt",
      "chars": 59,
      "language": "text"
    },
    {
      "file": "attachments/aef6398af80f/code/body-016.txt",
      "chars": 2149,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://llm.mlc.ai/docs/get_started/quick_start",
      "label": "mlc",
      "type": "a"
    },
    {
      "href": "https://github.com/OpenNMT/CTranslate2",
      "label": "CTranslate2",
      "type": "a"
    },
    {
      "href": "https://docs.vllm.ai/en/latest/",
      "label": "vLLM",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/text-generation-inference",
      "label": "Text Generation Inference",
      "type": "a"
    },
    {
      "href": "https://opentelemetry.io/docs/concepts/signals/traces/",
      "label": "via OpenTelemetry",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/inference-endpoints",
      "label": "inference endpoints",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/text-generation-inference/commit/bde25e62b33b05113519e5dbf75abda06a03328e",
      "label": "restrictive that may interfere with certain commercial uses",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/meta-llama/Llama-2-7b-hf",
      "label": "meta-llama/Llama-2-7b-hf",
      "type": "a"
    },
    {
      "href": "https://github.com/vllm-project/vllm.git",
      "label": "vLLM",
      "type": "a"
    },
    {
      "href": "https://www.simonwenkel.com/notes/ai/practical/latency-vs-throughput-in-machine-learning-pipelines.html",
      "label": "trade-offs between throughput and latency",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/serving/index.html",
      "label": "serving section",
      "type": "a"
    },
    {
      "href": "https://github.com/vllm-project/vllm",
      "label": "vLLM",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/optimum/concept_guides/quantization",
      "label": "quantization",
      "type": "a"
    },
    {
      "href": "https://vllm.ai/",
      "label": "Paged Attention",
      "type": "a"
    },
    {
      "href": "https://github.com/turboderp/exllama",
      "label": "Exllama",
      "type": "a"
    },
    {
      "href": "https://llm.mlc.ai/docs/compilation/compile_models.html",
      "label": "these docs",
      "type": "a"
    },
    {
      "href": "https://llm.mlc.ai/docs/deploy/python_engine.html",
      "label": "their python client",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/llama-inference/blob/master/mlc/mlc.py",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://llm.mlc.ai/docs/deploy/mlc_chat_config.html",
      "label": "this system prompt",
      "type": "a"
    },
    {
      "href": "https://twitter.com/abacaj/status/1685107222097903617?s=20",
      "label": "Anton",
      "type": "a"
    },
    {
      "href": "https://opennmt.net/CTranslate2/guides/transformers.html",
      "label": "specific instructions for llama models",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/llama-inference/blob/master/ctranslate/bench.py",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/text-generation-inference/issues/309",
      "label": "others have reported",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/text-generation-inference/tree/main/server",
      "label": "install the text-generation-server",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/text-generation-inference/blob/main/server/Makefile",
      "label": "Makefile",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/TheBloke/Llama-2-7B-GPTQ",
      "label": "TheBloke/Llama-2-7B-GPTQ",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/TheBloke/Llama-2-7B-GPTQ/blob/main/quantize_config.json",
      "label": "This config",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/text-generation-inference/pull/671",
      "label": "This PR",
      "type": "a"
    },
    {
      "href": "https://t.co/hSYaPTsEaK",
      "label": "https://t.co/hSYaPTsEaK",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid/status/1685187971400470528?ref_src=twsrc%5Etfw",
      "label": "July 29, 2023",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/4bit-transformers-bitsandbytes",
      "label": "followed this blog post",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/llama-inference/blob/master/hf/bench.py",
      "label": "this benchmarking code",
      "type": "a"
    },
    {
      "href": "https://github.com/PanQiWei/AutoGPTQ",
      "label": "AutoGPTQ",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/llama-inference/blob/master/hf/bench-gptq.py",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://twitter.com/tmm1/status/1683255057201135616?s=20",
      "label": "Aman",
      "type": "a"
    },
    {
      "href": "https://github.com/oobabooga/text-generation-webui",
      "label": "text-generation-web-ui",
      "type": "a"
    },
    {
      "href": "https://github.com/paul-gauthier/aider/issues/110",
      "label": "these instructions",
      "type": "a"
    },
    {
      "href": "https://github.com/ggerganov/ggml",
      "label": "ggml",
      "type": "a"
    },
    {
      "href": "https://github.com/turboderp/exllama/tree/master",
      "label": "exllama",
      "type": "a"
    },
    {
      "href": "https://github.com/turboderp/exllama/blob/master/test_benchmark_inference.py",
      "label": "test_benchmark_inference.py",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/llama-inference/blob/master/exllama/bench.py",
      "label": "this code",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/cuda.html",
      "label": "this approach",
      "type": "a"
    },
    {
      "href": "https://modal.com/docs/guide/ex/vllm_inference",
      "label": "these Modal docs",
      "type": "a"
    },
    {
      "href": "https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html",
      "label": "offers a server",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/llama-inference/blob/master/vllm/bench.py",
      "label": "found here",
      "type": "a"
    },
    {
      "href": "https://ui.endpoints.huggingface.co/",
      "label": "inference endpoint",
      "type": "a"
    },
    {
      "href": "https://huggingface.github.io/text-generation-inference/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/huggingface_hub/package_reference/inference_client",
      "label": "a python client",
      "type": "a"
    },
    {
      "href": "https://github.com/hamelsmu/llama-inference/blob/master/hf-endpoint/bench.py",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/mosaicml/llm-foundry/tree/main/scripts/inference/benchmarking",
      "label": "Here is an example",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/meta-llama",
      "label": "Llama v2 models",
      "type": "a"
    },
    {
      "href": "https://docs.modular.com/max/",
      "label": "The Modular Inference Engine",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:45Z",
  "source_url": "https://hamel.dev/notes/llm/inference/inference.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 18764,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/aef6398af80f/pages/source.html",
  "local_html": "attachments/aef6398af80f/pages/article.html",
  "body_base_path": "attachments/aef6398af80f/pages/",
  "body_markdown_path": "attachments/aef6398af80f/pages/article.md"
}