{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Optimizing Transformers with Optimum",
  "keywords": "Optimum,优化,量化",
  "domain": "机器学习",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/optimizing-transformers-with-optimum",
  "publish_date": "2022-06",
  "corpus_stage": "推理部署",
  "notes": "用Optimum优化Transformers",
  "id": "ccb6479e163c",
  "collected_at": "2026-10-02T14:29:18",
  "title": "Optimizing Transformers with Hugging Face Optimum",
  "body_markdown": "Optimizing Transformers with Hugging Face Optimum\nJune 30, 2022\n9\nminute read\nView Code\nlast update: 2022-11-18\nIn this session, you will learn how to optimize Hugging Face Transformers models using Optimum. The session will show you how to dynamically quantize and optimize a DistilBERT model using\nHugging Face Optimum\nand\nONNX Runtime\n. Hugging Face Optimum is an extension of 🤗 Transformers, providing a set of performance optimization tools enabling maximum efficiency to train and run models on targeted hardware.\nNote: dynamic quantization is currently only supported for CPUs, so we will not be utilizing GPUs / CUDA in this session.\nBy the end of this session, you see how quantization and optimization with Hugging Face Optimum can result in significant increase in model latency while keeping almost 100% of the full-precision model. Furthermore, you’ll see how to easily apply some advanced quantization and optimization techniques shown here so that your models take much less of an accuracy hit than they would otherwise.\nYou will learn how to:\nSetup Development Environment\nConvert a Hugging Face\nTransformers\nmodel to ONNX for inference\nApply graph optimization techniques to the ONNX model\nApply dynamic quantization using\nORTQuantizer\nfrom Optimum\nTest inference with the quantized model\nEvaluate the performance and speed\nPush the quantized model to the Hub\nLoad and run inference with a quantized model from the hub\nLet's get started! 🚀\nThis tutorial was created and run on an c6i.xlarge AWS EC2 Instance.\n1. Setup Development Environment\nOur first step is to install Optimum, along with Evaluate and some other libraries. Running the following cell will install all the required packages for us including Transformers, PyTorch, and ONNX Runtime utilities:\nPython\n!\npip install\n\"optimum[onnxruntime]==1.5.0\"\nevaluate[evaluator] sklearn mkl-include mkl\n--\nupgrade\nIf you want to run inference on a GPU, you can install 🤗 Optimum with\npip install optimum[onnxruntime-gpu]\n.\n2. Convert a Hugging Face\nTransformers\nmodel to ONNX for inference\nBefore we can start qunatizing we need to convert our vanilla\ntransformers\nmodel to the\nonnx\nformat. To do this we will use the new\nORTModelForSequenceClassification\nclass calling the\nfrom_pretrained()\nmethod with the\nfrom_transformers\nattribute. The model we are using is the\noptimum/distilbert-base-uncased-finetuned-banking77\na fine-tuned DistilBERT model on the Banking77 dataset achieving an Accuracy score of\n92.5\nand as the feature (task)\ntext-classification\n.\nPython\nfrom\noptimum.onnxruntime\nimport\nORTModelForSequenceClassification\nfrom\ntransformers\nimport\nAutoTokenizer\nfrom\npathlib\nimport\nPath\nmodel_id=\n\"optimum/distilbert-base-uncased-finetuned-banking77\"\ndataset_id=\n\"banking77\"\nonnx_path =\nPath\n(\n\"onnx\"\n)\n# load vanilla transformers and convert to onnx\nmodel = ORTModelForSequenceClassification.\nfrom_pretrained\n(model_id,\nfrom_transformers\n=\nTrue\n)\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id)\n# save onnx checkpoint and tokenizer\nmodel.\nsave_pretrained\n(onnx_path)\ntokenizer.\nsave_pretrained\n(onnx_path)\nOne neat thing about 🤗 Optimum, is that allows you to run ONNX models with the\npipeline()\nfunction from 🤗 Transformers. This means that you get all the pre- and post-processing features for free, without needing to re-implement them for each model! Here's how you can run inference with our vanilla ONNX model:\nPython\nfrom\ntransformers\nimport\npipeline\nvanilla_clf =\npipeline\n(\n\"text-classification\"\n,\nmodel\n=model,\ntokenizer\n=tokenizer)\nvanilla_clf\n(\n\"Could you assist me in finding my lost card?\"\n)\n#[{'label': 'lost_or_stolen_card', 'score': 0.9664045572280884}]\nIf you want to learn more about exporting transformers model check-out\nConvert Transformers to ONNX with Hugging Face Optimum\nblog post\n3. Apply graph optimization techniques to the ONNX model\nGraph optimizations are essentially graph-level transformations, ranging from small graph simplifications and node eliminations to more complex node fusions and layout optimizations.\nExamples of graph optimizations include:\nConstant folding\n: evaluate constant expressions at compile time instead of runtime\nRedundant node elimination\n: remove redundant nodes without changing graph structure\nOperator fusion\n: merge one node (i.e. operator) into another so they can be executed together\n![operator fusion](\nIf you want to learn more about graph optimization you take a look at the\nONNX Runtime documentation\n. We are going to first optimize the model and then dynamically quantize to be able to use transformers specific operators such as QAttention for quantization of attention layers.\nTo apply graph optimizations to our ONNX model, we will use the\nORTOptimizer()\n. The\nORTOptimizer\nmakes it with the help of a\nOptimizationConfig\neasy to optimize. The\nOptimizationConfig\nis the configuration class handling all the ONNX Runtime optimization parameters.\nPython\nfrom\noptimum.onnxruntime\nimport\nORTOptimizer\nfrom\noptimum.onnxruntime.configuration\nimport\nOptimizationConfig\n# create ORTOptimizer and define optimization configuration\noptimizer = ORTOptimizer.\nfrom_pretrained\n(model)\noptimization_config =\nOptimizationConfig\n(\noptimization_level\n=\n99\n)\n# enable all optimizations\n# apply the optimization configuration to the model\noptimizer.\noptimize\n(\nsave_dir\n=onnx_path,\noptimization_config\n=optimization_config,\n)\nTo test performance we can use the ORTModelForSequenceClassification class again and provide an additional\nfile_name\nparameter to load our optimized model.\n(This also works for models available on the hub).\nPython\nfrom\ntransformers\nimport\npipeline\n# load optimized model\nmodel = ORTModelForSequenceClassification.\nfrom_pretrained\n(onnx_path,\nfile_name\n=\n\"model_optimized.onnx\"\n)\n# create optimized pipeline\noptimized_clf =\npipeline\n(\n\"text-classification\"\n,\nmodel\n=model,\ntokenizer\n=tokenizer)\noptimized_clf\n(\n\"Could you assist me in finding my lost card?\"\n)\n4. Apply dynamic quantization using\nORTQuantizer\nfrom Optimum\nAfter we have optimized our model we can accelerate it even more by quantizing it using the\nORTQuantizer\n. The\nORTQuantizer\ncan be used to apply dynamic quantization to decrease the size of the model size and accelerate latency and inference.\nWe use the\navx512_vnni\nconfig since the instance is powered by an intel ice-lake CPU supporting avx512.\nPython\nfrom\noptimum.onnxruntime\nimport\nORTQuantizer\nfrom\noptimum.onnxruntime.configuration\nimport\nAutoQuantizationConfig\n# create ORTQuantizer and define quantization configuration\ndynamic_quantizer = ORTQuantizer.\nfrom_pretrained\n(model)\ndqconfig = AutoQuantizationConfig.\navx512_vnni\n(\nis_static\n=\nFalse\n,\nper_channel\n=\nFalse\n)\n# apply the quantization configuration to the model\nmodel_quantized_path = dynamic_quantizer.\nquantize\n(\nsave_dir\n=onnx_path,\nquantization_config\n=dqconfig,\n)\nLets quickly check the new model size.\nPython\nimport\nos\n# get model file size\nsize = os.path.\ngetsize\n(onnx_path /\n\"model_optimized.onnx\"\n)/(\n1024\n*\n1024\n)\nquantized_model = os.path.\ngetsize\n(onnx_path /\n\"model_optimized_quantized.onnx\"\n)/(\n1024\n*\n1024\n)\nprint(\nf\n\"Model file size:\n{\nsize\n:.2f\n}\nMB\"\n)\nprint(\nf\n\"Quantized Model file size:\n{\nquantized_model\n:.2f\n}\nMB\"\n)\n5. Test inference with the quantized model\nOptimum\nhas built-in support for\ntransformers pipelines\n. This allows us to leverage the same API that we know from using PyTorch and TensorFlow models.\nTherefore we can load our quantized model with\nORTModelForSequenceClassification\nclass and transformers\npipeline\n.\nPython\nfrom\noptimum.onnxruntime\nimport\nORTModelForSequenceClassification\nfrom\ntransformers\nimport\npipeline, AutoTokenizer\nmodel = ORTModelForSequenceClassification.\nfrom_pretrained\n(onnx_path,\nfile_name\n=\n\"model_optimized_quantized.onnx\"\n)\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(onnx_path)\nq8_clf =\npipeline\n(\n\"text-classification\"\n,\nmodel\n=model,\ntokenizer\n=tokenizer)\nq8_clf\n(\n\"What is the exchange rate like on this app?\"\n)\n6. Evaluate the performance and speed\nWe can now leverage the map function of datasets to iterate over the validation set of squad 2 and run prediction for each data point. Therefore we write a evaluate helper method which uses our pipelines and applies some transformation to work with the squad v2 metric.\nPython\nfrom\nevaluate\nimport\nevaluator\nfrom\ndatasets\nimport\nload_dataset\neval =\nevaluator\n(\n\"text-classification\"\n)\neval_dataset =\nload_dataset\n(\n\"banking77\"\n,\nsplit\n=\n\"test\"\n)\nresults = eval.\ncompute\n(\nmodel_or_pipeline\n=q8_clf,\ndata\n=eval_dataset,\nmetric\n=\n\"accuracy\"\n,\ninput_column\n=\n\"text\"\n,\nlabel_column\n=\n\"label\"\n,\nlabel_mapping\n=model.config.label2id,\nstrategy\n=\n\"simple\"\n,\n)\nprint(results)\nPython\nprint(\nf\n\"Vanilla model: 92.5%\"\n)\nprint(\nf\n\"Quantized model:\n{\nresults[\n'accuracy'\n]*\n100\n:.2f\n}\n%\"\n)\nprint(\nf\n\"The quantized model achieves\n{\nround(results[\n'accuracy'\n]/\n0.925\n,\n4\n)*\n100\n:.2f\n}\n% accuracy of the fp32 model\"\n)\n# Vanilla model: 92.5%\n# Quantized model: 92.24%\n# The quantized model achieves 99.72% accuracy of the fp32 model\nOkay, now let's test the performance (latency) of our quantized model. We are going to use a payload with a sequence length of 128 for the benchmark. To keep it simple, we are going to use a python loop and calculate the avg,mean & p95 latency for our vanilla model and for the quantized model.\nPython\nfrom\ntime\nimport\nperf_counter\nimport\nnumpy\nas\nnp\npayload=\n\"Hello my name is Philipp. I am getting in touch with you because i didn't get a response from you. What do I need to do to get my new card which I have requested 2 weeks ago? Please help me and answer this email in the next 7 days. Best regards and have a nice weekend \"\n*\n2\nprint(\nf\n'Payload sequence length:\n{\nlen(\ntokenizer\n(payload)[\n\"input_ids\"\n])\n}\n'\n)\ndef\nmeasure_latency\n(\npipe\n):\nlatencies = []\n# warm up\nfor\n_\nin\nrange(\n10\n):\n_ =\npipe\n(payload)\n# Timed run\nfor\n_\nin\nrange(\n300\n):\nstart_time =\nperf_counter\n()\n_ =\npipe\n(payload)\nlatency =\nperf_counter\n() - start_time\nlatencies.\nappend\n(latency)\n# Compute run statistics\ntime_avg_ms =\n1000\n* np.\nmean\n(latencies)\ntime_std_ms =\n1000\n* np.\nstd\n(latencies)\ntime_p95_ms =\n1000\n* np.\npercentile\n(latencies,\n95\n)\nreturn\nf\n\"P95 latency (ms) -\n{\ntime_p95_ms\n}\n; Average latency (ms) -\n{\ntime_avg_ms\n:.2f\n}\n+\\-\n{\ntime_std_ms\n:.2f\n}\n;\"\n, time_p95_ms\nvanilla_model=\nmeasure_latency\n(vanilla_clf)\nquantized_model=\nmeasure_latency\n(q8_clf)\nprint(\nf\n\"Vanilla model:\n{\nvanilla_model[\n0\n]\n}\n\"\n)\nprint(\nf\n\"Quantized model:\n{\nquantized_model[\n0\n]\n}\n\"\n)\nprint(\nf\n\"Improvement through quantization:\n{\nround(vanilla_model[\n1\n]/quantized_model[\n1\n],\n2\n)\n}\nx\"\n)\n# Payload sequence length: 128\n# Vanilla model: P95 latency (ms) - 68.4711932000539; Average latency (ms) - 56.28 +\\- 6.66;\n# Quantized model: P95 latency (ms) - 27.554391949979617; Average latency (ms) - 27.29 +\\- 0.15;\n# Improvement through quantization: 2.48x\nWe managed to accelerate our model latency from 68.4ms to 27.55ms or 2.48x while keeping 99.72% of the accuracy.\n7. Push the quantized model to the Hub\nThe Optimum model classes like\nORTModelForSequenceClassification\nare integrated with the Hugging Face Model Hub, which means you can not only load model from the Hub, but also push your models to the Hub with\npush_to_hub()\nmethod. That way we can now save our qunatized model on the Hub to be for example used inside our inference API.\nWe have to make sure that we are also saving the\ntokenizer\nas well as the\nconfig.json\nto have a good inference experience.\nIf you haven't logged into the\nhuggingface hub\nyet you can use the\nnotebook_login\nto do so.\nPython\nfrom\nhuggingface_hub\nimport\nnotebook_login\nnotebook_login\n()\nAfter we have configured our hugging face hub credentials we can push the model.\nPython\nfrom\ntransformers\nimport\nAutoTokenizer\nfrom\noptimum.onnxruntime\nimport\nORTModelForSequenceClassification\ntmp_store_directory=\n\"onnx_hub_repo\"\nrepository_id=\n\"distilbert-onnx-banking77\"\nmodel.\nsave_pretrained\n(tmp_store_directory)\ntokenizer.\nsave_pretrained\n(tmp_store_directory)\nmodel.\npush_to_hub\n(tmp_store_directory,\nrepository_id\n=repository_id,\nuse_auth_token\n=\nTrue\n)\n8. Load and run inference with a quantized model from the hub\nThis step serves as a demonstration of how you could use optimum in your api to load and use our qunatized model.\nPython\nfrom\noptimum.onnxruntime\nimport\nORTModelForSequenceClassification\nfrom\ntransformers\nimport\npipeline, AutoTokenizer\nmodel = ORTModelForSequenceClassification.\nfrom_pretrained\n(\n\"philschmid/distilbert-onnx-banking77\"\n)\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(\n\"philschmid/distilbert-onnx-banking77\"\n)\nremote_clx =\npipeline\n(\n\"text-classification\"\n,\nmodel\n=model,\ntokenizer\n=tokenizer)\nremote_clx\n(\n\"What is the exchange rate like on this app?\"\n)\nConclusion\nWe successfully quantized our vanilla Transformers model with Hugging Face Optimum and managed to decrease our model latency from 68.4ms to 27.55ms or 2.48x while keeping 99.72% of the accuracy.\nThanks for reading. If you have any questions, feel free to contact me, through\nGithub\n, or on the\nforum\n. You can also connect with me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Optimizing Transformers with Hugging Face Optimum",
    "description": "Learn how to optimize Hugging Face Transformers models using Optimum. The session will show you how to dynamically quantize and optimize a DistilBERT model using Hugging Face Optimum and ONNX Runtime. Hugging Face Optimum is an extension of 🤗 Transformers, providing a set of performance optimization tools enabling maximum efficiency to train and run models on targeted hardware.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Optimizing Transformers with Hugging Face Optimum",
    "og:description": "Learn how to optimize Hugging Face Transformers models using Optimum. The session will show you how to dynamically quantize and optimize a DistilBERT model using Hugging Face Optimum and ONNX Runtime. Hugging Face Optimum is an extension of 🤗 Transformers, providing a set of performance optimization tools enabling maximum efficiency to train and run models on targeted hardware.",
    "og:url": "https://www.philschmid.de/optimizing-transformers-with-optimum",
    "og:image": "https://www.philschmid.de/api/og?title=Optimizing+Transformers+with+Hugging+Face+Optimum",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Optimizing Transformers with Hugging Face Optimum",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Optimizing Transformers with Hugging Face Optimum",
    "twitter:description": "Learn how to optimize Hugging Face Transformers models using Optimum. The session will show you how to dynamically quantize and optimize a DistilBERT model using Hugging Face Optimum and ONNX Runtime. Hugging Face Optimum is an extension of 🤗 Transformers, providing a set of performance optimization tools enabling maximum efficiency to train and run models on targeted hardware.",
    "twitter:image": "https://www.philschmid.de/api/og?title=Optimizing+Transformers+with+Hugging+Face+Optimum",
    "article:published_time": "2022-06-30"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/ccb6479e163c/code/body-000.txt",
      "chars": 96,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-001.txt",
      "chars": 547,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-002.txt",
      "chars": 240,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-003.txt",
      "chars": 455,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-004.txt",
      "chars": 335,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-005.txt",
      "chars": 478,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-006.txt",
      "chars": 301,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-007.txt",
      "chars": 413,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-008.txt",
      "chars": 395,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-009.txt",
      "chars": 316,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-010.txt",
      "chars": 1534,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-011.txt",
      "chars": 61,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-012.txt",
      "chars": 422,
      "language": "text"
    },
    {
      "file": "attachments/ccb6479e163c/code/body-013.txt",
      "chars": 436,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/optimizing-transformers-with-optimum/operator_fusion.png",
      "bytes": 22910,
      "reused": false,
      "local_path": "attachments/ccb6479e163c/images/body-322ae103afd4ce4aef65.png",
      "src": "https://www.philschmid.de/static/blog/optimizing-transformers-with-optimum/operator_fusion.png",
      "alt": "performance",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/optimizing-transformers-with-optimum/performance.png",
      "bytes": 44649,
      "reused": false,
      "local_path": "attachments/ccb6479e163c/images/body-5ff891c9a274b1e8f98b.png",
      "src": "https://www.philschmid.de/static/blog/optimizing-transformers-with-optimum/performance.png",
      "alt": "performance",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/optimum-transformers-optimizations/blob/master/notebook.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/optimum/index",
      "label": "Hugging Face Optimum",
      "type": "a"
    },
    {
      "href": "https://onnxruntime.ai/",
      "label": "ONNX Runtime",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/optimum/main/en/onnxruntime/modeling_ort",
      "label": "ORTModelForSequenceClassification",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/optimum/distilbert-base-uncased-finetuned-banking77",
      "label": "optimum/distilbert-base-uncased-finetuned-banking77",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/convert-transformers-to-onnx",
      "label": "Convert Transformers to ONNX with Hugging Face Optimum",
      "type": "a"
    },
    {
      "href": "https://onnxruntime.ai/docs/performance/graph-optimizations.html",
      "label": "ONNX Runtime documentation",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/optimum/main/en/pipelines",
      "label": "Optimum",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/transformers/main/en/main_classes/pipelines",
      "label": "transformers pipelines",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/optimum",
      "label": "Github",
      "type": "a"
    },
    {
      "href": "https://discuss.huggingface.co/",
      "label": "forum",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:53Z",
  "source_url": "https://www.philschmid.de/optimizing-transformers-with-optimum",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 13043,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ccb6479e163c/pages/source.html",
  "local_html": "attachments/ccb6479e163c/pages/article.html",
  "body_base_path": "attachments/ccb6479e163c/pages/",
  "body_markdown_path": "attachments/ccb6479e163c/pages/article.md"
}