{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Chaim Rand",
  "title_listed": "Optimizing PyTorch Model Inference on CPU",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/optimizing-pytorch-model-inference-on-cpu/",
  "publish_date": "2025-12-08",
  "corpus_stage": "",
  "notes": "",
  "id": "138694355ace",
  "collected_at": "2026-10-02T11:08:23",
  "title": "Optimizing PyTorch Model Inference on CPU | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Optimizing PyTorch Model Inference on CPU\nFlyin’ Like a Lion on Intel Xeon\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nDecember 8, 202517 min read\n![](../images/000.jpg)\n[Photo](https://unsplash.com/photos/brown-lion-with-silver-chain-link-necklace-3tkxfe2GocY) by Ingo Stiller on Unsplash\nAs reliance on AI models grows, so does the criticality of optimizing their runtime performance. While the degree to which AI models will outperform human intelligence remains a heated topic of debate, their need for powerful and expensive compute resources is unquestionable — and even notorious.\nIn [previous posts](https://towardsdatascience.com/author/chaimrand/), we covered the topic of AI model optimization — primarily in the context of model training — and demonstrated how it can have a decisive impact on the cost and speed of AI model development. In this post, we focus our attention on AI model inference, where model optimization has an additional objective: To minimize the latency of inference requests and improve the user experience of the model consumer.\nIn this post, we will assume that the platform on which model inference is performed is a 4th Gen Intel® Xeon® Scalable CPU processor, more specifically, an Amazon EC2 [c7i.xlarge](https://aws.amazon.com/ec2/instance-types/c7i/) instance (with 4 Intel Xeon vCPUs) running a dedicated [Deep Learning Ubuntu (22.04) AMI](https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-x86-gpu-pytorch-2.7-ubuntu-22-04.html) and a CPU build of [PyTorch 2.8.0](https://pytorch.org/get-started/locally/). Of course, the choice of a model deployment platform is one of the many important decisions taken when designing an AI solution along with the choice of model architecture, development framework, training accelerator, data format, deployment strategy, etc. — each one of which must be taken with consideration of the associated costs and runtime speed. The choice of a CPU processor for running model inference may seem surprising in an era in which the number of dedicated AI inference accelerators is continuously growing. However, as we will see, there are some occasions when the best (and cheapest) option may very well be just a good old-fashioned CPU.\nWe will introduce a toy image-classification model and proceed to demonstrate some of the optimization opportunities for AI model inference on an Intel® Xeon® CPU. The deployment of an AI model typically includes a full inference server solution, but for the sake of simplicity, we will limit our discussion to just the model’s core execution. For a primer on model inference serving, please see our previous post: [The Case for Centralized AI Model Inference Serving](https://towardsdatascience.com/the-case-for-centralized-ai-model-inference-serving/).\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\nOur intention in this post is to demonstrate that: 1) a few simple optimization techniques can result in meaningful performance gains and 2) that reaching such results does not require specialized expertise in performance analyzers (such as [Intel® VTune™ Profiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html)) or on the inner workings of the low-level compute kernels. Importantly, the process of AI model optimization can differ considerably based on the model architecture and runtime environment. Optimizing for training will differ from optimizing for inference. Optimizing a transformer model will differ from optimizing a CNN model. Optimizing a 22-billion-parameter model will differ from optimizing a 100-million parameter model. Optimizing a model to run on a GPU will differ from optimizing it for a CPU. Even different generations of the same CPU family may have different computation components and, consequently, different optimization techniques. While the high-level steps for optimizing a given model on a given instance are pretty standard, the specific course it will take and the end result can vary greatly based on the project at hand.\nThe code snippets we will share are intended for demonstrative purposes. Please do not rely on their accuracy or their optimality. Please do not interpret our mention of any tool or technique as an endorsement for its use. Ultimately, the best design choices for your use case will greatly depend on the details of your project and, given the extent of the potential impact on performance, should be evaluated with the appropriate time and attention.\n## Why CPU?\nWith the ever-increasing number of hardware solutions for executing AI/ML model inference, our choice of a CPU may seem surprising. In this section, we describe some scenarios in which CPU may be the preferred platform for inference.\n  1. **Accessibility** : The use of dedicated AI accelerators — such as GPUs — typically requires dedicated deployment and maintenance or, alternatively, access to such instances on a cloud service platform. CPUs, on the other hand, are everywhere. Designing a solution to run on a CPU provides much greater flexibility and increases the opportunities for deployment.\n  2. **Availability** : Even if your algorithm can access an AI accelerator, there is the question of availability. AI accelerators are in extremely high demand, and even if/when you are able to acquire one, whether it be on-prem or in the cloud, you may choose to prioritize them for tasks that are even more resource intensive, such as AI model training.\n  3. **Reduced Latency** : There are many situations in which your AI model is just one component in a pipeline of software algorithms running on a standard CPU. While the AI model may perform significantly faster on an AI accelerator, when taking into account the time required to send an inference request over the network, it is quite possible that running it on the same CPU will be faster.\n  4. **Underuse of Accelerator** : AI accelerators are typically quite expensive. To justify their cost, your goal should be to keep them fully occupied, minimizing their idle time. In some cases, the inference load will not justify the cost of an expensive AI accelerator.\n  5. **Model Architecture** : These days, we tend to automatically assume that AI models will perform significantly better on AI accelerators than on CPUs. And while more often than not, this is indeed the case, your model may include layers that perform better on CPU. For example, sequential algorithms such as Non-Maximum Suppression (NMS) and the Hungarian matching algorithm tend to perform better on CPU than GPU and are often offloaded onto the CPU even if a GPU is available (e.g., see [here](https://github.com/pytorch/vision/blob/v0.19.1/torchvision/csrc/ops/cuda/nms_kernel.cu)). If your model contains many such layers, running it on a CPU might not be such a bad option.\n\n\n### Why Intel Xeon?\n[Intel® Xeon® Scalable CPU](https://www.intel.com/content/www/us/en/products/details/processors/xeon/scalable.html) processors come with built-in accelerators for the matrix and convolution operators that are common in typical AI/ML workloads. These include [AVX-512](https://www.intel.com/content/www/us/en/architecture-and-technology/avx-512-overview.html) (introduced in Gen1), the [VNNI extension](https://www.intel.com/content/www/us/en/developer/articles/guide/deep-learning-with-avx512-and-dl-boost.html) (Gen2), and [AMX](https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/what-is-intel-amx.html) (Gen4). The AMX engine, in particular, includes specialized hardware instructions for executing AI models using bfloat16 and int8 precision data types. The acceleration engines are tightly integrated with Intel’s optimized software stack, which includes [oneDNN](https://www.intel.com/content/www/us/en/developer/tools/oneapi/onednn.html), [OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html), and the [Intel Extension for PyTorch](https://www.intel.com/content/www/us/en/developer/articles/technical/accelerate-with-intel-extension-for-pytorch.html) (IPEX). These libraries utilize the dedicated Intel® Xeon® hardware capabilities to optimize model execution with minimal code changes.\nDespite the arguments made in this section, the choice of inference vehicle should be made after considering all options available and after assessing the opportunities for optimization on each one. In the next sections, we will introduce a toy experiment and explore some of the optimization opportunities on CPU.\n## Inference Experiment\nIn this section, we define a toy AI model inference experiment comprising a [Resnet50](https://docs.pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html) image classification model, a randomly generated input batch, and a simple benchmarking utility which we use to report the average number of input samples processed per second (SPS).\npython\n\n```\nimport torch, torchvision\nimport time\n\n\ndef get_model():\n    model = torchvision.models.resnet50()\n    model = model.eval()\n    return model\n\n\ndef get_input(batch_size):\n    batch = torch.randn(batch_size, 3, 224, 224)\n    return batch\n\n\ndef get_inference_fn(model):\n    def infer_fn(batch):\n        with torch.inference_mode():\n            output = model(batch)\n        return output\n    return infer_fn\n\n\ndef benchmark(infer_fn, batch):\n    # warm-up\n    for _ in range(10):\n        _ = infer_fn(batch)\n\n    iters = 100\n\n    start = time.time()\n    for _ in range(iters):\n        _ = infer_fn(batch)\n    end = time.time()\n\n    return (end - start) / iters\n\n\nbatch_size = 1\nmodel = get_model()\nbatch = get_input(batch_size)\ninfer_fn = get_inference_fn(model)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe baseline performance of our toy model is 22.76 samples per second (SPS).\n## Model Inference Optimization\nIn this section, we apply a number of optimizations to our toy experiment and assess their impact on runtime performance. Our focus will be on optimization techniques that can be applied with relative ease. While it is quite likely that additional performance gains can be achieved, these may require much greater specialization and a more significant time investment.\nOur focus will be on optimizations that do not change the model architecture; optimization techniques such as model distillation and model pruning are out of the context of this post. Also out of scope are methods for optimizing specific model components, e.g., by implementing custom PyTorch operators.\nIn a previous post we discussed AI model optimization on Intel XEON CPUs in the context of training workloads. In this section we will revisit some of the techniques mentioned there, this time in the context of AI model inference. We will complement these with optimization techniques that are unique to inference settings, including model compilation for inference, INT8 quantization, and multi-worker inference.\nThe order in which we present the optimization methods is not binding. In fact, some of the techniques are interdependent; for example, increasing the number of inference workers could impact the optimal choice of batch size.\n### Optimization 1: Batched Inference\nA common method for increasing resource utilization while reducing the average inference response time is to group input samples into batches. In real-world scenarios, we need to make sure to cap the batch size so that we meet the service level response time requirements, but for the purposes of our experiment we ignore this requirement. Experimenting with different batch sizes we find that a batch size of 8 results in a throughput of 26.28 SPS, 15% higher than the baseline result.\nNote that in the case that the shapes of the input samples vary, batching requires more handling (e.g., see [here](https://towardsdatascience.com/optimizing-transformer-models-for-variable-length-input-sequences-19fb88fddf71/)).\n### Optimization 2: Channels-Last Memory Format\nBy default in PyTorch, 4D tensors are stored in NCHW format, i.e., the four dimensions represent the batch size, channels, height, and width, respectively. However, the channels-last or NHWC format (i.e., batch size, height, width, and channels) exhibits better performance on CPU. Adjusting our inference script to apply the channels-last optimization is a simple matter of setting the memory format of both the model and the input to  _torch.channels_last_ as shown below:\npython\n\n```\ndef get_model(channels_last=False):\n    model = torchvision.models.resnet50()\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n    model = model.eval()\n    return model\n\ndef get_input(batch_size, channels_last=False):\n    batch = torch.randn(batch_size, 3, 224, 224)\n    if channels_last:\n        batch = batch.to(memory_format=torch.channels_last)\n    return batch\n\n\nbatch_size = 8\nmodel = get_model(channels_last=True)\nbatch = get_input(batch_size, channels_last=True)\ninfer_fn = get_inference_fn(model)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nApplying the  _channels-last_ memory optimization, results in a further boost of 25% in throughput.\nThe impact of this optimization is most noticeable on models that have many convolutional layers. It is not expected to make a noticeable impact on other model architectures (e.g., transformer models).\nPlease see the [PyTorch documentation](https://docs.pytorch.org/tutorials/intermediate/memory_format_tutorial.html) for more details on the memory format optimization and the [Intel documentation](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/nhwc.html) for details on how this is implemented internally in oneDNN.\n### Optimization 3: Automatic Mixed Precision\nModern Intel® Xeon® Scalable processors (from Gen3) include native support for the bfloat16 data type, a 16-bit floating point alternative to the standard float32. We can take advantage of this by applying PyTorch’s automatic mixed precision package, [torch.amp](https://pytorch.org/docs/stable/amp.html), as demonstrated below:\npython\n\n```\ndef get_inference_fn(model, enable_amp=False):\n    def infer_fn(batch):\n        with torch.inference_mode(), torch.amp.autocast(\n                'cpu',\n                dtype=torch.bfloat16,\n                enabled=enable_amp\n        ):\n            output = model(batch)\n        return output\n    return infer_fn\n\nbatch_size = 8\nmodel = get_model(channels_last=True)\nbatch = get_input(batch_size, channels_last=True)\ninfer_fn = get_inference_fn(model, enable_amp=True)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe result of applying mixed precision is a throughput of 86.95 samples per second, 2.6 times the previous experiment and 3.8 times the baseline result.\nNote that the use of a reduced precision floating point type can have an impact on numerical accuracy, and its effect on model quality performance must be evaluated.\n### Optimization 4: Memory Allocation Optimization\nTypical AI/ML workloads require the allocation and access of large blocks of memory. A number of optimization techniques are aimed at tuning the way memory is allocated and used during model execution. One common step is to replace the default system allocator (ptmalloc) with an alternative memory allocation libraries, such as [Jemalloc](https://github.com/jemalloc/jemalloc) and [TCMalloc](https://google.github.io/tcmalloc/overview.html), which have been shown to perform better on common AI/ML workloads (e.g., see [here](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html#choosing-an-optimized-memory-allocator)). To install [TCMalloc](https://google.github.io/tcmalloc/overview.html) run:\nbash\n\n```\nsudo apt-get install google-perftools\n```\n\nWe program its use via the  _LD_PRELOAD_ environment variable:\nbash\n\n```\nLD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 python main.py\n```\n\nThis optimization results in another significant performance boost: 117.54 SPS, 35% higher than our previous experiment!!\n### Optimization 5: Enable Huge Page Allocations\nBy default, the Linux kernel allocates memory in blocks of 4 KB, commonly referred to as pages. The mapping between the virtual and physical memory addresses is managed by the CPU’s Memory Management Unit (MMU), which uses a small hardware cache called the Translation Lookaside Buffer (TLB). The TLB is limited in the number entries it can hold. When you have  _many_ small pages (as in large neural network models), the number of TLB cache misses can climb quickly, increasing latency and slowing down the speed of the program. A common way to address this is to use “huge pages” — blocks of 2 MB (or 1 GB) per page. This reduces the number of TLB entries required, improving memory access efficiency and lowering allocation latency.\nbash\n\n```\nexport THP_MEM_ALLOC_ENABLE=1\n```\n\nIn the case of our model, the impact is negligible. However, this is an important optimization for many AI/ML workloads.\n#### Optimization 6: IPEX\n[Intel® Extension for PyTorch (IPEX)](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/) is a library extension for PyTorch with the latest performance optimizations for Intel hardware. To install it we run:\nbash\n\n```\npip install intel_extension_for_pytorch\n```\n\nIn the code block below, we demonstrate the basic use of the [ipex.optimize](https://intel.github.io/intel-extension-for-pytorch/latest/tutorials/api_doc.html) API.\npython\n\n```\nimport intel_extension_for_pytorch as ipex\n\ndef get_model(channels_last=False, ipex_optimize=False):\n    model = torchvision.models.resnet50()\n\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n\n    model = model.eval()\n\n    if ipex_optimize:\n        model = ipex.optimize(model, dtype=torch.bfloat16)\n\n    return model\n```\n\nThe resultant throughout is 159.31 SPS, for another 36% performance boost.\nPlease see the official documentation for more details on the many optimizations that IPEX has to offer.\n### Optimization 7: Model Compilation\nAnother popular PyTorch optimization is torch.compile. Introduced in PyTorch 2.0, this just-in-time (JIT) compilation feature, performs kernel fusion and other optimizations. In a previous post we covered PyTorch compilation in great detail, covering some its many features, controls, and limitations. Here we demonstrate its basic use:\npython\n\n```\ndef get_model(channels_last=False, ipex_optimize=False, compile=False):\n    model = torchvision.models.resnet50()\n\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n\n    model = model.eval()\n\n    if ipex_optimize:\n        model = ipex.optimize(model, dtype=torch.bfloat16)\n\n    if compile:\n        model = torch.compile(model)\n\n    return model\n```\n\nApplying torch.compile on the IPEX-optimized model results in a throughput of 144.5 SPS, which is lower than our previous experiment. In the case of our model, IPEX and torch.compile do not coexist well. When applying just the torch.compile the throughput is 133.36 SPS.\nThe general takeaway from this experiment is that, for a given model, any two optimization techniques could interfere with one another. This necessitates evaluating the impact of multiple configurations on the runtime performance of a given model in order to find the best one.\n### Optimization 8: Auto-tune Environment Setup With `**torch.xeon.run_cpu**`\nThere are a number of environment settings that control thread and memory management and can be used to further fine-tune the runtime performance of an AI/ML workload. Rather than setting these manually, PyTorch offers the [torch.xeon.run_cpu](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html) script that does this automatically. In preparation for the use of this script, we install Intel’s threading and multiprocessing libraries, [one TBB](https://www.intel.com/content/www/us/en/developer/tools/oneapi/onetbb.html) and [Intel OpenMP](https://www.intel.com/content/www/us/en/docs/oneapi/installation-guide-linux/2023-0/pip.html). We also add a symbolic link to our [TCMalloc](https://google.github.io/tcmalloc/overview.html) installation.\nbash\n\n```\n# install TBB\nsudo apt install -y libtbb12\n# install openMP\npip install intel-openmp\n# link to tcmalloc\nsudo ln -sf /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so\n```\n\nIn the case of our toy model, using torch.xeon.run_cpu increases the throughput to 162.15 SPS — a slight increase over our previous maximum of 159.31 SPS.\nPlease see the [PyTorch documentation](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html) for more features of the `torch.xeon.run_cpu` and more details on the environment variables it applies.\n### Optimization 9: Multi-worker Inference\nAnother popular technique for increasing resource utilization and scale is to load multiple instances of the AI model and run them in parallel in separate processes. Although this technique is more commonly applied on machines with many CPUs (separated into multiple [NUMA nodes](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html#applying-numa-access-control)) — not on our small 4-vCPU instance — we include it here for the sake of demonstration. In the script below we run 2 instances of our model in parallel:\nbash\n\n```\npython -m torch.backends.xeon.run_cpu --ninstances 2 main.py\n```\n\nThis results in a throughput of 169.4 SPS — additional modest but meaningful 4% increase.\n### Optimization 10: INT8 Quantization\nINT8 quantization is another common technique for accelerating AI model inference execution. In INT8 quantization, the floating point datatypes of the model weights and activations are replaced by 8-bit integers. Intel’s Xeon processors include dedicated accelerators for processing INT8 operations (e.g., see [here](https://pytorch.org/blog/int8-quantization/)). INT8 quantization can result in a meaningful increase in speed and a lower memory footprint. Importantly, the reduced bit-precision can have a significant impact on the quality of the model output. There are many different approaches to INT8 quantization some of which include calibration or retraining. There are also a wide variety of tools and libraries for applying quantization. A full discussion on the topic of quantization is beyond the scope of this post. \nSince in this post we are interested just in the potential performance impact, we demonstrate one quantization scheme using [TorchAO](https://github.com/pytorch/ao), without consideration of the impact on model quality. In the code block below, we implement [PyTorch 2 Export Quantization with X86 Backend through Inductor](https://docs.pytorch.org/ao/stable/tutorials_source/pt2e_quant_x86_inductor.html). INT8 quantization is another common technique for accelerating AI model inference execution. In INT8 quantization, the floating point datatypes of the model weights and activations are replaced by 8-bit integers. Intel’s Xeon processors include dedicated accelerators for processing INT8 operations (e.g., see [here](https://pytorch.org/blog/int8-quantization/)). INT8 quantization can result in a meaningful increase in speed and a lower memory footprint.\nImportantly, the reduced bit-precision can have a significant impact on the quality of the model output. There are many different approaches to INT8 quantization some of which include calibration or retraining. There are also a wide variety of tools and libraries for applying quantization. A full discussion on the topic of quantization is beyond the scope of this post. Since in this post we are interested just in the potential performance impact, we demonstrate one quantization scheme using [TorchAO](https://github.com/pytorch/ao), without consideration of the impact on model quality. In the code block below, we implement [PyTorch 2 Export Quantization with X86 Backend through Inductor](https://docs.pytorch.org/ao/stable/tutorials_source/pt2e_quant_x86_inductor.html). Please see the documentation for the full details:\npython\n\n```\nfrom torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e, convert_pt2e\nimport torchao.quantization.pt2e.quantizer.x86_inductor_quantizer as xiq\n\ndef quantize_model(model):\n    x = torch.randn(4, 3, 224, 224).contiguous(\n                            memory_format=torch.channels_last)\n    example_inputs = (x,)\n    batch_dim = torch.export.Dim(\"batch\")\n    with torch.no_grad():\n        exported_model = torch.export.export(\n            model,\n            example_inputs,\n            dynamic_shapes=((batch_dim,\n                             torch.export.Dim.STATIC,\n                             torch.export.Dim.STATIC,\n                             torch.export.Dim.STATIC),\n                            )\n        ).module()\n    quantizer = xiq.X86InductorQuantizer()\n    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())\n    prepared_model = prepare_pt2e(exported_model, quantizer)\n    prepared_model(*example_inputs)\n    converted_model = convert_pt2e(prepared_model)\n    optimized_model = torch.compile(converted_model)\n    return optimized_model\n\n\nbatch_size = 8\nmodel = get_model(channels_last=True)\nmodel = quantize_model(model)\nbatch = get_input(batch_size, channels_last=True)\ninfer_fn = get_inference_fn(model, enable_amp=True)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThis results in a throughput of 172.67 SPS.\nPlease see [here](https://docs.pytorch.org/docs/2.9/quantization.html) for more details on quantization in PyTorch.\n### Optimization 11: Graph Compilation and Execution With ONNX\nThere are a number of third party libraries that specialize in compiling PyTorch models into graph representations and optimizing them for runtime performance on target inference devices. One of the most popular libraries for this is [Open Neural Network Exchange](https://onnx.ai/) (ONNX). ONNX performs ahead-of-time compilation of AI/ML models and executes them using a dedicated runtime library.\nWhile ONNX compilation support is included in PyTorch, we require the following library for executing an ONNX model:\nbash\n\n```\npip install onnxruntime\n```\n\nIn the code block below, we demonstrate ONNX compilation and model execution:\npython\n\n```\ndef export_to_onnx(model, onnx_path=\"resnet50.onnx\"):\n    dummy_input = torch.randn(4, 3, 224, 224)\n    batch = torch.export.Dim(\"batch\")\n    torch.onnx.export(\n        model,\n        dummy_input,\n        onnx_path,\n        input_names=[\"input\"],\n        output_names=[\"output\"],\n        dynamic_shapes=((batch,\n                         torch.export.Dim.STATIC,\n                         torch.export.Dim.STATIC,\n                         torch.export.Dim.STATIC),\n                        ),\n        dynamo=True\n    )\n    return onnx_path\n\ndef onnx_infer_fn(onnx_path):\n    import onnxruntime as ort\n\n    sess = ort.InferenceSession(\n        onnx_path,\n        providers=[\"CPUExecutionProvider\"]\n    )\n    input_name = sess.get_inputs()[0].name\n\n    def infer_fn(batch):\n        result = sess.run(None, {input_name: batch})\n        return result\n    return infer_fn\n\nbatch_size = 8\nmodel = get_model()\nonnx_path = export_to_onnx(model)\nbatch = get_input(batch_size).numpy()\ninfer_fn = onnx_infer_fn(onnx_path)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe resultant throughput is 44.92 SPS, far lower than in our previous experiments. In the case of our toy model, the ONNX runtime does not provide a benefit.\n### Optimization 12: Graph Compilation and Execution with OpenVINO\nAnother opensource toolkit aimed at deploying highly performant AI solutions is [OpenVINO](https://docs.openvino.ai/2025/index.html). [OpenVINO](https://pypi.org/project/openvino/) is [highly optimized](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) for model execution on Intel hardware — e.g., by fully leveraging the Intel AMX instructions. A common way to apply OpenVINO in PyTorch is to first convert the model to ONNX:\npython\n\n```\nfrom openvino import Core\n\ndef compile_openvino_model(onnx_path):\n    core = Core()\n    model = core.read_model(onnx_path)\n    compiled = core.compile_model(model, \"CPU\")\n    return compiled\n\ndef openvino_infer_fn(compiled_model):\n    def infer_fn(batch):\n        result = compiled_model([batch])[0]\n        return result\n    return infer_fn\n\nbatch_size = 8\nmodel = get_model()\nonnx_path = export_to_onnx(model)\novm = compile_openvino_model(onnx_path)\nbatch = get_input(batch_size).numpy()\ninfer_fn = openvino_infer_fn(ovm)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe result of this optimization is a throughput of 297.33 SPS, nearly twice as fast as our previous best experiment!!\nPlease see the official documentation for more details on OpenVINO.\n### Optimization 13: INT8 Quantization in OpenVINO with NNCF\nAs our final optimization, we revisit INT8 quantization, this time in the framework of [OpenVINO compilation](https://docs.openvino.ai/2025/openvino-workflow/model-optimization.html). As before, there are a number of methods for performing quantization — aimed at minimizing the impact on quality performance. Here we demonstrate the basic flow using the [NNCF](https://pypi.org/project/nncf/) library as documented [here](https://docs.openvino.ai/2025/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.html).\npython\n\n```\nclass RandomDataset(torch.utils.data.Dataset):\n\n    def __len__(self):\n        return 10000\n\n    def __getitem__(self, idx):\n        return torch.randn(3, 224, 224)\n\ndef nncf_quantize(onnx_path):\n    import nncf\n\n    core = Core()\n    onnx_model = core.read_model(onnx_path)\n    calibration_loader = torch.utils.data.DataLoader(RandomDataset())\n    input_name = onnx_model.inputs[0].get_any_name()\n    transform_fn = lambda data_item: {input_name: data_item.numpy()}\n    calibration_dataset = nncf.Dataset(calibration_loader, transform_fn)\n    quantized_model = nncf.quantize(onnx_model, calibration_dataset)\n    return core.compile_model(quantized_model, \"CPU\")\n\nbatch_size = 8\nmodel = get_model()\nonnx_path = export_to_onnx(model)\nq_model = nncf_quantize(onnx_path)\nbatch = get_input(batch_size).numpy()\ninfer_fn = openvino_infer_fn(q_model)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThis results in a throughput of 482.46(!!) SPS, another drastic improvement and over 18 times faster than our baseline experiment.\n### Results\nWe summarize the results of our experiments in the table below:\n![](../images/001.webp)\nResNet50 Inference Experiment (by Author)\nIn the case of our toy model, the optimizations steps we demonstrated resulted in huge performance gains. Importantly, the impact of each optimization can vary greatly based on the details of the model. You may find that some of these techniques do not apply to your model, or do not result in improved performance. For example, when we reapply the same sequence of optimizations to a Vision Transformer (ViT) model, the resultant performance boost is 8.41X — still significant, but less than the 18.36X of our experiment. Please see the appendix to this post for details.\nOur focus has been on runtime performance, but it is critical that you also evaluate the impact of each optimization on other metrics that are important to you — most importantly model quality.  \nThere are, undoubtedly, many more optimization techniques that can be applied; we have merely scratched the surface. Hopefully, the\n## Summary\nThis post continues our [series](https://towardsdatascience.com/author/chaimrand/) on the important topic of AI/ML model runtime performance analysis and optimization. Our focus in this post was on model inference on Intel® Xeon® CPU processors. Given the ubiquity and prevalence of CPUs, the ability to execute models on them in a reliable and performant manner, can be extremely compelling. As we have shown, by applying a number of relatively simple techniques, we can achieve considerable gains in model performance with profound implications on inference costs and inference latency.\nPlease do not hesitate to reach out with comments, questions, or corrections.\n## Appendix: Vision Transformer Optimization\nTo demonstrate how the impact of the runtime optimizations we discussed depend on the details of the AI/ML model, we reran our experiment on a Vision Transformer (ViT) model from the popular [timm](https://pypi.org/project/timm/) library:\npython\n\n```\nfrom timm.models.vision_transformer import VisionTransformer\n\ndef get_model(channels_last=False, ipex_optimize=False, compile=False):\n    model = VisionTransformer()\n\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n\n    model = model.eval()\n\n    if ipex_optimize:\n        model = ipex.optimize(model, dtype=torch.bfloat16)\n\n    if compile:\n        model = torch.compile(model)\n\n    return model\n```\n\nOne modification in this experiment was to apply [OpenVINO compilation directly to the PyTorch model](https://docs.openvino.ai/2025/openvino-workflow/model-preparation/convert-model-pytorch.html) rather than an intermediate ONNX model. This was due to the fact that OpenVINO compilation failed on the ViT ONNX model. The revised [NNCF quantization](https://docs.openvino.ai/2025/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.html) and OpenVINO compilation sequence is shown below:\npython\n\n```\nimport openvino as ov\nimport nncf\n\n\nbatch_size = 8\nmodel = get_model()\ncalibration_loader = torch.utils.data.DataLoader(RandomDataset())\ncalibration_dataset = nncf.Dataset(calibration_loader)\n\n# quantize PyTorch model\nmodel = nncf.quantize(model, calibration_dataset)\novm = ov.convert_model(model, example_input=torch.randn(1, 3, 224, 224))\novm = ov.compile_model(ovm)\nbatch = get_input(batch_size).numpy()\ninfer_fn = openvino_infer_fn(ovm)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe table below summarizes the results of the optimizations discussed in this post when applied to the ViT model:\n![](../images/body-0b2fcab64ce95814c3e6.webp)\nVision Transformer Inference Experiment (by Author)  \n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-97f5483d57a9b8741146.jpeg)\nWritten By\n## [Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[See all from Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Inference](https://towardsdatascience.com/tag/inference/)[Onnx](https://towardsdatascience.com/tag/onnx/)[Pytorch](https://towardsdatascience.com/tag/pytorch/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-pytorch-model-inference-on-cpu%2F&title=Optimizing%20PyTorch%20Model%20Inference%20on%20CPU)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-pytorch-model-inference-on-cpu%2F&title=Optimizing%20PyTorch%20Model%20Inference%20on%20CPU)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-pytorch-model-inference-on-cpu%2F&text=Optimizing%20PyTorch%20Model%20Inference%20on%20CPU)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/002.jpeg)](https://towardsdatascience.com/slowly-changing-dimensions-6a08dc0386ae/)\n## [Modeling Slowly Changing Dimensions](https://towardsdatascience.com/slowly-changing-dimensions-6a08dc0386ae/)\n[Programming](https://towardsdatascience.com/category/programming/)\nA deep dive into the various SCD types and how they can be implemented in Data Warehouses\n[Giorgos Myrianthous](https://towardsdatascience.com/author/gmyrianthous/)\nMay 3, 202414 min read\n  * [![](../images/003.jpeg)](https://towardsdatascience.com/siamese-networks-and-stuart-weitzman-boots-c414be7eff78/)\n## [Siamese Networks and Stuart Weitzman Boots](https://towardsdatascience.com/siamese-networks-and-stuart-weitzman-boots-c414be7eff78/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nPytorch based siamese CNN for shoe similarity and showing how to build an end to end similarity pipeline\n[Michael Sugimura](https://towardsdatascience.com/author/michaelsugimura/)\nOctober 15, 201810 min read\n  * [![](../images/004.webp)](https://towardsdatascience.com/your-models-mse-is-lying-to-you-part-ii/)\n## [Your Model's MSE Is Lying to You: Part II](https://towardsdatascience.com/your-models-mse-is-lying-to-you-part-ii/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAutoregressive rollout and uncertainty propagation. Second in a series on probabilistic forecasting for physical signals.\n[Waleed Esmail](https://towardsdatascience.com/author/waleed.esmail/)\nSeptember 25, 202620 min read\n  * [![](../images/005.png)](https://towardsdatascience.com/pendragon-four-multi-agent-reinforcement-learning-with-fate-grand-order-80f6254754dd/)\n## [Pendragon Four: Multi-Agent Reinforcement Learning with Fate Grand Order](https://towardsdatascience.com/pendragon-four-multi-agent-reinforcement-learning-with-fate-grand-order-80f6254754dd/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nMulti-agent reinforcement learning in a custom game environment to train 4 agents and have them play the mobile phone game Fate Grand Order\n[Michael Sugimura](https://towardsdatascience.com/author/michaelsugimura/)\nDecember 17, 201923 min read\n  * [![](../images/006.png)](https://towardsdatascience.com/finding-the-fastest-lane-at-border-crossings-using-machine-vision-2e8dec6e03a3/)\n## [Finding the Fastest Lane at Border Crossings Using Machine Vision](https://towardsdatascience.com/finding-the-fastest-lane-at-border-crossings-using-machine-vision-2e8dec6e03a3/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nOptimizing border crossing with object detection and tracking\n[Danilo Najkov](https://towardsdatascience.com/author/danilo.najkov/)\nFebruary 16, 202312 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/machine-learning-with-r-a-complete-guide-to-decision-trees-1d4dfc88bc20/)\n## [Machine Learning with R: A Complete Guide to Decision Trees](https://towardsdatascience.com/machine-learning-with-r-a-complete-guide-to-decision-trees-1d4dfc88bc20/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn easy and straightforward guide to machine learning and classification with decision trees.\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nFebruary 10, 20215 min read\n  * [![](../images/008.jpg)](https://towardsdatascience.com/dataloader-for-sequential-data-using-pytorch-deep-learning-framework-part-2-ed3ad5f6ad82/)\n## [Dataloader for sequential data using PyTorch deep learning framework - Part 2](https://towardsdatascience.com/dataloader-for-sequential-data-using-pytorch-deep-learning-framework-part-2-ed3ad5f6ad82/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nEnable yourself for writing dataloader for both image and sequential data by reading this blog\n[Harsh Maheshwari](https://towardsdatascience.com/author/anveenaik/)\nMay 8, 20217 min read\n  * [![](../images/009.jpg)](https://towardsdatascience.com/design-patterns-with-python-for-machine-learning-engineers-builder-45b8e749f134/)\n## [Design Patterns with Python for Machine Learning Engineers: Builder](https://towardsdatascience.com/design-patterns-with-python-for-machine-learning-engineers-builder-45b8e749f134/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLearn how to use the Builder design pattern to enhance your code\n[Marcello Politi](https://towardsdatascience.com/author/marcellopoliti/)\nOctober 12, 20236 min read\n  * [![](../images/010.png)](https://towardsdatascience.com/3-must-know-advanced-analytics-for-understanding-your-customers-f650f4ac7f57/)\n## [3 must-know advanced analytics for understanding your customers](https://towardsdatascience.com/3-must-know-advanced-analytics-for-understanding-your-customers-f650f4ac7f57/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTop Use-cases for customer analytics and how to do them\n[Pranay Dave](https://towardsdatascience.com/author/pranay-dave9/)\nNovember 18, 20206 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n",
  "metadata": {
    "title": "Optimizing PyTorch Model Inference on CPU | Towards Data Science",
    "description": "Flyin’ Like a Lion on Intel Xeon",
    "keywords": null,
    "author": "Chaim Rand",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Optimizing PyTorch Model Inference on CPU",
    "og:description": "Flyin’ Like a Lion on Intel Xeon",
    "og:url": "https://towardsdatascience.com/optimizing-pytorch-model-inference-on-cpu/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/ingo-stiller-3tkxfe2GocY-unsplash.jpg",
    "og:image:width": "1920",
    "og:image:height": "1080",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Chaim Rand",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "17 min read",
    "article:published_time": "2025-12-08T07:00:11.000Z",
    "article:modified_time": "2025-12-08T07:00:11.000Z",
    "article:author": "https://towardsdatascience.com/author/chaimrand/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_000.txt",
      "chars": 866
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_001.txt",
      "chars": 644
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_002.txt",
      "chars": 574
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_003.txt",
      "chars": 38
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_004.txt",
      "chars": 69
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_005.txt",
      "chars": 30
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_006.txt",
      "chars": 40
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_007.txt",
      "chars": 351
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_008.txt",
      "chars": 376
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_009.txt",
      "chars": 183
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_010.txt",
      "chars": 65
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_011.txt",
      "chars": 1371
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_012.txt",
      "chars": 24
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_013.txt",
      "chars": 1114
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_014.txt",
      "chars": 630
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_015.txt",
      "chars": 950
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_016.txt",
      "chars": 428
    },
    {
      "language": "txt",
      "file": "attachments/138694355ace/code/block_017.txt",
      "chars": 547
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:37:57Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:37:57Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:37:57Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:37:57Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/chaimrand/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/the-case-for-centralized-ai-model-inference-serving/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/optimizing-transformer-models-for-variable-length-input-sequences-19fb88fddf71/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/inference/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/onnx/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/pytorch/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/slowly-changing-dimensions-6a08dc0386ae/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/programming/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/gmyrianthous/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/siamese-networks-and-stuart-weitzman-boots-c414be7eff78/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/michaelsugimura/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/photos/brown-lion-with-silver-chain-link-necklace-3tkxfe2GocY"
    },
    {
      "type": "external",
      "href": "https://aws.amazon.com/ec2/instance-types/c7i/"
    },
    {
      "type": "external",
      "href": "https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-x86-gpu-pytorch-2.7-ubuntu-22-04.html"
    },
    {
      "type": "external",
      "href": "https://pytorch.org/get-started/locally/"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html"
    },
    {
      "type": "external",
      "href": "https://github.com/pytorch/vision/blob/v0.19.1/torchvision/csrc/ops/cuda/nms_kernel.cu"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/products/details/processors/xeon/scalable.html"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/architecture-and-technology/avx-512-overview.html"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/developer/articles/guide/deep-learning-with-avx512-and-dl-boost.html"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/what-is-intel-amx.html"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/developer/tools/oneapi/onednn.html"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html"
    },
    {
      "type": "external",
      "href": "https://www.intel.com/content/www/us/en/developer/articles/technical/accelerate-with-intel-extension-for-pytorch.html"
    },
    {
      "type": "external",
      "href": "https://docs.pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html"
    },
    {
      "type": "external",
      "href": "https://docs.pytorch.org/tutorials/intermediate/memory_format_tutorial.html"
    },
    {
      "type": "external",
      "href": "https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/nhwc.html"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:24:00Z",
  "source_url": "https://towardsdatascience.com/optimizing-pytorch-model-inference-on-cpu/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Optimizing PyTorch Model Inference on CPU\nFlyin’ Like a Lion on Intel Xeon\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nDecember 8, 202517 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/ingo-stiller-3tkxfe2GocY-unsplash.jpg)\n[Photo](https://unsplash.com/photos/brown-lion-with-silver-chain-link-necklace-3tkxfe2GocY) by Ingo Stiller on Unsplash\nAs reliance on AI models grows, so does the criticality of optimizing their runtime performance. While the degree to which AI models will outperform human intelligence remains a heated topic of debate, their need for powerful and expensive compute resources is unquestionable — and even notorious.\nIn [previous posts](https://towardsdatascience.com/author/chaimrand/), we covered the topic of AI model optimization — primarily in the context of model training — and demonstrated how it can have a decisive impact on the cost and speed of AI model development. In this post, we focus our attention on AI model inference, where model optimization has an additional objective: To minimize the latency of inference requests and improve the user experience of the model consumer.\nIn this post, we will assume that the platform on which model inference is performed is a 4th Gen Intel® Xeon® Scalable CPU processor, more specifically, an Amazon EC2 [c7i.xlarge](https://aws.amazon.com/ec2/instance-types/c7i/) instance (with 4 Intel Xeon vCPUs) running a dedicated [Deep Learning Ubuntu (22.04) AMI](https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-x86-gpu-pytorch-2.7-ubuntu-22-04.html) and a CPU build of [PyTorch 2.8.0](https://pytorch.org/get-started/locally/). Of course, the choice of a model deployment platform is one of the many important decisions taken when designing an AI solution along with the choice of model architecture, development framework, training accelerator, data format, deployment strategy, etc. — each one of which must be taken with consideration of the associated costs and runtime speed. The choice of a CPU processor for running model inference may seem surprising in an era in which the number of dedicated AI inference accelerators is continuously growing. However, as we will see, there are some occasions when the best (and cheapest) option may very well be just a good old-fashioned CPU.\nWe will introduce a toy image-classification model and proceed to demonstrate some of the optimization opportunities for AI model inference on an Intel® Xeon® CPU. The deployment of an AI model typically includes a full inference server solution, but for the sake of simplicity, we will limit our discussion to just the model’s core execution. For a primer on model inference serving, please see our previous post: [The Case for Centralized AI Model Inference Serving](https://towardsdatascience.com/the-case-for-centralized-ai-model-inference-serving/).\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\nOur intention in this post is to demonstrate that: 1) a few simple optimization techniques can result in meaningful performance gains and 2) that reaching such results does not require specialized expertise in performance analyzers (such as [Intel® VTune™ Profiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html)) or on the inner workings of the low-level compute kernels. Importantly, the process of AI model optimization can differ considerably based on the model architecture and runtime environment. Optimizing for training will differ from optimizing for inference. Optimizing a transformer model will differ from optimizing a CNN model. Optimizing a 22-billion-parameter model will differ from optimizing a 100-million parameter model. Optimizing a model to run on a GPU will differ from optimizing it for a CPU. Even different generations of the same CPU family may have different computation components and, consequently, different optimization techniques. While the high-level steps for optimizing a given model on a given instance are pretty standard, the specific course it will take and the end result can vary greatly based on the project at hand.\nThe code snippets we will share are intended for demonstrative purposes. Please do not rely on their accuracy or their optimality. Please do not interpret our mention of any tool or technique as an endorsement for its use. Ultimately, the best design choices for your use case will greatly depend on the details of your project and, given the extent of the potential impact on performance, should be evaluated with the appropriate time and attention.\n## Why CPU?\nWith the ever-increasing number of hardware solutions for executing AI/ML model inference, our choice of a CPU may seem surprising. In this section, we describe some scenarios in which CPU may be the preferred platform for inference.\n  1. **Accessibility** : The use of dedicated AI accelerators — such as GPUs — typically requires dedicated deployment and maintenance or, alternatively, access to such instances on a cloud service platform. CPUs, on the other hand, are everywhere. Designing a solution to run on a CPU provides much greater flexibility and increases the opportunities for deployment.\n  2. **Availability** : Even if your algorithm can access an AI accelerator, there is the question of availability. AI accelerators are in extremely high demand, and even if/when you are able to acquire one, whether it be on-prem or in the cloud, you may choose to prioritize them for tasks that are even more resource intensive, such as AI model training.\n  3. **Reduced Latency** : There are many situations in which your AI model is just one component in a pipeline of software algorithms running on a standard CPU. While the AI model may perform significantly faster on an AI accelerator, when taking into account the time required to send an inference request over the network, it is quite possible that running it on the same CPU will be faster.\n  4. **Underuse of Accelerator** : AI accelerators are typically quite expensive. To justify their cost, your goal should be to keep them fully occupied, minimizing their idle time. In some cases, the inference load will not justify the cost of an expensive AI accelerator.\n  5. **Model Architecture** : These days, we tend to automatically assume that AI models will perform significantly better on AI accelerators than on CPUs. And while more often than not, this is indeed the case, your model may include layers that perform better on CPU. For example, sequential algorithms such as Non-Maximum Suppression (NMS) and the Hungarian matching algorithm tend to perform better on CPU than GPU and are often offloaded onto the CPU even if a GPU is available (e.g., see [here](https://github.com/pytorch/vision/blob/v0.19.1/torchvision/csrc/ops/cuda/nms_kernel.cu)). If your model contains many such layers, running it on a CPU might not be such a bad option.\n\n\n### Why Intel Xeon?\n[Intel® Xeon® Scalable CPU](https://www.intel.com/content/www/us/en/products/details/processors/xeon/scalable.html) processors come with built-in accelerators for the matrix and convolution operators that are common in typical AI/ML workloads. These include [AVX-512](https://www.intel.com/content/www/us/en/architecture-and-technology/avx-512-overview.html) (introduced in Gen1), the [VNNI extension](https://www.intel.com/content/www/us/en/developer/articles/guide/deep-learning-with-avx512-and-dl-boost.html) (Gen2), and [AMX](https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/what-is-intel-amx.html) (Gen4). The AMX engine, in particular, includes specialized hardware instructions for executing AI models using bfloat16 and int8 precision data types. The acceleration engines are tightly integrated with Intel’s optimized software stack, which includes [oneDNN](https://www.intel.com/content/www/us/en/developer/tools/oneapi/onednn.html), [OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html), and the [Intel Extension for PyTorch](https://www.intel.com/content/www/us/en/developer/articles/technical/accelerate-with-intel-extension-for-pytorch.html) (IPEX). These libraries utilize the dedicated Intel® Xeon® hardware capabilities to optimize model execution with minimal code changes.\nDespite the arguments made in this section, the choice of inference vehicle should be made after considering all options available and after assessing the opportunities for optimization on each one. In the next sections, we will introduce a toy experiment and explore some of the optimization opportunities on CPU.\n## Inference Experiment\nIn this section, we define a toy AI model inference experiment comprising a [Resnet50](https://docs.pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html) image classification model, a randomly generated input batch, and a simple benchmarking utility which we use to report the average number of input samples processed per second (SPS).\npython\n\n```\nimport torch, torchvision\nimport time\n\n\ndef get_model():\n    model = torchvision.models.resnet50()\n    model = model.eval()\n    return model\n\n\ndef get_input(batch_size):\n    batch = torch.randn(batch_size, 3, 224, 224)\n    return batch\n\n\ndef get_inference_fn(model):\n    def infer_fn(batch):\n        with torch.inference_mode():\n            output = model(batch)\n        return output\n    return infer_fn\n\n\ndef benchmark(infer_fn, batch):\n    # warm-up\n    for _ in range(10):\n        _ = infer_fn(batch)\n\n    iters = 100\n\n    start = time.time()\n    for _ in range(iters):\n        _ = infer_fn(batch)\n    end = time.time()\n\n    return (end - start) / iters\n\n\nbatch_size = 1\nmodel = get_model()\nbatch = get_input(batch_size)\ninfer_fn = get_inference_fn(model)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe baseline performance of our toy model is 22.76 samples per second (SPS).\n## Model Inference Optimization\nIn this section, we apply a number of optimizations to our toy experiment and assess their impact on runtime performance. Our focus will be on optimization techniques that can be applied with relative ease. While it is quite likely that additional performance gains can be achieved, these may require much greater specialization and a more significant time investment.\nOur focus will be on optimizations that do not change the model architecture; optimization techniques such as model distillation and model pruning are out of the context of this post. Also out of scope are methods for optimizing specific model components, e.g., by implementing custom PyTorch operators.\nIn a previous post we discussed AI model optimization on Intel XEON CPUs in the context of training workloads. In this section we will revisit some of the techniques mentioned there, this time in the context of AI model inference. We will complement these with optimization techniques that are unique to inference settings, including model compilation for inference, INT8 quantization, and multi-worker inference.\nThe order in which we present the optimization methods is not binding. In fact, some of the techniques are interdependent; for example, increasing the number of inference workers could impact the optimal choice of batch size.\n### Optimization 1: Batched Inference\nA common method for increasing resource utilization while reducing the average inference response time is to group input samples into batches. In real-world scenarios, we need to make sure to cap the batch size so that we meet the service level response time requirements, but for the purposes of our experiment we ignore this requirement. Experimenting with different batch sizes we find that a batch size of 8 results in a throughput of 26.28 SPS, 15% higher than the baseline result.\nNote that in the case that the shapes of the input samples vary, batching requires more handling (e.g., see [here](https://towardsdatascience.com/optimizing-transformer-models-for-variable-length-input-sequences-19fb88fddf71/)).\n### Optimization 2: Channels-Last Memory Format\nBy default in PyTorch, 4D tensors are stored in NCHW format, i.e., the four dimensions represent the batch size, channels, height, and width, respectively. However, the channels-last or NHWC format (i.e., batch size, height, width, and channels) exhibits better performance on CPU. Adjusting our inference script to apply the channels-last optimization is a simple matter of setting the memory format of both the model and the input to  _torch.channels_last_ as shown below:\npython\n\n```\ndef get_model(channels_last=False):\n    model = torchvision.models.resnet50()\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n    model = model.eval()\n    return model\n\ndef get_input(batch_size, channels_last=False):\n    batch = torch.randn(batch_size, 3, 224, 224)\n    if channels_last:\n        batch = batch.to(memory_format=torch.channels_last)\n    return batch\n\n\nbatch_size = 8\nmodel = get_model(channels_last=True)\nbatch = get_input(batch_size, channels_last=True)\ninfer_fn = get_inference_fn(model)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nApplying the  _channels-last_ memory optimization, results in a further boost of 25% in throughput.\nThe impact of this optimization is most noticeable on models that have many convolutional layers. It is not expected to make a noticeable impact on other model architectures (e.g., transformer models).\nPlease see the [PyTorch documentation](https://docs.pytorch.org/tutorials/intermediate/memory_format_tutorial.html) for more details on the memory format optimization and the [Intel documentation](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/nhwc.html) for details on how this is implemented internally in oneDNN.\n### Optimization 3: Automatic Mixed Precision\nModern Intel® Xeon® Scalable processors (from Gen3) include native support for the bfloat16 data type, a 16-bit floating point alternative to the standard float32. We can take advantage of this by applying PyTorch’s automatic mixed precision package, [torch.amp](https://pytorch.org/docs/stable/amp.html), as demonstrated below:\npython\n\n```\ndef get_inference_fn(model, enable_amp=False):\n    def infer_fn(batch):\n        with torch.inference_mode(), torch.amp.autocast(\n                'cpu',\n                dtype=torch.bfloat16,\n                enabled=enable_amp\n        ):\n            output = model(batch)\n        return output\n    return infer_fn\n\nbatch_size = 8\nmodel = get_model(channels_last=True)\nbatch = get_input(batch_size, channels_last=True)\ninfer_fn = get_inference_fn(model, enable_amp=True)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe result of applying mixed precision is a throughput of 86.95 samples per second, 2.6 times the previous experiment and 3.8 times the baseline result.\nNote that the use of a reduced precision floating point type can have an impact on numerical accuracy, and its effect on model quality performance must be evaluated.\n### Optimization 4: Memory Allocation Optimization\nTypical AI/ML workloads require the allocation and access of large blocks of memory. A number of optimization techniques are aimed at tuning the way memory is allocated and used during model execution. One common step is to replace the default system allocator (ptmalloc) with an alternative memory allocation libraries, such as [Jemalloc](https://github.com/jemalloc/jemalloc) and [TCMalloc](https://google.github.io/tcmalloc/overview.html), which have been shown to perform better on common AI/ML workloads (e.g., see [here](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html#choosing-an-optimized-memory-allocator)). To install [TCMalloc](https://google.github.io/tcmalloc/overview.html) run:\nbash\n\n```\nsudo apt-get install google-perftools\n```\n\nWe program its use via the  _LD_PRELOAD_ environment variable:\nbash\n\n```\nLD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 python main.py\n```\n\nThis optimization results in another significant performance boost: 117.54 SPS, 35% higher than our previous experiment!!\n### Optimization 5: Enable Huge Page Allocations\nBy default, the Linux kernel allocates memory in blocks of 4 KB, commonly referred to as pages. The mapping between the virtual and physical memory addresses is managed by the CPU’s Memory Management Unit (MMU), which uses a small hardware cache called the Translation Lookaside Buffer (TLB). The TLB is limited in the number entries it can hold. When you have  _many_ small pages (as in large neural network models), the number of TLB cache misses can climb quickly, increasing latency and slowing down the speed of the program. A common way to address this is to use “huge pages” — blocks of 2 MB (or 1 GB) per page. This reduces the number of TLB entries required, improving memory access efficiency and lowering allocation latency.\nbash\n\n```\nexport THP_MEM_ALLOC_ENABLE=1\n```\n\nIn the case of our model, the impact is negligible. However, this is an important optimization for many AI/ML workloads.\n#### Optimization 6: IPEX\n[Intel® Extension for PyTorch (IPEX)](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/) is a library extension for PyTorch with the latest performance optimizations for Intel hardware. To install it we run:\nbash\n\n```\npip install intel_extension_for_pytorch\n```\n\nIn the code block below, we demonstrate the basic use of the [ipex.optimize](https://intel.github.io/intel-extension-for-pytorch/latest/tutorials/api_doc.html) API.\npython\n\n```\nimport intel_extension_for_pytorch as ipex\n\ndef get_model(channels_last=False, ipex_optimize=False):\n    model = torchvision.models.resnet50()\n\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n\n    model = model.eval()\n\n    if ipex_optimize:\n        model = ipex.optimize(model, dtype=torch.bfloat16)\n\n    return model\n```\n\nThe resultant throughout is 159.31 SPS, for another 36% performance boost.\nPlease see the official documentation for more details on the many optimizations that IPEX has to offer.\n### Optimization 7: Model Compilation\nAnother popular PyTorch optimization is torch.compile. Introduced in PyTorch 2.0, this just-in-time (JIT) compilation feature, performs kernel fusion and other optimizations. In a previous post we covered PyTorch compilation in great detail, covering some its many features, controls, and limitations. Here we demonstrate its basic use:\npython\n\n```\ndef get_model(channels_last=False, ipex_optimize=False, compile=False):\n    model = torchvision.models.resnet50()\n\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n\n    model = model.eval()\n\n    if ipex_optimize:\n        model = ipex.optimize(model, dtype=torch.bfloat16)\n\n    if compile:\n        model = torch.compile(model)\n\n    return model\n```\n\nApplying torch.compile on the IPEX-optimized model results in a throughput of 144.5 SPS, which is lower than our previous experiment. In the case of our model, IPEX and torch.compile do not coexist well. When applying just the torch.compile the throughput is 133.36 SPS.\nThe general takeaway from this experiment is that, for a given model, any two optimization techniques could interfere with one another. This necessitates evaluating the impact of multiple configurations on the runtime performance of a given model in order to find the best one.\n### Optimization 8: Auto-tune Environment Setup With `**torch.xeon.run_cpu**`\nThere are a number of environment settings that control thread and memory management and can be used to further fine-tune the runtime performance of an AI/ML workload. Rather than setting these manually, PyTorch offers the [torch.xeon.run_cpu](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html) script that does this automatically. In preparation for the use of this script, we install Intel’s threading and multiprocessing libraries, [one TBB](https://www.intel.com/content/www/us/en/developer/tools/oneapi/onetbb.html) and [Intel OpenMP](https://www.intel.com/content/www/us/en/docs/oneapi/installation-guide-linux/2023-0/pip.html). We also add a symbolic link to our [TCMalloc](https://google.github.io/tcmalloc/overview.html) installation.\nbash\n\n```\n# install TBB\nsudo apt install -y libtbb12\n# install openMP\npip install intel-openmp\n# link to tcmalloc\nsudo ln -sf /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so\n```\n\nIn the case of our toy model, using torch.xeon.run_cpu increases the throughput to 162.15 SPS — a slight increase over our previous maximum of 159.31 SPS.\nPlease see the [PyTorch documentation](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html) for more features of the `torch.xeon.run_cpu` and more details on the environment variables it applies.\n### Optimization 9: Multi-worker Inference\nAnother popular technique for increasing resource utilization and scale is to load multiple instances of the AI model and run them in parallel in separate processes. Although this technique is more commonly applied on machines with many CPUs (separated into multiple [NUMA nodes](https://docs.pytorch.org/tutorials/recipes/xeon_run_cpu.html#applying-numa-access-control)) — not on our small 4-vCPU instance — we include it here for the sake of demonstration. In the script below we run 2 instances of our model in parallel:\nbash\n\n```\npython -m torch.backends.xeon.run_cpu --ninstances 2 main.py\n```\n\nThis results in a throughput of 169.4 SPS — additional modest but meaningful 4% increase.\n### Optimization 10: INT8 Quantization\nINT8 quantization is another common technique for accelerating AI model inference execution. In INT8 quantization, the floating point datatypes of the model weights and activations are replaced by 8-bit integers. Intel’s Xeon processors include dedicated accelerators for processing INT8 operations (e.g., see [here](https://pytorch.org/blog/int8-quantization/)). INT8 quantization can result in a meaningful increase in speed and a lower memory footprint. Importantly, the reduced bit-precision can have a significant impact on the quality of the model output. There are many different approaches to INT8 quantization some of which include calibration or retraining. There are also a wide variety of tools and libraries for applying quantization. A full discussion on the topic of quantization is beyond the scope of this post. \nSince in this post we are interested just in the potential performance impact, we demonstrate one quantization scheme using [TorchAO](https://github.com/pytorch/ao), without consideration of the impact on model quality. In the code block below, we implement [PyTorch 2 Export Quantization with X86 Backend through Inductor](https://docs.pytorch.org/ao/stable/tutorials_source/pt2e_quant_x86_inductor.html). INT8 quantization is another common technique for accelerating AI model inference execution. In INT8 quantization, the floating point datatypes of the model weights and activations are replaced by 8-bit integers. Intel’s Xeon processors include dedicated accelerators for processing INT8 operations (e.g., see [here](https://pytorch.org/blog/int8-quantization/)). INT8 quantization can result in a meaningful increase in speed and a lower memory footprint.\nImportantly, the reduced bit-precision can have a significant impact on the quality of the model output. There are many different approaches to INT8 quantization some of which include calibration or retraining. There are also a wide variety of tools and libraries for applying quantization. A full discussion on the topic of quantization is beyond the scope of this post. Since in this post we are interested just in the potential performance impact, we demonstrate one quantization scheme using [TorchAO](https://github.com/pytorch/ao), without consideration of the impact on model quality. In the code block below, we implement [PyTorch 2 Export Quantization with X86 Backend through Inductor](https://docs.pytorch.org/ao/stable/tutorials_source/pt2e_quant_x86_inductor.html). Please see the documentation for the full details:\npython\n\n```\nfrom torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e, convert_pt2e\nimport torchao.quantization.pt2e.quantizer.x86_inductor_quantizer as xiq\n\ndef quantize_model(model):\n    x = torch.randn(4, 3, 224, 224).contiguous(\n                            memory_format=torch.channels_last)\n    example_inputs = (x,)\n    batch_dim = torch.export.Dim(\"batch\")\n    with torch.no_grad():\n        exported_model = torch.export.export(\n            model,\n            example_inputs,\n            dynamic_shapes=((batch_dim,\n                             torch.export.Dim.STATIC,\n                             torch.export.Dim.STATIC,\n                             torch.export.Dim.STATIC),\n                            )\n        ).module()\n    quantizer = xiq.X86InductorQuantizer()\n    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())\n    prepared_model = prepare_pt2e(exported_model, quantizer)\n    prepared_model(*example_inputs)\n    converted_model = convert_pt2e(prepared_model)\n    optimized_model = torch.compile(converted_model)\n    return optimized_model\n\n\nbatch_size = 8\nmodel = get_model(channels_last=True)\nmodel = quantize_model(model)\nbatch = get_input(batch_size, channels_last=True)\ninfer_fn = get_inference_fn(model, enable_amp=True)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThis results in a throughput of 172.67 SPS.\nPlease see [here](https://docs.pytorch.org/docs/2.9/quantization.html) for more details on quantization in PyTorch.\n### Optimization 11: Graph Compilation and Execution With ONNX\nThere are a number of third party libraries that specialize in compiling PyTorch models into graph representations and optimizing them for runtime performance on target inference devices. One of the most popular libraries for this is [Open Neural Network Exchange](https://onnx.ai/) (ONNX). ONNX performs ahead-of-time compilation of AI/ML models and executes them using a dedicated runtime library.\nWhile ONNX compilation support is included in PyTorch, we require the following library for executing an ONNX model:\nbash\n\n```\npip install onnxruntime\n```\n\nIn the code block below, we demonstrate ONNX compilation and model execution:\npython\n\n```\ndef export_to_onnx(model, onnx_path=\"resnet50.onnx\"):\n    dummy_input = torch.randn(4, 3, 224, 224)\n    batch = torch.export.Dim(\"batch\")\n    torch.onnx.export(\n        model,\n        dummy_input,\n        onnx_path,\n        input_names=[\"input\"],\n        output_names=[\"output\"],\n        dynamic_shapes=((batch,\n                         torch.export.Dim.STATIC,\n                         torch.export.Dim.STATIC,\n                         torch.export.Dim.STATIC),\n                        ),\n        dynamo=True\n    )\n    return onnx_path\n\ndef onnx_infer_fn(onnx_path):\n    import onnxruntime as ort\n\n    sess = ort.InferenceSession(\n        onnx_path,\n        providers=[\"CPUExecutionProvider\"]\n    )\n    input_name = sess.get_inputs()[0].name\n\n    def infer_fn(batch):\n        result = sess.run(None, {input_name: batch})\n        return result\n    return infer_fn\n\nbatch_size = 8\nmodel = get_model()\nonnx_path = export_to_onnx(model)\nbatch = get_input(batch_size).numpy()\ninfer_fn = onnx_infer_fn(onnx_path)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe resultant throughput is 44.92 SPS, far lower than in our previous experiments. In the case of our toy model, the ONNX runtime does not provide a benefit.\n### Optimization 12: Graph Compilation and Execution with OpenVINO\nAnother opensource toolkit aimed at deploying highly performant AI solutions is [OpenVINO](https://docs.openvino.ai/2025/index.html). [OpenVINO](https://pypi.org/project/openvino/) is [highly optimized](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) for model execution on Intel hardware — e.g., by fully leveraging the Intel AMX instructions. A common way to apply OpenVINO in PyTorch is to first convert the model to ONNX:\npython\n\n```\nfrom openvino import Core\n\ndef compile_openvino_model(onnx_path):\n    core = Core()\n    model = core.read_model(onnx_path)\n    compiled = core.compile_model(model, \"CPU\")\n    return compiled\n\ndef openvino_infer_fn(compiled_model):\n    def infer_fn(batch):\n        result = compiled_model([batch])[0]\n        return result\n    return infer_fn\n\nbatch_size = 8\nmodel = get_model()\nonnx_path = export_to_onnx(model)\novm = compile_openvino_model(onnx_path)\nbatch = get_input(batch_size).numpy()\ninfer_fn = openvino_infer_fn(ovm)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe result of this optimization is a throughput of 297.33 SPS, nearly twice as fast as our previous best experiment!!\nPlease see the official documentation for more details on OpenVINO.\n### Optimization 13: INT8 Quantization in OpenVINO with NNCF\nAs our final optimization, we revisit INT8 quantization, this time in the framework of [OpenVINO compilation](https://docs.openvino.ai/2025/openvino-workflow/model-optimization.html). As before, there are a number of methods for performing quantization — aimed at minimizing the impact on quality performance. Here we demonstrate the basic flow using the [NNCF](https://pypi.org/project/nncf/) library as documented [here](https://docs.openvino.ai/2025/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.html).\npython\n\n```\nclass RandomDataset(torch.utils.data.Dataset):\n\n    def __len__(self):\n        return 10000\n\n    def __getitem__(self, idx):\n        return torch.randn(3, 224, 224)\n\ndef nncf_quantize(onnx_path):\n    import nncf\n\n    core = Core()\n    onnx_model = core.read_model(onnx_path)\n    calibration_loader = torch.utils.data.DataLoader(RandomDataset())\n    input_name = onnx_model.inputs[0].get_any_name()\n    transform_fn = lambda data_item: {input_name: data_item.numpy()}\n    calibration_dataset = nncf.Dataset(calibration_loader, transform_fn)\n    quantized_model = nncf.quantize(onnx_model, calibration_dataset)\n    return core.compile_model(quantized_model, \"CPU\")\n\nbatch_size = 8\nmodel = get_model()\nonnx_path = export_to_onnx(model)\nq_model = nncf_quantize(onnx_path)\nbatch = get_input(batch_size).numpy()\ninfer_fn = openvino_infer_fn(q_model)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThis results in a throughput of 482.46(!!) SPS, another drastic improvement and over 18 times faster than our baseline experiment.\n### Results\nWe summarize the results of our experiments in the table below:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/11/2-2.webp)\nResNet50 Inference Experiment (by Author)\nIn the case of our toy model, the optimizations steps we demonstrated resulted in huge performance gains. Importantly, the impact of each optimization can vary greatly based on the details of the model. You may find that some of these techniques do not apply to your model, or do not result in improved performance. For example, when we reapply the same sequence of optimizations to a Vision Transformer (ViT) model, the resultant performance boost is 8.41X — still significant, but less than the 18.36X of our experiment. Please see the appendix to this post for details.\nOur focus has been on runtime performance, but it is critical that you also evaluate the impact of each optimization on other metrics that are important to you — most importantly model quality.  \nThere are, undoubtedly, many more optimization techniques that can be applied; we have merely scratched the surface. Hopefully, the\n## Summary\nThis post continues our [series](https://towardsdatascience.com/author/chaimrand/) on the important topic of AI/ML model runtime performance analysis and optimization. Our focus in this post was on model inference on Intel® Xeon® CPU processors. Given the ubiquity and prevalence of CPUs, the ability to execute models on them in a reliable and performant manner, can be extremely compelling. As we have shown, by applying a number of relatively simple techniques, we can achieve considerable gains in model performance with profound implications on inference costs and inference latency.\nPlease do not hesitate to reach out with comments, questions, or corrections.\n## Appendix: Vision Transformer Optimization\nTo demonstrate how the impact of the runtime optimizations we discussed depend on the details of the AI/ML model, we reran our experiment on a Vision Transformer (ViT) model from the popular [timm](https://pypi.org/project/timm/) library:\npython\n\n```\nfrom timm.models.vision_transformer import VisionTransformer\n\ndef get_model(channels_last=False, ipex_optimize=False, compile=False):\n    model = VisionTransformer()\n\n    if channels_last:\n        model= model.to(memory_format=torch.channels_last)\n\n    model = model.eval()\n\n    if ipex_optimize:\n        model = ipex.optimize(model, dtype=torch.bfloat16)\n\n    if compile:\n        model = torch.compile(model)\n\n    return model\n```\n\nOne modification in this experiment was to apply [OpenVINO compilation directly to the PyTorch model](https://docs.openvino.ai/2025/openvino-workflow/model-preparation/convert-model-pytorch.html) rather than an intermediate ONNX model. This was due to the fact that OpenVINO compilation failed on the ViT ONNX model. The revised [NNCF quantization](https://docs.openvino.ai/2025/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.html) and OpenVINO compilation sequence is shown below:\npython\n\n```\nimport openvino as ov\nimport nncf\n\n\nbatch_size = 8\nmodel = get_model()\ncalibration_loader = torch.utils.data.DataLoader(RandomDataset())\ncalibration_dataset = nncf.Dataset(calibration_loader)\n\n# quantize PyTorch model\nmodel = nncf.quantize(model, calibration_dataset)\novm = ov.convert_model(model, example_input=torch.randn(1, 3, 224, 224))\novm = ov.compile_model(ovm)\nbatch = get_input(batch_size).numpy()\ninfer_fn = openvino_infer_fn(ovm)\navg_time = benchmark(infer_fn, batch)\nprint(f\"\\nAverage samples per second: {(batch_size/avg_time):.2f}\")\n```\n\nThe table below summarizes the results of the optimizations discussed in this post when applied to the ViT model:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/11/3-2.webp)\nVision Transformer Inference Experiment (by Author)  \n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657094227_7ymyo0.jpeg)\nWritten By\n## [Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[See all from Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Inference](https://towardsdatascience.com/tag/inference/)[Onnx](https://towardsdatascience.com/tag/onnx/)[Pytorch](https://towardsdatascience.com/tag/pytorch/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-pytorch-model-inference-on-cpu%2F&title=Optimizing%20PyTorch%20Model%20Inference%20on%20CPU)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-pytorch-model-inference-on-cpu%2F&title=Optimizing%20PyTorch%20Model%20Inference%20on%20CPU)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-pytorch-model-inference-on-cpu%2F&text=Optimizing%20PyTorch%20Model%20Inference%20on%20CPU)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1uIjq3zp3k4UMEQsp7JXnmg-scaled.jpeg)](https://towardsdatascience.com/slowly-changing-dimensions-6a08dc0386ae/)\n## [Modeling Slowly Changing Dimensions](https://towardsdatascience.com/slowly-changing-dimensions-6a08dc0386ae/)\n[Programming](https://towardsdatascience.com/category/programming/)\nA deep dive into the various SCD types and how they can be implemented in Data Warehouses\n[Giorgos Myrianthous](https://towardsdatascience.com/author/gmyrianthous/)\nMay 3, 202414 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/10/1mV7arhViLrL5MhadNvf5IQ.jpeg)](https://towardsdatascience.com/siamese-networks-and-stuart-weitzman-boots-c414be7eff78/)\n## [Siamese Networks and Stuart Weitzman Boots](https://towardsdatascience.com/siamese-networks-and-stuart-weitzman-boots-c414be7eff78/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nPytorch based siamese CNN for shoe similarity and showing how to build an end to end similarity pipeline\n[Michael Sugimura](https://towardsdatascience.com/author/michaelsugimura/)\nOctober 15, 201810 min read\n  * [![](https://assets.insightmediagroup.io/media/1790194171394_nd8aim.webp)](https://towardsdatascience.com/your-models-mse-is-lying-to-you-part-ii/)\n## [Your Model's MSE Is Lying to You: Part II](https://towardsdatascience.com/your-models-mse-is-lying-to-you-part-ii/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAutoregressive rollout and uncertainty propagation. Second in a series on probabilistic forecasting for physical signals.\n[Waleed Esmail](https://towardsdatascience.com/author/waleed.esmail/)\nSeptember 25, 202620 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/12/1igK-vCoAbj1nL7QIw25Hw.png)](https://towardsdatascience.com/pendragon-four-multi-agent-reinforcement-learning-with-fate-grand-order-80f6254754dd/)\n## [Pendragon Four: Multi-Agent Reinforcement Learning with Fate Grand Order](https://towardsdatascience.com/pendragon-four-multi-agent-reinforcement-learning-with-fate-grand-order-80f6254754dd/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nMulti-agent reinforcement learning in a custom game environment to train 4 agents and have them play the mobile phone game Fate Grand Order\n[Michael Sugimura](https://towardsdatascience.com/author/michaelsugimura/)\nDecember 17, 201923 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/02/19A9HXQgIN_q0LEkn2AbMlQ.png)](https://towardsdatascience.com/finding-the-fastest-lane-at-border-crossings-using-machine-vision-2e8dec6e03a3/)\n## [Finding the Fastest Lane at Border Crossings Using Machine Vision](https://towardsdatascience.com/finding-the-fastest-lane-at-border-crossings-using-machine-vision-2e8dec6e03a3/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nOptimizing border crossing with object detection and tracking\n[Danilo Najkov](https://towardsdatascience.com/author/danilo.najkov/)\nFebruary 16, 202312 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/08Lg5u47mzgHZyhuG-scaled.jpg)](https://towardsdatascience.com/machine-learning-with-r-a-complete-guide-to-decision-trees-1d4dfc88bc20/)\n## [Machine Learning with R: A Complete Guide to Decision Trees](https://towardsdatascience.com/machine-learning-with-r-a-complete-guide-to-decision-trees-1d4dfc88bc20/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn easy and straightforward guide to machine learning and classification with decision trees.\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nFebruary 10, 20215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/0TQJuvYnX4sJEb9lz-scaled.jpg)](https://towardsdatascience.com/dataloader-for-sequential-data-using-pytorch-deep-learning-framework-part-2-ed3ad5f6ad82/)\n## [Dataloader for sequential data using PyTorch deep learning framework - Part 2](https://towardsdatascience.com/dataloader-for-sequential-data-using-pytorch-deep-learning-framework-part-2-ed3ad5f6ad82/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nEnable yourself for writing dataloader for both image and sequential data by reading this blog\n[Harsh Maheshwari](https://towardsdatascience.com/author/anveenaik/)\nMay 8, 20217 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0_43R9Srwce_5mtsG-scaled.jpg)](https://towardsdatascience.com/design-patterns-with-python-for-machine-learning-engineers-builder-45b8e749f134/)\n## [Design Patterns with Python for Machine Learning Engineers: Builder](https://towardsdatascience.com/design-patterns-with-python-for-machine-learning-engineers-builder-45b8e749f134/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLearn how to use the Builder design pattern to enhance your code\n[Marcello Politi](https://towardsdatascience.com/author/marcellopoliti/)\nOctober 12, 20236 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/1deedp8I7-x7GxJlL3LKsTw.png)](https://towardsdatascience.com/3-must-know-advanced-analytics-for-understanding-your-customers-f650f4ac7f57/)\n## [3 must-know advanced analytics for understanding your customers](https://towardsdatascience.com/3-must-know-advanced-analytics-for-understanding-your-customers-f650f4ac7f57/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTop Use-cases for customer analytics and how to do them\n[Pranay Dave](https://towardsdatascience.com/author/pranay-dave9/)\nNovember 18, 20206 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n",
  "body_base_path": "attachments/138694355ace/pages/",
  "body_markdown_path": "attachments/138694355ace/pages/archived-body.md",
  "local_html": "attachments/138694355ace/pages/archived-body.html",
  "body_source": "prior_capture"
}