{ "source_file": "0930", "source_platform": "Medium", "author": "Chaim Rand", "title_listed": "Maximizing AI/ML Model Performance with PyTorch Compilation", "keywords": "", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/maximizing-ai-ml-model-performance-with-pytorch-compilation/", "publish_date": "2025-08-18", "corpus_stage": "", "notes": "", "id": "cec6d4ebea83", "collected_at": "2026-10-02T11:17:32", "title": "Maximizing AI/ML Model Performance with PyTorch Compilation | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# Maximizing AI/ML Model Performance with PyTorch Compilation\nPractical Tips for Getting the Most Out of torch.compile\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nAugust 18, 202526 min read\n![](../images/000.webp)\n[Photo](https://unsplash.com/photos/red-boat-near-mosque-painting--1h_NN3nqzI) by Mg Cthu on Unsplash\nSince its inception in [PyTorch 2.0](https://towardsdatascience.com/tips-and-tricks-for-upgrading-to-pytorch-2-3127db1d1f3d/) in March 2023, the evolution of torch.compile has been one of the most exciting things to follow. Given that PyTorch’s popularity was due to its “Pythonic” nature, its ease of use, and its line-by-line (a.k.a., eager) execution, the success of a just-in-time (JIT) graph compilation mode should not have been taken for granted. And yet, just over two years later, the importance of this feature cannot be overstated: It is an essential tool in optimizing the runtime performance of AI/ML workloads.\nUnfortunately, the use of torch.compile still feels a bit like a dark art. When it works it is awesome and everyone is happy. However, when it doesn’t, figuring out the reason can be difficult. It has multiple API controls, but knowing which ones to apply and when — can seem like black magic. Moreover, its documentation is currently somewhat decentralized, with the details of many of its key features scattered across multiple posts and tutorials.\nAlthough covered in a [previous post](https://medium.com/data-science/tips-and-tricks-for-upgrading-to-pytorch-2-3127db1d1f3d), we felt that the rapid evolution of torch.compile warranted a renewed discussion. This post attempts to unveil some of the mystique surrounding torch.compile. We will review how it works, demonstrate its use, discuss a few strategies for how to apply it most effectively, and evaluate the impact of some of its features on the runtime performance of a toy model. We will cover the following topics:\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n * techniques for avoiding the two “compilation-killers”, graph-breaks and recompilations,\n * strategies for debugging compilation issues\n * squeezing maximum performance using some of torch.compile’s advanced features and configuration settings,\n * making the most of the torch.compile logs to debug compilation issues,\n * modular application of torch.compile,\n * methods for reducing compilation time,\n * and more.\n\n\nAs in our previous posts, we will define a toy PyTorch model which we will use to demonstrate the application and impact of torch.compile. We will run our experiments on an [Amazon EC2 p4d.96xlarge](https://aws.amazon.com/ec2/instance-types/p4/) instance (containing 8 NVIDIA A100 GPUs) running a [PyTorch (2.7) Deep Learning AMI (DLAMI)](https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-ami-gpu-pytorch-2.7-ubuntu-22-04.html).\n### Disclaimers:\nPyTorch compilation is a complex topic with a continuously growing set of features. This post makes no attempt to encompass the full scope of torch.compile, but rather aims to offer some practical tips on how to approach it. For a complete reference, please see the [official PyTorch documentation](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html). But keep in mind that you may need to surf through multiple pages to collect all the information you need (e.g., [here](https://docs.pytorch.org/docs/stable/generated/torch.compile.html) for the API documentation, [here](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) for an introductory tutorial, [here](https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_deepdive.html) for a deep-dive on TorchDynamo, [here](https://docs.pytorch.org/tutorials/compilers_index.html) and [here](https://docs.pytorch.org/docs/stable/torch.compiler.html#read-more) for indices to many other pages covering a wide range of compilation features, etc.).\nIf you prefer a single source with a comprehensive overview of torch.compile, its inner workings, and detailed examples of its use, we recommend chapter 14 of the book [AI Systems Performance Engineering](https://github.com/cfregly/ai-performance-engineering/tree/main), by Chris Fregly.\nThe code we will share is intended for demonstrative purposes and should not be relied on for correctness or optimality — especially for other projects. Please do not interpret our choice of platform, framework, or any other tool or library as an endorsement for its use.\nThe impact of torch.compile can vary greatly based on the details of the AI/ML model and runtime environment. The results we will share on our toy model may not be indicative of the results you will get on your own model. In fact, compilation of some models may result in worse performance.\nWhen applied correctly, torch.compile should not affect the quality of your model (in the case of inference) or its ability to converge (in the case of training). However, there are likely to be numerical differences due to the use of different compute kernels. It is essential that you verify that applying torch.compile does not degrade your quality-performance metrics before deploying it to a production environment.\nImportantly, torch.compile continues to evolve with each PyTorch release. The contents of this post are based on PyTorch 2.7. Staying up-to-date with latest PyTorch releases is essential for taking advantage of the latest and greatest available optimization opportunities.\n## PyTorch Compilation: How it Works\nIn PyTorch’s default _eager execution_ mode, each line of Python code is processed independently. While this mode of execution is extremely user-friendly — making it easy to follow and debug line-per-line what the model is doing — it misses a great deal of opportunity to optimize performance, e.g.:\n 1. GPU operations are performed independently. This misses the opportunity for _operator fusion_ where GPU operations are combined into a single, more efficient, GPU kernel.\n 2. Potential optimizations from ahead-of-time (AOT) compilation, such as out-of-order execution and memory layout optimizations, are missed.\n 3. The Python runtime is involved in all stages of the model execution. Every time an operation is launched on the GPU, control is passed from the Python interpreter to the CUDA backend and back. This can introduce significant overhead.\n\n\n### How torch.compile Fixes This\nFirst introduced in PyTorch 2.0, torch.compile acts as a just-in-time (JIT) compiler: The first time you call a compiled function, the compiler traces the Python code and converts it into an intermediate graph representation (IR) using [TorchDynamo](https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_overview.html), sometimes referred to as an [FX Graph](https://docs.pytorch.org/docs/stable/fx.html#torch.fx.Graph). If the compiled function requires backpropagation, the FX Graph is passed to the [AOTAutograd](https://docs.pytorch.org/functorch/stable/notebooks/aot_autograd_optimizations.html) library which captures the backward pass ahead-of-time (AOT) and generates a combined forward and backward graph. The FX Graph is then passed to the compiler backend which performs kernel fusion, out-of-order execution, and other techniques to generate machine code that is highly optimized for the target hardware.\nThe default PyTorch compiler backend is [TorchInductor](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747) which supports both GPU and CPU targets. When compiling for NVIDIA GPUs, TorchInductor uses: 1) the [Triton](https://openai.com/index/triton/) compiler (previously covered in [this post](https://towardsdatascience.com/unleashing-the-power-of-triton-mastering-gpu-kernel-optimization-in-python-160a3f52701e/)) to create optimal GPU kernels and 2) [CUDA Graphs](https://developer.nvidia.com/blog/cuda-graphs/) (whenever possible) to combine multiple GPU kernels into efficient, re-playable sequences.\nThe final, machine-specific computation graph is [cached](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_configuration_tutorial.html) and used for each subsequent invocation of the compiled function/model. Note that although the bulk of the compilation is performed on the first invocation, several additional _warm-up_ passes are often required to reach peak performance.\nThe combined JIT and AOT properties of torch.compile allow it to maximize opportunities for graph optimization, while the use of the compiled execution graph avoids the line-by-line involvement of the Python interpreter — thereby addressing the three aforementioned inefficiencies of eager execution.\n## Avoiding Compilation Pitfalls\nUsually, applying torch.compile will boost your model throughput (e.g., see the [TorchInductor performance dashboard](https://hud.pytorch.org/benchmark/compilers)). However, sometimes you may find that torch compilation results in the same or even worse performance than in eager mode. There could be a number of reasons for this:\n 1. There may be a bottleneck in the training step that is overshadowing the torch.compile optimization, e.g., a data input pipeline bottleneck. This can be identified and solved through appropriate performance analysis and optimization.\n 2. Your function or model might already be so efficient that the application of torch.compile is negligible.\n 3. You may be suffering from one of two compilation killers, graph-breaks and recompilations, which we elaborate on in the next sections.\n\n\n### PyTorch Compilation Killer #1: Graph-Breaks\nGraph-breaks are one of the most common events that interfere with efficient torch compilation. Graph-breaks occur when the TorchDynamo or AOTAutograd libraries encounter Python operations that they cannot convert into a graph operation. In such cases, the sections of code before and after the problematic operation, are compiled separately and the resultant graph is said to contain a _graph-break_. Graph-breaks interfere with the compiler’s capacity for optimization in two primary ways: First, optimizations such as kernel fusion cannot be performed across graph breaks and, second, a graph break implies a return of control to the Python interpreter. The presence of a large number of graph breaks can completely cancel out the potential benefit of torch.compile. Common examples of graph breaks include print() operations, conditional logic, and asserts.\nWhat is frustrating is that, more often than not, graph-breaks can be easily avoided. What is even more frustrating is that the default behavior is to handle graph breaks by silently falling back to eager execution for the problematic code segment.\n### Avoiding Graph-Breaks\nThe first step to handling graph-breaks is to configure the compiler to report them. Here are several ways of doing this:\n 1. Apply the torch._dynamo.explain operator to your (uncompiled) model and run it on a sample input (as demonstrated [here](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html#torchdynamo-and-fx-graphs)). This will result in a log containing a list of all of the graph-breaks.\n 2. Set the _TORCH_LOGS_ environment variable to include “ _graph_breaks_ ”. This will cause the compiler to print the graph-breaks it encounters during compilation.\n 3. Call with torch.compile with _fullgraph_ =_True_. This will cause the compilation to fail each time it encounters a graph-break — thereby forcing the developer to acknowledge its presence and potentially fix it.\n\n\nWhile our personal preference is option three, it is important to note that there are times where graph-breaks cannot be avoided, which means that we may need to disable _fullgraph_ in a production setting. The best example of this is distributed training (e.g., [DDP](https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html) and [FSDP](https://docs.pytorch.org/docs/stable/fsdp.html)) where the computation group includes communication calls which (as of the time of this writing) are not supported by torch.compile and, thus, result in graph-breaks.\nWith knowledge of the location of our graph breaks, we address each one individually. We remove redundant prints and assertions, replace conditional blocks with graph-friendly alternatives such as [torch.where](https://docs.pytorch.org/docs/stable/generated/torch.where.html) or [torch.cond](https://docs.pytorch.org/docs/stable/generated/torch.cond.html), and adjust our model implementation to minimize untraceable Python control flow and native operations. In some cases, we may desire to maintain some of the prints or assertions for running in eager mode; in this case, we can wrap them in a conditional check like `if not torch.compiler.is_compiling()`. There may be cases (e.g., DDP) where graph-breaks are unavoidable.\nSee [here](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#resolving-graph-breaks) for more on avoiding graph-breaks.\n### PyTorch Compilation Killer #2: Recompilations\nThe second potential compilation killer is the graph recompilation. During the initial graph compilation phase, several assumptions are made and relied upon for generating the resultant graph. In torch.compile lingo these assumptions are referred to as _guards_. Common guards include the data types and shapes of input tensors. On each iteration, these guards are verified on the current tensor inputs and training state. If one of the guards is violated, the current graph is deemed invalid for the current state and a new graph is generated, i.e., the graph is recompiled. Graph compilation takes an extremely long time relative to the time it takes to execute a compiled graph. Consequently, multiple recompilations is likely to erase any potential performance gains from torch.compile. Moreover, torch.compile has a recompilation limit (the default is 8) after which it will raise a torch._dynamo.exc.RecompileLimitExceeded exception and fall back to eager mode.\n### Avoiding Recompiles\nHere too, the first step is identifying the causes of the recompilations. Once again, there are several options:\n 1. Use [torch_compiler.set_stance](https://docs.pytorch.org/tutorials/recipes/torch_compiler_set_stance_tutorial.html) operator to fail on recompile: `torch.compiler.set_stance(“fail_on_recompile”)`. In practice, this option can sometimes prove to be too limiting.\n 2. Set the _TORCH_LOGS_ environment variable to include “ _recompiles_ ”. This will cause the compiler to report each time it performs recompilation along with the guards that were violated.\n\n\n### Compiling Graphs with Variable-Shaped Tensors\nOne of the most common causes of recompilations is the presence of tensors with dynamic shapes. The first time a graph is compiled it creates guards according to the shapes of the tensors it traced. When a tensor changes shape in a subsequent step, the guard is violated and the graph is recompiled. There are multiple ways of handling tensors with dynamic shapes:\n 1. **Default Compilation Behavior** : If the _dynamic_ field of the torch.compile call is not set (or set to _None_), each time the compiler encounters a new dynamic tensor, it will perform recompilation to generate a new graph that supports the dynamism it identified. In this option, the graph modification is applied surgically, allowing for “static” optimizations to be applied to other portions of the graph. If new dynamism is discovered in multiple iterations, we may hit the recompilation limit and fall back to eager execution. Consequently, this option should only be used for models with limited dynamism.\n 2. **Mark Dynamic Tensors** : Another option is to explicitly mark the dynamic tensors and associated dynamic axis using the torch._dynamo.mark_dynamic API. This informs the compiler to build a graph that supports the reported dynamism and prevents recompilations altogether. This is a great option in situations in which you know upfront what your dynamic shapes are (which you absolutely should!!).\n 3. **Dynamic Compilation** : The third option is to apply torch.compile with _dynamic_ =_True_. This instructs the compiler to construct a graph that is as dynamic as possible in order to avoid recompilations. When enabled, dynamic shape tracing is applied to all of the tensors in the graph. This is often overkill. Keep in mind that many graph optimization techniques (e.g., CUDA graphs) assume static shapes. These are automatically disabled when this setting is applied. This option should be avoided whenever possible.\n 4. **Generate a Limited Number of Static Graphs** : When torch.compile is applied with _dynamic_ =_False,_ the compiler will never generate dynamic graphs. Each time a guard is violated a new static graph is created, supporting the newly encountered tensor shape, and added to the compilation cache. While limited (by the recompilation limit) in the number of shapes it can support, this option is compelling due to the fact that it allows for optimizations that assume a static graph. To benefit from this capability, a common approach is to remove dynamism from the graph by padding dynamic tensors to a fixed length. A more advanced approach that reduces the amount of padding is to set a number of fixed length values (e.g., powers of two) and pad the variable shaped tensors to the closest length. The number of length values should not exceed the recompilation limit. This will result in a fixed number of recompilations and a fixed number of highly optimized graphs. We can ensure that all graphs are created during the model warmup phase.\n\n\nAs before, there are some situations where graph recompilations cannot be avoided, and we may have no choice but to run our model in eager mode.\nSee [here](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#dealing-with-recompilations) for more on avoiding recompilations and [here](https://docs.pytorch.org/docs/stable/torch.compiler_dynamic_shapes.html) for details on how torch.compile handles dynamic shapes.\n## Debugging Compilation Issues\nInevitably, you will encounter some situations where torch compilation fails. Often, you will get a long error message and callstack, but it may as well be in a foreign language. You will likely be encouraged to _Set TORCH_LOGS=”+dynamo” and TORCHDYNAMO_VERBOSE=1_ but you may find that this does little to help you solve the problem.\nThe [torch.compile troubleshooting](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html) guide offers multiple tips for diagnosing compilation errors (e.g., by compiling with “eager”, “aot_eager” and “inductor” backends), for fixing or avoiding them, and if all else fails, for [reporting](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#reporting-issues) them to PyTorch. In this post we call out two different approaches for tackling tough compilation issues.\n### Top-Down VS. Bottom-Up Approach\nIn a top-down approach, we apply torch compilation at the highest-level function/model — come what may. We then begin to work through the compilation issues as they come up by either fixing them or removing them from the graph via the [torch.compiler.disable](https://docs.pytorch.org/docs/stable/generated/torch.compiler.disable.html) utility. This approach assumes that we are sufficiently able to decipher the compilation logs — at least well enough to navigate to the problematic line of code.\nIn a bottom-up approach, we begin by applying compilation to a few low-level components and slowly increase the scope of compilation until we hit an error. This approach makes it easy to pinpoint the sources of the compilation issue. An additional advantage is that we can benefit from the results of a partially compiled graph while we continue to work on additional optimizations. This is contrary to the Top-Down approach where we will only have a workable graph once all issues are addressed.\nThe best approach depends on the model at hand and your personal inclination.Often, a combination of the two delivers the best results: for example, identifying issues via a bottom-up approach, resolving them, and then testing if the full graph compilation works.\n## Tuning for Maximal Performance\nOnce you have succeeded in compiling your model, there are a wide range of controls for trying to squeeze out even better performance. In this section we will cover some of the available options. It should be noted that the additional performance gains from these options are usually a small fraction of the gains from the initial application of standard compilation.\n### Advanced Compiler Modes and Options\nThe [torch.compile API](https://docs.pytorch.org/docs/stable/generated/torch.compile.html) allows for tuning the compiler-backend behavior via via the `mode` and `options` parameters. There are dozens of knobs that can be applied and assessed. Some of the most notable ones are “reduce-overhead” — that optimizes more aggressively to further reduce the overhead of the kernel loading and Python interpreter, and “max-autotune” — the most aggressive optimization option that performs benchmarking of multiple kernel options before choosing the most efficient one. Both of these, particularly “max-autotune”, increase the compilation time, but usually result in more efficient graphs.\n### Varying the Compiler Backend\nThe default compiler backend is TorchInductor which supports a variety of target devices. You can specify the compiler backend via the _backend_ parameter of the [torch.compile API](https://docs.pytorch.org/docs/stable/generated/torch.compile.html). While other backends are unlikely to beat TorchInductor when running on NVIDIA GPUs, you may find them to perform better on other hardware devices (e.g., the [ _ipex_](https://github.com/intel/intel-extension-for-pytorch) backend includes optimizations that leverage the unique capabilities of Intel® CPUs).\n### Applying Modular Compilation\nWhile it is usually recommended to apply compilation to the entire model, there are times where the model can be broken into submodules that respond very differently to the compiler controls. For example, if your model contains one component that includes many tensors with dynamic shapes and another component that is static, you may find that compiling the first in “max-autotune-no-cudagraphs” mode and the second in “max-autotune” mode, results in maximum performance.\n### Compiling the Optimizer\nIn addition to compiling the model execution, as of PyTorch 2.2, you can further optimize your training workload by [compiling the optimizer](https://docs.pytorch.org/tutorials/recipes/compiling_optimizer.html). This will be demonstrated below.\n### New Compiler Features\nSince the initial release of torch.compile in PyTorch 2.0, each PyTorch release has included enhancements to the torch.compile offering. Sometimes released as “prototypes”, new features offerings challenge developers to extract even greater performance out of graph compilation. For example, the [PyTorch 2.7 release](https://pytorch.org/blog/pytorch-2-7/) included the [foreach_map](https://docs.pytorch.org/tutorials/recipes/foreach_map.html) prototype feature, the use of which we will demonstrate below.\n## Reducing Compilation Time\nWhile the initial compilation and warm-up time can be quite long compared to the subsequent training steps, it is usually negligible compared to the overall life of the model (i.e., the training or inference time). In some cases, however, the lengthy compilation time can become an issue. If the model is extremely large and we are tuning for optimal performance, compilation could take hours. If we are using our model in an inference server setup, the model start-up time could have a direct impact on the server response time and user experience.\nIn this section we cover two techniques for reducing model compilation time: [compile-time caching](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [regional compilation](https://docs.pytorch.org/tutorials/recipes/regional_compilation.html).\n### Compile Time Caching\nIn compile-time caching we upload the results of the local graph compilation to persistent storage. Every time we need to run the same model in the same runtime environment (e.g., same hardware and same library versions) we pull the cache state from persistent storage to the local disk, instead of compiling from scratch.\n### Regional Compilation\nRegional compilation relies on the fact that large models typically consist of computation blocks that are repeated multiple times. In regional compilation, torch.compile is applied to the repeating block, instead of the entire model. The result is a single, relatively small graph that is created and reused for each of the blocks.\n## How to Configure the TORCH_LOGS Environment Variable\nTorch compilation supports a wide variety of [logging controls](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#summary-of-torch-logs-options). While the log reports can be extremely useful for debugging issues and maximizing performance, it’s important to find the right balance where the logs are helpful but not excessive. In this post we propose using the following initial configuration and adapting as needed:\nbash\n\n```\nexport TORCH_LOGS=\"graph_breaks,recompiles,perf_hints\"\n```\n\n * “graph_breaks” — reports each time a graph-break is encountered (see above)\n * “recompiles” — reports each time a recompilation is performed along with the guard-violation that triggered it.\n * “perf_hints” — outputs performance logs from the inductor backend along with hints for additional optimizations\n\n\nNote that sometimes “perf_hints” will flood the console with unactionable messages, in which case you may opt to disable it.\n## A Toy PyTorch Model: Image Captioning\nTo demonstrate torch.compile in action, we define a toy image captioning model using the popular [Hugging Face](https://huggingface.co/) [transformers](https://huggingface.co/docs/transformers/en/index) library (version 4.54.1). Specifically, we define an image-to-text model using a [VisionEncoderDecoderModel](https://huggingface.co/docs/transformers/v4.53.3/en/model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel), with a [Vision Transformer (ViT)](https://huggingface.co/papers/2010.11929) encoder and a [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) decoder, and train it on a synthetic dataset of fixed-sized images and random sequences (“captions”) of variable length.\nWe begin by defining our image-to-text model:\npython\n\n```\nimport os, shutil, time, random, torch\nfrom transformers import (\n VisionEncoderDecoderModel,\n VisionEncoderDecoderConfig,\n AutoConfig\n)\n\ntorch.manual_seed(42)\nrandom.seed(42)\n\nBATCH_SIZE = 64\nNUM_WORKERS = 12\nNUM_TOKENS = 1024\nMAX_SEQ_LEN = 256\nPAD_ID = 0\nSTART_ID = 1\nEND_ID = 2\n\n\n# set up image-to-text model\ndef get_model():\n config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(\n encoder_config=AutoConfig.for_model(\"vit\"), # vit encoder\n decoder_config=AutoConfig.for_model(\"gpt2\") # gpt2 decoder\n )\n config.decoder.vocab_size = NUM_TOKENS\n config.decoder.use_cache = False\n config.decoder_start_token_id = START_ID\n config.pad_token_id = PAD_ID\n config.eos_token_id = END_ID\n config.max_length = MAX_SEQ_LEN\n\n model = VisionEncoderDecoderModel(config=config)\n\n # remove unused pooler\n model.encoder.pooler = None\n\n # uncomment to specify the loss function\n # from transformers.loss.loss_utils import ForCausalLMLoss\n # model.loss_function = ForCausalLMLoss\n return model\n```\n\nNext, we define a synthetic dataset that generates pairs of random images of fixed size and random sequences of variable size. We use a weighted distribution for the sequence length to mimic a scenario where the vast majority of sequences are short.\nGiven the varying length of the input captions, we require a strategy for dealing with dynamically shaped input. Here, we offer two alternatives, both of which use padding: padding to the maximum input length and padding to the length of the longest sequence in the batch, along with an option to align it to a given multiple. Please see our [previous post](https://towardsdatascience.com/optimizing-transformer-models-for-variable-length-input-sequences-19fb88fddf71/) for additional strategies for handling variable-length input sequences.\npython\n\n```\nfrom torch.utils.data import Dataset, DataLoader\nfrom functools import partial\n\n# A synthetic Dataset with random images and captions\nclass FakeDataset(Dataset):\n def __init__(self):\n self.length_dist = {\n 'short': {'range': (5, 32), 'weight': 0.90},\n 'medium': {'range': (33, 64), 'weight': 0.09},\n 'long': {'range': (65, 256), 'weight': 0.01}\n }\n super().__init__()\n\n def __len__(self):\n return 1000000\n\n def __getitem__(self, index):\n length_bin = random.choices(\n list(self.length_dist.keys()),\n weights=[d['weight'] for d in self.length_dist.values()],\n k=1\n )[0]\n\n range_start, range_end = self.length_dist[length_bin]['range']\n image = torch.randn(3, 224, 224)\n length = random.randint(range_start, range_end - 1)\n labels = torch.cat([torch.randint(1, NUM_TOKENS, (length,)),\n torch.tensor([END_ID])],\n dim=0)\n input_ids = torch.cat([torch.tensor([START_ID]),\n labels[:-1]],\n dim=0)\n return {\n 'image': image,\n 'input_ids': input_ids,\n 'labels': labels\n }\n\ndef pad_sequence(sequence, length, pad_val):\n return torch.nn.functional.pad(\n sequence,\n (0, length - sequence.shape[0]),\n value=pad_val\n )\n\ndef collate_with_padding(batch, pad_to_longest=False, align=None):\n padded_inputs = []\n padded_labels = []\n if pad_to_longest:\n pad_len = max([b['input_ids'].shape[0] for b in batch])\n if align:\n pad_len = ((pad_len + align - 1) // align) * align\n else:\n pad_len = MAX_SEQ_LEN\n\n for b in batch:\n input_ids = b['input_ids']\n labels = b['labels']\n padded_inputs.append(pad_sequence(input_ids, pad_len, PAD_ID))\n padded_labels.append(pad_sequence(labels, pad_len, -100))\n\n padded_inputs = torch.stack(padded_inputs, dim=0)\n padded_labels = torch.stack(padded_labels, dim=0)\n images = torch.stack([b['image'] for b in batch], dim=0)\n return {\n 'pixel_values': images,\n 'decoder_input_ids': padded_inputs,\n 'labels': padded_labels,\n 'decoder_attention_mask': (padded_inputs != PAD_ID)\n }\n\ndef get_dataloader(pad_to_longest=False, align=None):\n return DataLoader(\n dataset=FakeDataset(),\n batch_size=BATCH_SIZE,\n num_workers=NUM_WORKERS,\n collate_fn=partial(\n collate_with_padding,\n pad_to_longest=pad_to_longest,\n align=align\n )\n )\n```\n\nLast, we define our training step and main training function:\npython\n\n```\ndef copy_to_device(batch, device):\n return {\n key: val.to(device=device, non_blocking=True)\n for key, val in batch.items()\n }\n\ndef train_step(model, device, optimizer, batch):\n # copy data to device\n batch = copy_to_device(batch, device)\n optimizer.zero_grad()\n with torch.amp.autocast('cuda', dtype=torch.bfloat16):\n outputs = model(**batch)\n loss = outputs.loss\n loss.backward()\n optimizer.step()\n return loss\n\ndef train(local_rank=0, world_size=1, compile=False):\n # specify log settings\n torch._logging.set_logs(\n graph_breaks=True,\n recompiles=True,\n perf_hints=True\n )\n\n torch.cuda.set_device(local_rank)\n device = torch.cuda.current_device()\n\n if world_size > 1:\n # DDP setup\n import torch.distributed as dist\n from torch.nn.parallel import DistributedDataParallel as DDP\n os.environ['MASTER_ADDR'] = 'localhost'\n os.environ['MASTER_PORT'] = str(2222)\n dist.init_process_group('nccl', rank=local_rank,\n world_size=world_size)\n\n # configure pad_to_longest and optional alignment\n dataloader = get_dataloader(pad_to_longest=False, align=None)\n\n model = get_model()\n model = model.to(device)\n if world_size > 1:\n model = DDP(model, [local_rank])\n optimizer = torch.optim.Adam(model.parameters())\n\n if compile:\n # uncomment to run pre-compile warmup - required for some optimizations\n # batch = next(iter(dataloader))\n # train_step(model, device, optimizer, batch)\n model, optimizer = apply_compilation(model, optimizer)\n\n warmup = 20\n active = 100\n total_steps = warmup + active\n t0 = time.perf_counter()\n\n for idx, batch in enumerate(dataloader, start=1):\n # apply train step\n train_step(model, device, optimizer, batch)\n\n if idx == warmup:\n torch.cuda.synchronize()\n print(f'warmup time: {time.perf_counter()-t0}')\n t0 = time.perf_counter()\n elif idx == total_steps:\n break\n\n if local_rank == 0:\n torch.cuda.synchronize()\n total_time = time.perf_counter() - t0\n print(f'average throughput: {active / total_time}')\n\n if world_size > 1:\n dist.destroy_process_group()\n\n\nif __name__ == '__main__':\n # specify inductor cache dir\n inductor_cache_dir = '/tmp/inductor_cache'\n os.environ['TORCHINDUCTOR_CACHE_DIR'] = inductor_cache_dir\n\n # clean up compiler cache\n torch._dynamo.reset()\n shutil.rmtree(inductor_cache_dir, ignore_errors=True)\n\n world_size = 1\n torch.multiprocessing.spawn(\n fn=train,\n args=(world_size,),\n nprocs=world_size,\n join=True\n )\n```\n\n### Baseline Performance\nRunning the training script without compilation yields the following baseline performance results:\n![](../images/001.webp)\nBaseline Model Performance (by Author)\nWe can see clearly that the collation strategy that reduces padding results in much better performance.\n## Applying Model Compilation\nIn this section we will apply torch compilation with different configurations and measure its impact on the training throughput. We will begin by applying compilation without dynamism, i.e., when padding all inputs to max sequence length. In the following section we will evaluate its impact in the case of inputs with dynamic shapes.\n### Model Compilation Step #1: Fixing Graph Breaks\nWe introduce the following compilation utility function and apply it to our model:\npython\n\n```\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True)\n return model, optimizer\n```\n\nThe _fullgraph_ setting ensures that compilation will fail whenever it encounters a graph break. Sure enough, our first compilation attempt results in an error coming from the transformer library. Here is a small snippet:\nshell-session\n\n```\nfrom user code: File \"/opt/pytorch/lib/python3.12/site-packages/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py\", line 574, in forward loss = self.loss_function( File \"/opt/pytorch/lib/python3.12/site-packages/transformers/modeling_utils.py\", line 5776, in loss_function\n```\n\nThe reason for this error is that when the [VisionEncoderDecoderModel](https://huggingface.co/docs/transformers/v4.53.3/en/model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel) loss function is not specified, the transformers library uses native Python code to determine what loss function to apply. This is easy to fix by specifying the model loss function, as follows:\npython\n\n```\nfrom transformers.loss.loss_utils import ForCausalLMLoss\nmodel.loss_function = ForCausalLMLoss\n```\n\nFollowing this fix, model compilation succeeds. The resultant throughput is 5.17 steps per second — a 66% speed-up over the baseline (fixed-input) throughput.\nNote that in the current scenario of a static graph, the compiler did not report any recompilations, but it did report the following _perf_hint_ :\nshell-session\n\n```\nI0805 13:37:52.406000 51587 torch/_inductor/codegen/simd.py:1976] [0/0] [__perf_hints] Reduction over non-contiguous dims.I0805 13:37:52.406000 51587 torch/_inductor/codegen/simd.py:1976] [0/0] [__perf_hints] Consider setting config.triton.tile_reductions to True.\n```\n\nHowever, applying the suggested configuration results in a compilation error, so we ignore it going forward.\n### Model Compilation Step #2: Tuning the Compiler Configuration\nLet’s try to increase the performance further by applying some of the advanced compilation controls. The code block below includes three alternative modifications:\npython\n\n```\n# reduce-overhead\nmodel = torch.compile(model, fullgraph=True, mode=\"reduce-overhead\")\n\n# max-autotune\nmodel = torch.compile(model, fullgraph=True, mode=\"max-autotune\")\n\n# shapes padding\nmodel = torch.compile(model, fullgraph=True, options={\"shape_padding\":True})\n```\n\nThe results are captured in the table below:\n![](../images/body-71769591bb9c75f4216b.webp)\ntorch.compile results (by Author)\nThe subsequent experiments in this section will be run with the “max-autotune” optimization.\n### Model Compilation Step #3: Compiling the Optimizer\nNext, we extend our solution to apply compilation to the optimizer. Since optimizer compilation currently requires graph-breaks, we apply it without the _fullgraph_ flag:\npython\n\n```\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True, mode=\"max-autotune\")\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nCompiling the optimizer further increases the throughput to 5.54 steps per second!!\nWhen compiling the optimizer, the following performance hint is printed:\nshell-session\n\n```\n will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.\n```\n\nThe proposal is to fix the addresses of gradient tensors and mark them. To implement the suggestion, we introduce the following two utility functions:\npython\n\n```\n# this replaces default optimizer.zero_grad() and verifies reuse\n# of same gradient tensors\ndef zero_grads(model):\n for p in model.parameters():\n if p.grad is not None:\n p.grad.zero_()\n\n# uses dynamo utility to mark each of the gradient tensors as static\ndef mark_static_address(optimizer):\n for group in optimizer.param_groups:\n for p in group['params']:\n if p.grad is not None:\n torch._dynamo.mark_static_address(p.grad)\n```\n\nThe updated training step appears below:\npython\n\n```\ndef train_step(model, device, optimizer, batch):\n # copy data to device\n batch = copy_to_device(batch, device)\n zero_grads(model)\n with torch.amp.autocast('cuda', dtype=torch.bfloat16):\n outputs = model(**batch)\n loss = outputs.loss\n loss.backward()\n mark_static_address(optimizer)\n optimizer.step()\n return loss\n```\n\nIn our case, implementing the performance hint decreases the throughput result to 5.32 steps per second — so we disregard it.\n### Model Compilation Step #4: Foreach Map Optimization\nConstantly be on the lookout for torch.compile enhancements and additions. Here we will apply [horizontal fusion with foreach_map](https://docs.pytorch.org/tutorials/recipes/foreach_map.html) — an optimization introduced in the latest PyTorch release — to the optimizer step. Using the utility functions from the [Foreach Map tutorial](https://docs.pytorch.org/tutorials/recipes/foreach_map.html), we create an optimized Adam optimizer step function, and apply it to our optimizer:\npython\n\n```\ndef get_compiled_adam_step(optimizer):\n compiled_adam = torch.compile(foreach_map_adam)\n inputs = get_inputs(optimizer)\n def compiled_adam_step():\n compiled_adam(*inputs)\n return compiled_adam_step\n\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True, mode=\"max-autotune\")\n optimizer.step = get_compiled_adam_step(optimizer)\n return model, optimizer\n```\n\nThis optimization requires use of the _zero_grads_ utility from above. It also requires that we run a warmup training step before compilation to populate all of the gradient tensors.\nThe modified optimizer step results in a reduced throughput of 5.28 steps per second. We presume that our toy model is too small to reap the benefit of the new compilation feature.\nOur best result, 5.54 steps per second, is 78% faster than our baseline result. Let’s see what happens when we extend our solution to multiple GPUs.\n### Model Compilation Step #5: Extending to DDP\nThe final step is to extend the training script to use all 8 GPUs. For this step we need to disable the _fullgraph_ setting since the cross-GPU gradient sharing requires graph-breaking communication calls.\nThe resultant throughput is 4.59 steps per second, nearly two times faster than our baseline result.\n### Results\nThe table below summarizes the results of our static-graph experiments:\n![](../images/body-f2b817fa04effe14232c.webp)\nStatic Graph Compilation Results (by Author)\nThus far, all of our experiments have assumed fixed-sized input tensors. Since the vast majority of input sequences are small, our graph is performing a huge amount of wasteful computation.\nIn the next section we will evaluate torch.compile when padding to variable-length inputs.\n## Dynamic Model Compilation\nIn this section we introduce dynamism into our toy model definition by padding the inputs sequences in each batch to the length of the longest sequence. In a previous section we described several strategies for compiling dynamic graphs. We will apply these strategies and assess their impact on the training throughput.\nThe experiments in this section were run on a single NVIDIA A100 GPU.\n### Option #1: Auto-Detect Dynamism\nThe default behavior (_dynamic_ =_None_) of torch.compile is to auto-detect dynamism and recompile the graph accordingly. When running in this setting, we indeed see the recompilation due to the variation in input size, but we also get the following print:\nshell-session\n\n```\nV0806 09:31:00.624000 175763 torch/_dynamo/guards.py:2997] [0/1] [__recompiles] - 0/1: ((decoder_input_ids.size()[1]*decoder_input_ids.size()[1]) % 8) != 0 # attn_output = torch.nn.functional.scaled_dot_product_attention( # transformers/integrations/sdpa_attention.py:89 in sdpa_attention_forward (_dynamo/utils.py:3284 in run_node)\n```\n\nThe source of this recompilation is the [scaled_dot_product_attention](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) operator, which requires that input shapes be aligned to multiples of eight for optimal use. To address this issue and avoid the recompilation, we modify our padding operation to pad to a multiple of eight.\nTo avoid the recompilation that is triggered by the variable-length inputs, we define the following utility and apply it to the input tensors:\npython\n\n```\ndef mark_dynamic(batch):\n for key in ['decoder_input_ids', 'labels', 'decoder_attention_mask']:\n torch._dynamo.mark_dynamic(batch[key], 1)\n\ndef train_step(model, device, optimizer, batch):\n # copy data to device\n batch = copy_to_device(batch, device)\n # mark inputs as dynamic to avoid recompilation\n mark_dynamic(batch)\n optimizer.zero_grad()\n with torch.amp.autocast('cuda', dtype=torch.bfloat16):\n outputs = model(**batch)\n loss = outputs.loss\n loss.backward()\n optimizer.step()\n return loss\n```\n\nThis option results in a throughput of 7.78 steps per second, 64% higher than the baseline throughput (4.73).\nAn additional speed-up is achieved when we apply the “max-autotune” mode — 8.13 steps per second.\n### Option #2: Dynamic Compilation\nAnother way to avoid recompilations is to call torch.compile with _dynamic_ =_True_ :\npython\n\n```\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True, dynamic=True)\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nThis results in a throughput of 7.77 steps per second. Since setting _dynamic_ =_True_ precludes the use of CUDA graphs, we attempt to optimize further by setting _mode_ =”max-autotune-no-cudagraphs”. This results in a throughput of 7.89 steps per second.\n### Option #3: Compile a Fixed Number of Static Graphs\nThe last option we explore is to set a fixed number of supported input shapes and compile a corresponding fixed number of static graphs. Since the default number of recompilations supported is eight, we program our collator to emit eight different tensor shapes by aligning the padding to multiples of 32. To force the recompilations, we set _dynamic_ =_False_.\nThe resultant throughputs are for 7.77 steps per second for the default mode and 8.04 for _mode_ =”max-autotune”.\nNote that this option may require a greater number of warmup steps to ensure that all shape variations are processed. (An alternative is to manually feed the model with all shape variations before starting the training loop.)\n### Modular Compilation\nSince our model naturally splits into two submodules — a static encoder and a dynamic decoder — it is tempting to explore the option of applying separate compilation to each component. Note that in an inference setting, it is essential to compile the encoder and decoder separately, since the encoder is called only once, while the decoder is called repeatedly in an auto-regressive loop.\npython\n\n```\ndef apply_compilation(model, optimizer):\n model.encoder = torch.compile(model.encoder, fullgraph=True)\n model.decoder = torch.compile(model.decoder, fullgraph=True)\n model.loss_function = torch.compile(model.loss_function, fullgraph=True)\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nThe result of this strategy is a throughput of 7.93, which is slightly higher than the result we got (in default mode) when compiling the entire model.\nOne advantage to this approach is the ability to tune the compilation controls to each submodule independently. For example, setting _mode_ =”max-autotune” to just the encoder, further increased the throughput to 8.04 steps per second.\n### Results\nWe summarize the results of our dynamic-graph experiments in the table below:\n![](../images/body-1c165d4497c53b3d25b2.webp)\nDynamic Graph Compilation Results (by Author)\nThe best result was 8.13 steps per second, 72% higher than the baseline result (4.73). It is likely that further tuning could result in additional gains.\nKeep in mind that the impact of torch.compile can vary greatly based on the details of the model and the runtime environment.\n## Reducing Compilation Time\nWe now turn our attention to the duration of the torch.compile warmup. We will assess the two optimizations discussed above, [compile-time caching](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [regional compilation](https://docs.pytorch.org/tutorials/recipes/regional_compilation.html). We limit our experiments to a single GPU. We use the default application of torch.compile and measure the duration of the first 20 training iterations.\n### Pre-Loading Compilation Cache\nIn the following demonstration of compile-time caching, we use an [Amazon S3](https://aws.amazon.com/s3/) bucket as our persistent storage location:\npython\n\n```\nimport boto3\n\nS3_BUCKET = \"\"\nS3_KEY = \"\"\n\ndef download_cache():\n s3_client = boto3.client('s3')\n t0 = time.perf_counter()\n try:\n response = s3_client.get_object(Bucket=S3_BUCKET, Key=S3_KEY)\n artifact_bytes = response['Body'].read()\n torch.compiler.load_cache_artifacts(artifact_bytes)\n print(f\"Cache restored. Time: {time.perf_counter()-t0} sec\")\n except:\n return False\n return True\n\ndef upload_cache():\n s3_client = boto3.client('s3')\n artifact_bytes, cache_info = torch.compiler.save_cache_artifacts()\n s3_client.put_object(\n Bucket=S3_BUCKET,\n Key=S3_KEY,\n Body=artifact_bytes\n )\n\n\nif __name__ == '__main__':\n # specify inductor cache dir\n inductor_cache_dir = '/tmp/inductor_cache'\n os.environ['TORCHINDUCTOR_CACHE_DIR'] = inductor_cache_dir\n\n # clean up compiler cache\n torch._dynamo.reset()\n shutil.rmtree(inductor_cache_dir, ignore_errors=True)\n\n # upload the compilation artifacts\n download_cache()\n \n # train the model\n train()\n\n # upload the compilation artifacts\n upload_cache()\n```\n\nThis method reduces the compilation warmup from 196 seconds to 56 seconds — a 3.5X speed-up.\n### Regional Compilation\nTo implement regional compilation, we apply compilation to the internal blocks of both the encoder and the decoder:\npython\n\n```\ndef apply_compilation(model, optimizer):\n model.encoder.encoder.layer = torch.nn.ModuleList(\n [torch.compile(layer, fullgraph=True)\n for layer in model.encoder.encoder.layer]\n )\n model.decoder.transformer.h = torch.nn.ModuleList(\n [torch.compile(layer, fullgraph=True)\n for layer in model.decoder.transformer.h]\n )\n model.loss_function = torch.compile(model.loss_function, fullgraph=True)\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nThis change reduces the throughput from 7.78 steps per second to 7.61 steps per second. On the other hand, the compilation warmup drops from 196 seconds to 80 seconds — a 2.45X speed-up.\nIn the case of our toy model — which is extremely small by today’s standards — the gains we have demonstrated are modest. But for large models, these types of compilation-time optimization techniques could prove essential.\n## Summary\nAs AI/ML models grow in size to hundreds of billions or even trillions of parameters, optimizing their runtime performance becomes increasingly essential. For PyTorch models, torch.compile is one of the most powerful optimization tools at your disposal. This post has aimed to ease the adoption of torch.compile by addressing some of its intricacies and demonstrating its practical use. Some of the main techniques we covered were:\n * Reducing graph-breaks and recompilations\n * Tuning compilation settings to maximize performance gains\n * Effective use of the PyTorch logs\n * Top-down vs. bottom-up debugging strategies\n * Modular application of torch.compile\n * Reducing the duration of compilation warmup\n\n\nPyTorch compilation is a complex and nuanced topic. In this post we have covered just some of its many features. For more on the topic, be refer to the [official documentation](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html).\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-97f5483d57a9b8741146.jpeg)\nWritten By\n## [Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[See all from Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Python](https://towardsdatascience.com/tag/python/)[Pytorch](https://towardsdatascience.com/tag/pytorch/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmaximizing-ai-ml-model-performance-with-pytorch-compilation%2F&title=Maximizing%20AI%2FML%20Model%20Performance%20with%20PyTorch%20Compilation)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmaximizing-ai-ml-model-performance-with-pytorch-compilation%2F&title=Maximizing%20AI%2FML%20Model%20Performance%20with%20PyTorch%20Compilation)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmaximizing-ai-ml-model-performance-with-pytorch-compilation%2F&text=Maximizing%20AI%2FML%20Model%20Performance%20with%20PyTorch%20Compilation)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/002.png)](https://towardsdatascience.com/time-series-transformations-selection-using-edaspy-d97e0dc1fcca/)\n## [Time series transformations selection using EDAspy](https://towardsdatascience.com/time-series-transformations-selection-using-edaspy-d97e0dc1fcca/)\n[](https://towardsdatascience.com/maximizing-ai-ml-model-performance-with-pytorch-compilation/)\nEasy way to find the optimum Time series transformations for a forecasting model in a Machine Learning project.\n[Vicente P. Soloviev](https://towardsdatascience.com/author/vicente-perez-soloviev/)\nNovember 25, 20204 min read\n * [![](../images/003.jpeg)](https://towardsdatascience.com/want-to-break-into-data-science-start-building-db5ed24a7d77/)\n## [Want to break into data science? Start building](https://towardsdatascience.com/want-to-break-into-data-science-start-building-db5ed24a7d77/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWhy you should build first and ask questions later\n[Russell Pollari](https://towardsdatascience.com/author/russellpollari/)\nOctober 19, 20204 min read\n * [![](../images/004.jpg)](https://towardsdatascience.com/how-to-select-loss-function-and-activation-function-for-classification-problems-b7f32c724507/)\n## [How to Select Loss Function and Activation Function for Classification Problems](https://towardsdatascience.com/how-to-select-loss-function-and-activation-function-for-classification-problems-b7f32c724507/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nUnderstanding how to choose activation function in the final layer and loss function for neural network\n[Ibrahim Kovan](https://towardsdatascience.com/author/ibrahimkovan/)\nApril 14, 20226 min read\n * [![](../images/005.jpg)](https://towardsdatascience.com/three-main-components-of-machine-learning-modeling-c9186658f7fe/)\n## [Three Main Components of Machine Learning Modeling](https://towardsdatascience.com/three-main-components-of-machine-learning-modeling-c9186658f7fe/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThe art of learning through data\n[Abiodun Olaoye](https://towardsdatascience.com/author/aolaoye/)\nOctober 24, 20224 min read\n * [![](../images/006.jpg)](https://towardsdatascience.com/teaching-is-hard-how-to-train-small-models-and-outperforming-large-counterparts-f131f9d463e1/)\n## [Teaching is Hard: How to Train Small Models and Outperforming Large Counterparts](https://towardsdatascience.com/teaching-is-hard-how-to-train-small-models-and-outperforming-large-counterparts-f131f9d463e1/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDistilling the knowledge of a large model is complex but a new method shows incredible performances\n[Salvatore Raieli](https://towardsdatascience.com/author/salvatore-raieli/)\nNovember 11, 202311 min read\n * [![](../images/007.jpeg)](https://towardsdatascience.com/exploring-reddit-wallstreetbets-posts-data-bed613386029/)\n## [Exploring Reddit WallStreetBets Posts Data](https://towardsdatascience.com/exploring-reddit-wallstreetbets-posts-data-bed613386029/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nEDA and Sentiment Analysis of Reddit Data\n[Sadrach Pierre, Ph.D.](https://towardsdatascience.com/author/spierre91/)\nFebruary 8, 20215 min read\n * [![](../images/008.png)](https://towardsdatascience.com/linear-regression-in-time-series-sources-of-spurious-regression/)\n## [Linear Regression in Time Series: Sources of Spurious Regression](https://towardsdatascience.com/linear-regression-in-time-series-sources-of-spurious-regression/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWhy does the autocorrelation of the errors term matter?\n[JUNIOR JUMBONG](https://towardsdatascience.com/author/jumbongjunior1999/)\nMarch 10, 202513 min read\n * [![](../images/009.png)](https://towardsdatascience.com/how-can-machine-learning-help-visually-impaired-people-4fcdc76816b2/)\n## [How machine learning can help visually impaired people](https://towardsdatascience.com/how-can-machine-learning-help-visually-impaired-people-4fcdc76816b2/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAn outdoor obstacle identification project proposal for blind people with DeepLabv3+\n[Shuyang Xiang](https://towardsdatascience.com/author/vanillaxiangshuyang/)\nJanuary 4, 20226 min read\n * [![](../images/010.png)](https://towardsdatascience.com/the-end-of-anchors-improving-object-detection-models-and-annotations-73828c7b39f6/)\n## [The End of Anchors - Improving Object Detection Models and Annotations](https://towardsdatascience.com/the-end-of-anchors-improving-object-detection-models-and-annotations-73828c7b39f6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nIf you have ever had to tinker with anchor boxes, you were probably frustrated, confused and saying to yourself, \"There must be another...\n[Anders Christiansen](https://towardsdatascience.com/author/andersasac/)\nApril 2, 20194 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n", "metadata": { "title": "Maximizing AI/ML Model Performance with PyTorch Compilation | Towards Data Science", "description": "Practical Tips for Getting the Most Out of torch.compile", "keywords": null, "author": "Chaim Rand", "og:locale": "en_US", "og:type": "article", "og:title": "Maximizing AI/ML Model Performance with PyTorch Compilation", "og:description": "Practical Tips for Getting the Most Out of torch.compile", "og:url": "https://towardsdatascience.com/maximizing-ai-ml-model-performance-with-pytorch-compilation/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/08/1.webp", "og:image:width": "1100", "og:image:height": "751", "og:image:type": "image/webp", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Chaim Rand", "twitter:label2": "Est. reading time", "twitter:data2": "26 min read", "article:published_time": "2025-08-18T12:31:33.000Z", "article:modified_time": "2025-08-18T12:31:33.000Z", "article:author": "https://towardsdatascience.com/author/chaimrand/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_000.txt", "chars": 55 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_001.txt", "chars": 1060 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_002.txt", "chars": 2667 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_003.txt", "chars": 2748 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_004.txt", "chars": 118 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_005.txt", "chars": 307 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_006.txt", "chars": 95 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_007.txt", "chars": 265 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_008.txt", "chars": 264 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_009.txt", "chars": 190 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_010.txt", "chars": 208 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_011.txt", "chars": 480 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_012.txt", "chars": 347 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_013.txt", "chars": 412 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_014.txt", "chars": 339 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_015.txt", "chars": 542 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_016.txt", "chars": 183 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_017.txt", "chars": 327 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_018.txt", "chars": 1141 }, { "language": "txt", "file": "attachments/cec6d4ebea83/code/block_019.txt", "chars": 513 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:05Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:05Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:05Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:05Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/chaimrand/" }, { "type": "internal", "href": "https://towardsdatascience.com/tips-and-tricks-for-upgrading-to-pytorch-2-3127db1d1f3d/" }, { "type": "internal", "href": "https://towardsdatascience.com/unleashing-the-power-of-triton-mastering-gpu-kernel-optimization-in-python-160a3f52701e/" }, { "type": "internal", "href": "https://towardsdatascience.com/optimizing-transformer-models-for-variable-length-input-sequences-19fb88fddf71/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/python/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/pytorch/" }, { "type": "internal", "href": "https://towardsdatascience.com/time-series-transformations-selection-using-edaspy-d97e0dc1fcca/" }, { "type": "internal", "href": "https://towardsdatascience.com/maximizing-ai-ml-model-performance-with-pytorch-compilation/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/vicente-perez-soloviev/" }, { "type": "internal", "href": "https://towardsdatascience.com/want-to-break-into-data-science-start-building-db5ed24a7d77/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/russellpollari/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://unsplash.com/photos/red-boat-near-mosque-painting--1h_NN3nqzI" }, { "type": "external", "href": "https://medium.com/data-science/tips-and-tricks-for-upgrading-to-pytorch-2-3127db1d1f3d" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://aws.amazon.com/ec2/instance-types/p4/" }, { "type": "external", "href": "https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-ami-gpu-pytorch-2.7-ubuntu-22-04.html" }, { "type": "external", "href": "https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html" }, { "type": "external", "href": "https://docs.pytorch.org/docs/stable/generated/torch.compile.html" }, { "type": "external", "href": "https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_deepdive.html" }, { "type": "external", "href": "https://docs.pytorch.org/tutorials/compilers_index.html" }, { "type": "external", "href": "https://docs.pytorch.org/docs/stable/torch.compiler.html" }, { "type": "external", "href": "https://github.com/cfregly/ai-performance-engineering/tree/main" }, { "type": "external", "href": "https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_overview.html" }, { "type": "external", "href": "https://docs.pytorch.org/docs/stable/fx.html" }, { "type": "external", "href": "https://docs.pytorch.org/functorch/stable/notebooks/aot_autograd_optimizations.html" }, { "type": "external", "href": "https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747" }, { "type": "external", "href": "https://openai.com/index/triton/" }, { "type": "external", "href": "https://developer.nvidia.com/blog/cuda-graphs/" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:27:48Z", "source_url": "https://towardsdatascience.com/maximizing-ai-ml-model-performance-with-pytorch-compilation/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# Maximizing AI/ML Model Performance with PyTorch Compilation\nPractical Tips for Getting the Most Out of torch.compile\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nAugust 18, 202526 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/08/1.webp)\n[Photo](https://unsplash.com/photos/red-boat-near-mosque-painting--1h_NN3nqzI) by Mg Cthu on Unsplash\nSince its inception in [PyTorch 2.0](https://towardsdatascience.com/tips-and-tricks-for-upgrading-to-pytorch-2-3127db1d1f3d/) in March 2023, the evolution of torch.compile has been one of the most exciting things to follow. Given that PyTorch’s popularity was due to its “Pythonic” nature, its ease of use, and its line-by-line (a.k.a., eager) execution, the success of a just-in-time (JIT) graph compilation mode should not have been taken for granted. And yet, just over two years later, the importance of this feature cannot be overstated: It is an essential tool in optimizing the runtime performance of AI/ML workloads.\nUnfortunately, the use of torch.compile still feels a bit like a dark art. When it works it is awesome and everyone is happy. However, when it doesn’t, figuring out the reason can be difficult. It has multiple API controls, but knowing which ones to apply and when — can seem like black magic. Moreover, its documentation is currently somewhat decentralized, with the details of many of its key features scattered across multiple posts and tutorials.\nAlthough covered in a [previous post](https://medium.com/data-science/tips-and-tricks-for-upgrading-to-pytorch-2-3127db1d1f3d), we felt that the rapid evolution of torch.compile warranted a renewed discussion. This post attempts to unveil some of the mystique surrounding torch.compile. We will review how it works, demonstrate its use, discuss a few strategies for how to apply it most effectively, and evaluate the impact of some of its features on the runtime performance of a toy model. We will cover the following topics:\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n * techniques for avoiding the two “compilation-killers”, graph-breaks and recompilations,\n * strategies for debugging compilation issues\n * squeezing maximum performance using some of torch.compile’s advanced features and configuration settings,\n * making the most of the torch.compile logs to debug compilation issues,\n * modular application of torch.compile,\n * methods for reducing compilation time,\n * and more.\n\n\nAs in our previous posts, we will define a toy PyTorch model which we will use to demonstrate the application and impact of torch.compile. We will run our experiments on an [Amazon EC2 p4d.96xlarge](https://aws.amazon.com/ec2/instance-types/p4/) instance (containing 8 NVIDIA A100 GPUs) running a [PyTorch (2.7) Deep Learning AMI (DLAMI)](https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-ami-gpu-pytorch-2.7-ubuntu-22-04.html).\n### Disclaimers:\nPyTorch compilation is a complex topic with a continuously growing set of features. This post makes no attempt to encompass the full scope of torch.compile, but rather aims to offer some practical tips on how to approach it. For a complete reference, please see the [official PyTorch documentation](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html). But keep in mind that you may need to surf through multiple pages to collect all the information you need (e.g., [here](https://docs.pytorch.org/docs/stable/generated/torch.compile.html) for the API documentation, [here](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) for an introductory tutorial, [here](https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_deepdive.html) for a deep-dive on TorchDynamo, [here](https://docs.pytorch.org/tutorials/compilers_index.html) and [here](https://docs.pytorch.org/docs/stable/torch.compiler.html#read-more) for indices to many other pages covering a wide range of compilation features, etc.).\nIf you prefer a single source with a comprehensive overview of torch.compile, its inner workings, and detailed examples of its use, we recommend chapter 14 of the book [AI Systems Performance Engineering](https://github.com/cfregly/ai-performance-engineering/tree/main), by Chris Fregly.\nThe code we will share is intended for demonstrative purposes and should not be relied on for correctness or optimality — especially for other projects. Please do not interpret our choice of platform, framework, or any other tool or library as an endorsement for its use.\nThe impact of torch.compile can vary greatly based on the details of the AI/ML model and runtime environment. The results we will share on our toy model may not be indicative of the results you will get on your own model. In fact, compilation of some models may result in worse performance.\nWhen applied correctly, torch.compile should not affect the quality of your model (in the case of inference) or its ability to converge (in the case of training). However, there are likely to be numerical differences due to the use of different compute kernels. It is essential that you verify that applying torch.compile does not degrade your quality-performance metrics before deploying it to a production environment.\nImportantly, torch.compile continues to evolve with each PyTorch release. The contents of this post are based on PyTorch 2.7. Staying up-to-date with latest PyTorch releases is essential for taking advantage of the latest and greatest available optimization opportunities.\n## PyTorch Compilation: How it Works\nIn PyTorch’s default _eager execution_ mode, each line of Python code is processed independently. While this mode of execution is extremely user-friendly — making it easy to follow and debug line-per-line what the model is doing — it misses a great deal of opportunity to optimize performance, e.g.:\n 1. GPU operations are performed independently. This misses the opportunity for _operator fusion_ where GPU operations are combined into a single, more efficient, GPU kernel.\n 2. Potential optimizations from ahead-of-time (AOT) compilation, such as out-of-order execution and memory layout optimizations, are missed.\n 3. The Python runtime is involved in all stages of the model execution. Every time an operation is launched on the GPU, control is passed from the Python interpreter to the CUDA backend and back. This can introduce significant overhead.\n\n\n### How torch.compile Fixes This\nFirst introduced in PyTorch 2.0, torch.compile acts as a just-in-time (JIT) compiler: The first time you call a compiled function, the compiler traces the Python code and converts it into an intermediate graph representation (IR) using [TorchDynamo](https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_overview.html), sometimes referred to as an [FX Graph](https://docs.pytorch.org/docs/stable/fx.html#torch.fx.Graph). If the compiled function requires backpropagation, the FX Graph is passed to the [AOTAutograd](https://docs.pytorch.org/functorch/stable/notebooks/aot_autograd_optimizations.html) library which captures the backward pass ahead-of-time (AOT) and generates a combined forward and backward graph. The FX Graph is then passed to the compiler backend which performs kernel fusion, out-of-order execution, and other techniques to generate machine code that is highly optimized for the target hardware.\nThe default PyTorch compiler backend is [TorchInductor](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747) which supports both GPU and CPU targets. When compiling for NVIDIA GPUs, TorchInductor uses: 1) the [Triton](https://openai.com/index/triton/) compiler (previously covered in [this post](https://towardsdatascience.com/unleashing-the-power-of-triton-mastering-gpu-kernel-optimization-in-python-160a3f52701e/)) to create optimal GPU kernels and 2) [CUDA Graphs](https://developer.nvidia.com/blog/cuda-graphs/) (whenever possible) to combine multiple GPU kernels into efficient, re-playable sequences.\nThe final, machine-specific computation graph is [cached](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_configuration_tutorial.html) and used for each subsequent invocation of the compiled function/model. Note that although the bulk of the compilation is performed on the first invocation, several additional _warm-up_ passes are often required to reach peak performance.\nThe combined JIT and AOT properties of torch.compile allow it to maximize opportunities for graph optimization, while the use of the compiled execution graph avoids the line-by-line involvement of the Python interpreter — thereby addressing the three aforementioned inefficiencies of eager execution.\n## Avoiding Compilation Pitfalls\nUsually, applying torch.compile will boost your model throughput (e.g., see the [TorchInductor performance dashboard](https://hud.pytorch.org/benchmark/compilers)). However, sometimes you may find that torch compilation results in the same or even worse performance than in eager mode. There could be a number of reasons for this:\n 1. There may be a bottleneck in the training step that is overshadowing the torch.compile optimization, e.g., a data input pipeline bottleneck. This can be identified and solved through appropriate performance analysis and optimization.\n 2. Your function or model might already be so efficient that the application of torch.compile is negligible.\n 3. You may be suffering from one of two compilation killers, graph-breaks and recompilations, which we elaborate on in the next sections.\n\n\n### PyTorch Compilation Killer #1: Graph-Breaks\nGraph-breaks are one of the most common events that interfere with efficient torch compilation. Graph-breaks occur when the TorchDynamo or AOTAutograd libraries encounter Python operations that they cannot convert into a graph operation. In such cases, the sections of code before and after the problematic operation, are compiled separately and the resultant graph is said to contain a _graph-break_. Graph-breaks interfere with the compiler’s capacity for optimization in two primary ways: First, optimizations such as kernel fusion cannot be performed across graph breaks and, second, a graph break implies a return of control to the Python interpreter. The presence of a large number of graph breaks can completely cancel out the potential benefit of torch.compile. Common examples of graph breaks include print() operations, conditional logic, and asserts.\nWhat is frustrating is that, more often than not, graph-breaks can be easily avoided. What is even more frustrating is that the default behavior is to handle graph breaks by silently falling back to eager execution for the problematic code segment.\n### Avoiding Graph-Breaks\nThe first step to handling graph-breaks is to configure the compiler to report them. Here are several ways of doing this:\n 1. Apply the torch._dynamo.explain operator to your (uncompiled) model and run it on a sample input (as demonstrated [here](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html#torchdynamo-and-fx-graphs)). This will result in a log containing a list of all of the graph-breaks.\n 2. Set the _TORCH_LOGS_ environment variable to include “ _graph_breaks_ ”. This will cause the compiler to print the graph-breaks it encounters during compilation.\n 3. Call with torch.compile with _fullgraph_ =_True_. This will cause the compilation to fail each time it encounters a graph-break — thereby forcing the developer to acknowledge its presence and potentially fix it.\n\n\nWhile our personal preference is option three, it is important to note that there are times where graph-breaks cannot be avoided, which means that we may need to disable _fullgraph_ in a production setting. The best example of this is distributed training (e.g., [DDP](https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html) and [FSDP](https://docs.pytorch.org/docs/stable/fsdp.html)) where the computation group includes communication calls which (as of the time of this writing) are not supported by torch.compile and, thus, result in graph-breaks.\nWith knowledge of the location of our graph breaks, we address each one individually. We remove redundant prints and assertions, replace conditional blocks with graph-friendly alternatives such as [torch.where](https://docs.pytorch.org/docs/stable/generated/torch.where.html) or [torch.cond](https://docs.pytorch.org/docs/stable/generated/torch.cond.html), and adjust our model implementation to minimize untraceable Python control flow and native operations. In some cases, we may desire to maintain some of the prints or assertions for running in eager mode; in this case, we can wrap them in a conditional check like `if not torch.compiler.is_compiling()`. There may be cases (e.g., DDP) where graph-breaks are unavoidable.\nSee [here](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#resolving-graph-breaks) for more on avoiding graph-breaks.\n### PyTorch Compilation Killer #2: Recompilations\nThe second potential compilation killer is the graph recompilation. During the initial graph compilation phase, several assumptions are made and relied upon for generating the resultant graph. In torch.compile lingo these assumptions are referred to as _guards_. Common guards include the data types and shapes of input tensors. On each iteration, these guards are verified on the current tensor inputs and training state. If one of the guards is violated, the current graph is deemed invalid for the current state and a new graph is generated, i.e., the graph is recompiled. Graph compilation takes an extremely long time relative to the time it takes to execute a compiled graph. Consequently, multiple recompilations is likely to erase any potential performance gains from torch.compile. Moreover, torch.compile has a recompilation limit (the default is 8) after which it will raise a torch._dynamo.exc.RecompileLimitExceeded exception and fall back to eager mode.\n### Avoiding Recompiles\nHere too, the first step is identifying the causes of the recompilations. Once again, there are several options:\n 1. Use [torch_compiler.set_stance](https://docs.pytorch.org/tutorials/recipes/torch_compiler_set_stance_tutorial.html) operator to fail on recompile: `torch.compiler.set_stance(“fail_on_recompile”)`. In practice, this option can sometimes prove to be too limiting.\n 2. Set the _TORCH_LOGS_ environment variable to include “ _recompiles_ ”. This will cause the compiler to report each time it performs recompilation along with the guards that were violated.\n\n\n### Compiling Graphs with Variable-Shaped Tensors\nOne of the most common causes of recompilations is the presence of tensors with dynamic shapes. The first time a graph is compiled it creates guards according to the shapes of the tensors it traced. When a tensor changes shape in a subsequent step, the guard is violated and the graph is recompiled. There are multiple ways of handling tensors with dynamic shapes:\n 1. **Default Compilation Behavior** : If the _dynamic_ field of the torch.compile call is not set (or set to _None_), each time the compiler encounters a new dynamic tensor, it will perform recompilation to generate a new graph that supports the dynamism it identified. In this option, the graph modification is applied surgically, allowing for “static” optimizations to be applied to other portions of the graph. If new dynamism is discovered in multiple iterations, we may hit the recompilation limit and fall back to eager execution. Consequently, this option should only be used for models with limited dynamism.\n 2. **Mark Dynamic Tensors** : Another option is to explicitly mark the dynamic tensors and associated dynamic axis using the torch._dynamo.mark_dynamic API. This informs the compiler to build a graph that supports the reported dynamism and prevents recompilations altogether. This is a great option in situations in which you know upfront what your dynamic shapes are (which you absolutely should!!).\n 3. **Dynamic Compilation** : The third option is to apply torch.compile with _dynamic_ =_True_. This instructs the compiler to construct a graph that is as dynamic as possible in order to avoid recompilations. When enabled, dynamic shape tracing is applied to all of the tensors in the graph. This is often overkill. Keep in mind that many graph optimization techniques (e.g., CUDA graphs) assume static shapes. These are automatically disabled when this setting is applied. This option should be avoided whenever possible.\n 4. **Generate a Limited Number of Static Graphs** : When torch.compile is applied with _dynamic_ =_False,_ the compiler will never generate dynamic graphs. Each time a guard is violated a new static graph is created, supporting the newly encountered tensor shape, and added to the compilation cache. While limited (by the recompilation limit) in the number of shapes it can support, this option is compelling due to the fact that it allows for optimizations that assume a static graph. To benefit from this capability, a common approach is to remove dynamism from the graph by padding dynamic tensors to a fixed length. A more advanced approach that reduces the amount of padding is to set a number of fixed length values (e.g., powers of two) and pad the variable shaped tensors to the closest length. The number of length values should not exceed the recompilation limit. This will result in a fixed number of recompilations and a fixed number of highly optimized graphs. We can ensure that all graphs are created during the model warmup phase.\n\n\nAs before, there are some situations where graph recompilations cannot be avoided, and we may have no choice but to run our model in eager mode.\nSee [here](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#dealing-with-recompilations) for more on avoiding recompilations and [here](https://docs.pytorch.org/docs/stable/torch.compiler_dynamic_shapes.html) for details on how torch.compile handles dynamic shapes.\n## Debugging Compilation Issues\nInevitably, you will encounter some situations where torch compilation fails. Often, you will get a long error message and callstack, but it may as well be in a foreign language. You will likely be encouraged to _Set TORCH_LOGS=”+dynamo” and TORCHDYNAMO_VERBOSE=1_ but you may find that this does little to help you solve the problem.\nThe [torch.compile troubleshooting](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html) guide offers multiple tips for diagnosing compilation errors (e.g., by compiling with “eager”, “aot_eager” and “inductor” backends), for fixing or avoiding them, and if all else fails, for [reporting](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#reporting-issues) them to PyTorch. In this post we call out two different approaches for tackling tough compilation issues.\n### Top-Down VS. Bottom-Up Approach\nIn a top-down approach, we apply torch compilation at the highest-level function/model — come what may. We then begin to work through the compilation issues as they come up by either fixing them or removing them from the graph via the [torch.compiler.disable](https://docs.pytorch.org/docs/stable/generated/torch.compiler.disable.html) utility. This approach assumes that we are sufficiently able to decipher the compilation logs — at least well enough to navigate to the problematic line of code.\nIn a bottom-up approach, we begin by applying compilation to a few low-level components and slowly increase the scope of compilation until we hit an error. This approach makes it easy to pinpoint the sources of the compilation issue. An additional advantage is that we can benefit from the results of a partially compiled graph while we continue to work on additional optimizations. This is contrary to the Top-Down approach where we will only have a workable graph once all issues are addressed.\nThe best approach depends on the model at hand and your personal inclination.Often, a combination of the two delivers the best results: for example, identifying issues via a bottom-up approach, resolving them, and then testing if the full graph compilation works.\n## Tuning for Maximal Performance\nOnce you have succeeded in compiling your model, there are a wide range of controls for trying to squeeze out even better performance. In this section we will cover some of the available options. It should be noted that the additional performance gains from these options are usually a small fraction of the gains from the initial application of standard compilation.\n### Advanced Compiler Modes and Options\nThe [torch.compile API](https://docs.pytorch.org/docs/stable/generated/torch.compile.html) allows for tuning the compiler-backend behavior via via the `mode` and `options` parameters. There are dozens of knobs that can be applied and assessed. Some of the most notable ones are “reduce-overhead” — that optimizes more aggressively to further reduce the overhead of the kernel loading and Python interpreter, and “max-autotune” — the most aggressive optimization option that performs benchmarking of multiple kernel options before choosing the most efficient one. Both of these, particularly “max-autotune”, increase the compilation time, but usually result in more efficient graphs.\n### Varying the Compiler Backend\nThe default compiler backend is TorchInductor which supports a variety of target devices. You can specify the compiler backend via the _backend_ parameter of the [torch.compile API](https://docs.pytorch.org/docs/stable/generated/torch.compile.html). While other backends are unlikely to beat TorchInductor when running on NVIDIA GPUs, you may find them to perform better on other hardware devices (e.g., the [ _ipex_](https://github.com/intel/intel-extension-for-pytorch) backend includes optimizations that leverage the unique capabilities of Intel® CPUs).\n### Applying Modular Compilation\nWhile it is usually recommended to apply compilation to the entire model, there are times where the model can be broken into submodules that respond very differently to the compiler controls. For example, if your model contains one component that includes many tensors with dynamic shapes and another component that is static, you may find that compiling the first in “max-autotune-no-cudagraphs” mode and the second in “max-autotune” mode, results in maximum performance.\n### Compiling the Optimizer\nIn addition to compiling the model execution, as of PyTorch 2.2, you can further optimize your training workload by [compiling the optimizer](https://docs.pytorch.org/tutorials/recipes/compiling_optimizer.html). This will be demonstrated below.\n### New Compiler Features\nSince the initial release of torch.compile in PyTorch 2.0, each PyTorch release has included enhancements to the torch.compile offering. Sometimes released as “prototypes”, new features offerings challenge developers to extract even greater performance out of graph compilation. For example, the [PyTorch 2.7 release](https://pytorch.org/blog/pytorch-2-7/) included the [foreach_map](https://docs.pytorch.org/tutorials/recipes/foreach_map.html) prototype feature, the use of which we will demonstrate below.\n## Reducing Compilation Time\nWhile the initial compilation and warm-up time can be quite long compared to the subsequent training steps, it is usually negligible compared to the overall life of the model (i.e., the training or inference time). In some cases, however, the lengthy compilation time can become an issue. If the model is extremely large and we are tuning for optimal performance, compilation could take hours. If we are using our model in an inference server setup, the model start-up time could have a direct impact on the server response time and user experience.\nIn this section we cover two techniques for reducing model compilation time: [compile-time caching](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [regional compilation](https://docs.pytorch.org/tutorials/recipes/regional_compilation.html).\n### Compile Time Caching\nIn compile-time caching we upload the results of the local graph compilation to persistent storage. Every time we need to run the same model in the same runtime environment (e.g., same hardware and same library versions) we pull the cache state from persistent storage to the local disk, instead of compiling from scratch.\n### Regional Compilation\nRegional compilation relies on the fact that large models typically consist of computation blocks that are repeated multiple times. In regional compilation, torch.compile is applied to the repeating block, instead of the entire model. The result is a single, relatively small graph that is created and reused for each of the blocks.\n## How to Configure the TORCH_LOGS Environment Variable\nTorch compilation supports a wide variety of [logging controls](https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html#summary-of-torch-logs-options). While the log reports can be extremely useful for debugging issues and maximizing performance, it’s important to find the right balance where the logs are helpful but not excessive. In this post we propose using the following initial configuration and adapting as needed:\nbash\n\n```\nexport TORCH_LOGS=\"graph_breaks,recompiles,perf_hints\"\n```\n\n * “graph_breaks” — reports each time a graph-break is encountered (see above)\n * “recompiles” — reports each time a recompilation is performed along with the guard-violation that triggered it.\n * “perf_hints” — outputs performance logs from the inductor backend along with hints for additional optimizations\n\n\nNote that sometimes “perf_hints” will flood the console with unactionable messages, in which case you may opt to disable it.\n## A Toy PyTorch Model: Image Captioning\nTo demonstrate torch.compile in action, we define a toy image captioning model using the popular [Hugging Face](https://huggingface.co/) [transformers](https://huggingface.co/docs/transformers/en/index) library (version 4.54.1). Specifically, we define an image-to-text model using a [VisionEncoderDecoderModel](https://huggingface.co/docs/transformers/v4.53.3/en/model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel), with a [Vision Transformer (ViT)](https://huggingface.co/papers/2010.11929) encoder and a [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) decoder, and train it on a synthetic dataset of fixed-sized images and random sequences (“captions”) of variable length.\nWe begin by defining our image-to-text model:\npython\n\n```\nimport os, shutil, time, random, torch\nfrom transformers import (\n VisionEncoderDecoderModel,\n VisionEncoderDecoderConfig,\n AutoConfig\n)\n\ntorch.manual_seed(42)\nrandom.seed(42)\n\nBATCH_SIZE = 64\nNUM_WORKERS = 12\nNUM_TOKENS = 1024\nMAX_SEQ_LEN = 256\nPAD_ID = 0\nSTART_ID = 1\nEND_ID = 2\n\n\n# set up image-to-text model\ndef get_model():\n config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(\n encoder_config=AutoConfig.for_model(\"vit\"), # vit encoder\n decoder_config=AutoConfig.for_model(\"gpt2\") # gpt2 decoder\n )\n config.decoder.vocab_size = NUM_TOKENS\n config.decoder.use_cache = False\n config.decoder_start_token_id = START_ID\n config.pad_token_id = PAD_ID\n config.eos_token_id = END_ID\n config.max_length = MAX_SEQ_LEN\n\n model = VisionEncoderDecoderModel(config=config)\n\n # remove unused pooler\n model.encoder.pooler = None\n\n # uncomment to specify the loss function\n # from transformers.loss.loss_utils import ForCausalLMLoss\n # model.loss_function = ForCausalLMLoss\n return model\n```\n\nNext, we define a synthetic dataset that generates pairs of random images of fixed size and random sequences of variable size. We use a weighted distribution for the sequence length to mimic a scenario where the vast majority of sequences are short.\nGiven the varying length of the input captions, we require a strategy for dealing with dynamically shaped input. Here, we offer two alternatives, both of which use padding: padding to the maximum input length and padding to the length of the longest sequence in the batch, along with an option to align it to a given multiple. Please see our [previous post](https://towardsdatascience.com/optimizing-transformer-models-for-variable-length-input-sequences-19fb88fddf71/) for additional strategies for handling variable-length input sequences.\npython\n\n```\nfrom torch.utils.data import Dataset, DataLoader\nfrom functools import partial\n\n# A synthetic Dataset with random images and captions\nclass FakeDataset(Dataset):\n def __init__(self):\n self.length_dist = {\n 'short': {'range': (5, 32), 'weight': 0.90},\n 'medium': {'range': (33, 64), 'weight': 0.09},\n 'long': {'range': (65, 256), 'weight': 0.01}\n }\n super().__init__()\n\n def __len__(self):\n return 1000000\n\n def __getitem__(self, index):\n length_bin = random.choices(\n list(self.length_dist.keys()),\n weights=[d['weight'] for d in self.length_dist.values()],\n k=1\n )[0]\n\n range_start, range_end = self.length_dist[length_bin]['range']\n image = torch.randn(3, 224, 224)\n length = random.randint(range_start, range_end - 1)\n labels = torch.cat([torch.randint(1, NUM_TOKENS, (length,)),\n torch.tensor([END_ID])],\n dim=0)\n input_ids = torch.cat([torch.tensor([START_ID]),\n labels[:-1]],\n dim=0)\n return {\n 'image': image,\n 'input_ids': input_ids,\n 'labels': labels\n }\n\ndef pad_sequence(sequence, length, pad_val):\n return torch.nn.functional.pad(\n sequence,\n (0, length - sequence.shape[0]),\n value=pad_val\n )\n\ndef collate_with_padding(batch, pad_to_longest=False, align=None):\n padded_inputs = []\n padded_labels = []\n if pad_to_longest:\n pad_len = max([b['input_ids'].shape[0] for b in batch])\n if align:\n pad_len = ((pad_len + align - 1) // align) * align\n else:\n pad_len = MAX_SEQ_LEN\n\n for b in batch:\n input_ids = b['input_ids']\n labels = b['labels']\n padded_inputs.append(pad_sequence(input_ids, pad_len, PAD_ID))\n padded_labels.append(pad_sequence(labels, pad_len, -100))\n\n padded_inputs = torch.stack(padded_inputs, dim=0)\n padded_labels = torch.stack(padded_labels, dim=0)\n images = torch.stack([b['image'] for b in batch], dim=0)\n return {\n 'pixel_values': images,\n 'decoder_input_ids': padded_inputs,\n 'labels': padded_labels,\n 'decoder_attention_mask': (padded_inputs != PAD_ID)\n }\n\ndef get_dataloader(pad_to_longest=False, align=None):\n return DataLoader(\n dataset=FakeDataset(),\n batch_size=BATCH_SIZE,\n num_workers=NUM_WORKERS,\n collate_fn=partial(\n collate_with_padding,\n pad_to_longest=pad_to_longest,\n align=align\n )\n )\n```\n\nLast, we define our training step and main training function:\npython\n\n```\ndef copy_to_device(batch, device):\n return {\n key: val.to(device=device, non_blocking=True)\n for key, val in batch.items()\n }\n\ndef train_step(model, device, optimizer, batch):\n # copy data to device\n batch = copy_to_device(batch, device)\n optimizer.zero_grad()\n with torch.amp.autocast('cuda', dtype=torch.bfloat16):\n outputs = model(**batch)\n loss = outputs.loss\n loss.backward()\n optimizer.step()\n return loss\n\ndef train(local_rank=0, world_size=1, compile=False):\n # specify log settings\n torch._logging.set_logs(\n graph_breaks=True,\n recompiles=True,\n perf_hints=True\n )\n\n torch.cuda.set_device(local_rank)\n device = torch.cuda.current_device()\n\n if world_size > 1:\n # DDP setup\n import torch.distributed as dist\n from torch.nn.parallel import DistributedDataParallel as DDP\n os.environ['MASTER_ADDR'] = 'localhost'\n os.environ['MASTER_PORT'] = str(2222)\n dist.init_process_group('nccl', rank=local_rank,\n world_size=world_size)\n\n # configure pad_to_longest and optional alignment\n dataloader = get_dataloader(pad_to_longest=False, align=None)\n\n model = get_model()\n model = model.to(device)\n if world_size > 1:\n model = DDP(model, [local_rank])\n optimizer = torch.optim.Adam(model.parameters())\n\n if compile:\n # uncomment to run pre-compile warmup - required for some optimizations\n # batch = next(iter(dataloader))\n # train_step(model, device, optimizer, batch)\n model, optimizer = apply_compilation(model, optimizer)\n\n warmup = 20\n active = 100\n total_steps = warmup + active\n t0 = time.perf_counter()\n\n for idx, batch in enumerate(dataloader, start=1):\n # apply train step\n train_step(model, device, optimizer, batch)\n\n if idx == warmup:\n torch.cuda.synchronize()\n print(f'warmup time: {time.perf_counter()-t0}')\n t0 = time.perf_counter()\n elif idx == total_steps:\n break\n\n if local_rank == 0:\n torch.cuda.synchronize()\n total_time = time.perf_counter() - t0\n print(f'average throughput: {active / total_time}')\n\n if world_size > 1:\n dist.destroy_process_group()\n\n\nif __name__ == '__main__':\n # specify inductor cache dir\n inductor_cache_dir = '/tmp/inductor_cache'\n os.environ['TORCHINDUCTOR_CACHE_DIR'] = inductor_cache_dir\n\n # clean up compiler cache\n torch._dynamo.reset()\n shutil.rmtree(inductor_cache_dir, ignore_errors=True)\n\n world_size = 1\n torch.multiprocessing.spawn(\n fn=train,\n args=(world_size,),\n nprocs=world_size,\n join=True\n )\n```\n\n### Baseline Performance\nRunning the training script without compilation yields the following baseline performance results:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/08/2.webp)\nBaseline Model Performance (by Author)\nWe can see clearly that the collation strategy that reduces padding results in much better performance.\n## Applying Model Compilation\nIn this section we will apply torch compilation with different configurations and measure its impact on the training throughput. We will begin by applying compilation without dynamism, i.e., when padding all inputs to max sequence length. In the following section we will evaluate its impact in the case of inputs with dynamic shapes.\n### Model Compilation Step #1: Fixing Graph Breaks\nWe introduce the following compilation utility function and apply it to our model:\npython\n\n```\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True)\n return model, optimizer\n```\n\nThe _fullgraph_ setting ensures that compilation will fail whenever it encounters a graph break. Sure enough, our first compilation attempt results in an error coming from the transformer library. Here is a small snippet:\nshell-session\n\n```\nfrom user code: File \"/opt/pytorch/lib/python3.12/site-packages/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py\", line 574, in forward loss = self.loss_function( File \"/opt/pytorch/lib/python3.12/site-packages/transformers/modeling_utils.py\", line 5776, in loss_function\n```\n\nThe reason for this error is that when the [VisionEncoderDecoderModel](https://huggingface.co/docs/transformers/v4.53.3/en/model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel) loss function is not specified, the transformers library uses native Python code to determine what loss function to apply. This is easy to fix by specifying the model loss function, as follows:\npython\n\n```\nfrom transformers.loss.loss_utils import ForCausalLMLoss\nmodel.loss_function = ForCausalLMLoss\n```\n\nFollowing this fix, model compilation succeeds. The resultant throughput is 5.17 steps per second — a 66% speed-up over the baseline (fixed-input) throughput.\nNote that in the current scenario of a static graph, the compiler did not report any recompilations, but it did report the following _perf_hint_ :\nshell-session\n\n```\nI0805 13:37:52.406000 51587 torch/_inductor/codegen/simd.py:1976] [0/0] [__perf_hints] Reduction over non-contiguous dims.I0805 13:37:52.406000 51587 torch/_inductor/codegen/simd.py:1976] [0/0] [__perf_hints] Consider setting config.triton.tile_reductions to True.\n```\n\nHowever, applying the suggested configuration results in a compilation error, so we ignore it going forward.\n### Model Compilation Step #2: Tuning the Compiler Configuration\nLet’s try to increase the performance further by applying some of the advanced compilation controls. The code block below includes three alternative modifications:\npython\n\n```\n# reduce-overhead\nmodel = torch.compile(model, fullgraph=True, mode=\"reduce-overhead\")\n\n# max-autotune\nmodel = torch.compile(model, fullgraph=True, mode=\"max-autotune\")\n\n# shapes padding\nmodel = torch.compile(model, fullgraph=True, options={\"shape_padding\":True})\n```\n\nThe results are captured in the table below:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/08/3.webp)\ntorch.compile results (by Author)\nThe subsequent experiments in this section will be run with the “max-autotune” optimization.\n### Model Compilation Step #3: Compiling the Optimizer\nNext, we extend our solution to apply compilation to the optimizer. Since optimizer compilation currently requires graph-breaks, we apply it without the _fullgraph_ flag:\npython\n\n```\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True, mode=\"max-autotune\")\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nCompiling the optimizer further increases the throughput to 5.54 steps per second!!\nWhen compiling the optimizer, the following performance hint is printed:\nshell-session\n\n```\n will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.\n```\n\nThe proposal is to fix the addresses of gradient tensors and mark them. To implement the suggestion, we introduce the following two utility functions:\npython\n\n```\n# this replaces default optimizer.zero_grad() and verifies reuse\n# of same gradient tensors\ndef zero_grads(model):\n for p in model.parameters():\n if p.grad is not None:\n p.grad.zero_()\n\n# uses dynamo utility to mark each of the gradient tensors as static\ndef mark_static_address(optimizer):\n for group in optimizer.param_groups:\n for p in group['params']:\n if p.grad is not None:\n torch._dynamo.mark_static_address(p.grad)\n```\n\nThe updated training step appears below:\npython\n\n```\ndef train_step(model, device, optimizer, batch):\n # copy data to device\n batch = copy_to_device(batch, device)\n zero_grads(model)\n with torch.amp.autocast('cuda', dtype=torch.bfloat16):\n outputs = model(**batch)\n loss = outputs.loss\n loss.backward()\n mark_static_address(optimizer)\n optimizer.step()\n return loss\n```\n\nIn our case, implementing the performance hint decreases the throughput result to 5.32 steps per second — so we disregard it.\n### Model Compilation Step #4: Foreach Map Optimization\nConstantly be on the lookout for torch.compile enhancements and additions. Here we will apply [horizontal fusion with foreach_map](https://docs.pytorch.org/tutorials/recipes/foreach_map.html) — an optimization introduced in the latest PyTorch release — to the optimizer step. Using the utility functions from the [Foreach Map tutorial](https://docs.pytorch.org/tutorials/recipes/foreach_map.html), we create an optimized Adam optimizer step function, and apply it to our optimizer:\npython\n\n```\ndef get_compiled_adam_step(optimizer):\n compiled_adam = torch.compile(foreach_map_adam)\n inputs = get_inputs(optimizer)\n def compiled_adam_step():\n compiled_adam(*inputs)\n return compiled_adam_step\n\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True, mode=\"max-autotune\")\n optimizer.step = get_compiled_adam_step(optimizer)\n return model, optimizer\n```\n\nThis optimization requires use of the _zero_grads_ utility from above. It also requires that we run a warmup training step before compilation to populate all of the gradient tensors.\nThe modified optimizer step results in a reduced throughput of 5.28 steps per second. We presume that our toy model is too small to reap the benefit of the new compilation feature.\nOur best result, 5.54 steps per second, is 78% faster than our baseline result. Let’s see what happens when we extend our solution to multiple GPUs.\n### Model Compilation Step #5: Extending to DDP\nThe final step is to extend the training script to use all 8 GPUs. For this step we need to disable the _fullgraph_ setting since the cross-GPU gradient sharing requires graph-breaking communication calls.\nThe resultant throughput is 4.59 steps per second, nearly two times faster than our baseline result.\n### Results\nThe table below summarizes the results of our static-graph experiments:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/08/4.webp)\nStatic Graph Compilation Results (by Author)\nThus far, all of our experiments have assumed fixed-sized input tensors. Since the vast majority of input sequences are small, our graph is performing a huge amount of wasteful computation.\nIn the next section we will evaluate torch.compile when padding to variable-length inputs.\n## Dynamic Model Compilation\nIn this section we introduce dynamism into our toy model definition by padding the inputs sequences in each batch to the length of the longest sequence. In a previous section we described several strategies for compiling dynamic graphs. We will apply these strategies and assess their impact on the training throughput.\nThe experiments in this section were run on a single NVIDIA A100 GPU.\n### Option #1: Auto-Detect Dynamism\nThe default behavior (_dynamic_ =_None_) of torch.compile is to auto-detect dynamism and recompile the graph accordingly. When running in this setting, we indeed see the recompilation due to the variation in input size, but we also get the following print:\nshell-session\n\n```\nV0806 09:31:00.624000 175763 torch/_dynamo/guards.py:2997] [0/1] [__recompiles] - 0/1: ((decoder_input_ids.size()[1]*decoder_input_ids.size()[1]) % 8) != 0 # attn_output = torch.nn.functional.scaled_dot_product_attention( # transformers/integrations/sdpa_attention.py:89 in sdpa_attention_forward (_dynamo/utils.py:3284 in run_node)\n```\n\nThe source of this recompilation is the [scaled_dot_product_attention](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) operator, which requires that input shapes be aligned to multiples of eight for optimal use. To address this issue and avoid the recompilation, we modify our padding operation to pad to a multiple of eight.\nTo avoid the recompilation that is triggered by the variable-length inputs, we define the following utility and apply it to the input tensors:\npython\n\n```\ndef mark_dynamic(batch):\n for key in ['decoder_input_ids', 'labels', 'decoder_attention_mask']:\n torch._dynamo.mark_dynamic(batch[key], 1)\n\ndef train_step(model, device, optimizer, batch):\n # copy data to device\n batch = copy_to_device(batch, device)\n # mark inputs as dynamic to avoid recompilation\n mark_dynamic(batch)\n optimizer.zero_grad()\n with torch.amp.autocast('cuda', dtype=torch.bfloat16):\n outputs = model(**batch)\n loss = outputs.loss\n loss.backward()\n optimizer.step()\n return loss\n```\n\nThis option results in a throughput of 7.78 steps per second, 64% higher than the baseline throughput (4.73).\nAn additional speed-up is achieved when we apply the “max-autotune” mode — 8.13 steps per second.\n### Option #2: Dynamic Compilation\nAnother way to avoid recompilations is to call torch.compile with _dynamic_ =_True_ :\npython\n\n```\ndef apply_compilation(model, optimizer):\n model = torch.compile(model, fullgraph=True, dynamic=True)\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nThis results in a throughput of 7.77 steps per second. Since setting _dynamic_ =_True_ precludes the use of CUDA graphs, we attempt to optimize further by setting _mode_ =”max-autotune-no-cudagraphs”. This results in a throughput of 7.89 steps per second.\n### Option #3: Compile a Fixed Number of Static Graphs\nThe last option we explore is to set a fixed number of supported input shapes and compile a corresponding fixed number of static graphs. Since the default number of recompilations supported is eight, we program our collator to emit eight different tensor shapes by aligning the padding to multiples of 32. To force the recompilations, we set _dynamic_ =_False_.\nThe resultant throughputs are for 7.77 steps per second for the default mode and 8.04 for _mode_ =”max-autotune”.\nNote that this option may require a greater number of warmup steps to ensure that all shape variations are processed. (An alternative is to manually feed the model with all shape variations before starting the training loop.)\n### Modular Compilation\nSince our model naturally splits into two submodules — a static encoder and a dynamic decoder — it is tempting to explore the option of applying separate compilation to each component. Note that in an inference setting, it is essential to compile the encoder and decoder separately, since the encoder is called only once, while the decoder is called repeatedly in an auto-regressive loop.\npython\n\n```\ndef apply_compilation(model, optimizer):\n model.encoder = torch.compile(model.encoder, fullgraph=True)\n model.decoder = torch.compile(model.decoder, fullgraph=True)\n model.loss_function = torch.compile(model.loss_function, fullgraph=True)\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nThe result of this strategy is a throughput of 7.93, which is slightly higher than the result we got (in default mode) when compiling the entire model.\nOne advantage to this approach is the ability to tune the compilation controls to each submodule independently. For example, setting _mode_ =”max-autotune” to just the encoder, further increased the throughput to 8.04 steps per second.\n### Results\nWe summarize the results of our dynamic-graph experiments in the table below:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/08/5.webp)\nDynamic Graph Compilation Results (by Author)\nThe best result was 8.13 steps per second, 72% higher than the baseline result (4.73). It is likely that further tuning could result in additional gains.\nKeep in mind that the impact of torch.compile can vary greatly based on the details of the model and the runtime environment.\n## Reducing Compilation Time\nWe now turn our attention to the duration of the torch.compile warmup. We will assess the two optimizations discussed above, [compile-time caching](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [regional compilation](https://docs.pytorch.org/tutorials/recipes/regional_compilation.html). We limit our experiments to a single GPU. We use the default application of torch.compile and measure the duration of the first 20 training iterations.\n### Pre-Loading Compilation Cache\nIn the following demonstration of compile-time caching, we use an [Amazon S3](https://aws.amazon.com/s3/) bucket as our persistent storage location:\npython\n\n```\nimport boto3\n\nS3_BUCKET = \"\"\nS3_KEY = \"\"\n\ndef download_cache():\n s3_client = boto3.client('s3')\n t0 = time.perf_counter()\n try:\n response = s3_client.get_object(Bucket=S3_BUCKET, Key=S3_KEY)\n artifact_bytes = response['Body'].read()\n torch.compiler.load_cache_artifacts(artifact_bytes)\n print(f\"Cache restored. Time: {time.perf_counter()-t0} sec\")\n except:\n return False\n return True\n\ndef upload_cache():\n s3_client = boto3.client('s3')\n artifact_bytes, cache_info = torch.compiler.save_cache_artifacts()\n s3_client.put_object(\n Bucket=S3_BUCKET,\n Key=S3_KEY,\n Body=artifact_bytes\n )\n\n\nif __name__ == '__main__':\n # specify inductor cache dir\n inductor_cache_dir = '/tmp/inductor_cache'\n os.environ['TORCHINDUCTOR_CACHE_DIR'] = inductor_cache_dir\n\n # clean up compiler cache\n torch._dynamo.reset()\n shutil.rmtree(inductor_cache_dir, ignore_errors=True)\n\n # upload the compilation artifacts\n download_cache()\n \n # train the model\n train()\n\n # upload the compilation artifacts\n upload_cache()\n```\n\nThis method reduces the compilation warmup from 196 seconds to 56 seconds — a 3.5X speed-up.\n### Regional Compilation\nTo implement regional compilation, we apply compilation to the internal blocks of both the encoder and the decoder:\npython\n\n```\ndef apply_compilation(model, optimizer):\n model.encoder.encoder.layer = torch.nn.ModuleList(\n [torch.compile(layer, fullgraph=True)\n for layer in model.encoder.encoder.layer]\n )\n model.decoder.transformer.h = torch.nn.ModuleList(\n [torch.compile(layer, fullgraph=True)\n for layer in model.decoder.transformer.h]\n )\n model.loss_function = torch.compile(model.loss_function, fullgraph=True)\n optimizer.step = torch.compile(optimizer.step)\n return model, optimizer\n```\n\nThis change reduces the throughput from 7.78 steps per second to 7.61 steps per second. On the other hand, the compilation warmup drops from 196 seconds to 80 seconds — a 2.45X speed-up.\nIn the case of our toy model — which is extremely small by today’s standards — the gains we have demonstrated are modest. But for large models, these types of compilation-time optimization techniques could prove essential.\n## Summary\nAs AI/ML models grow in size to hundreds of billions or even trillions of parameters, optimizing their runtime performance becomes increasingly essential. For PyTorch models, torch.compile is one of the most powerful optimization tools at your disposal. This post has aimed to ease the adoption of torch.compile by addressing some of its intricacies and demonstrating its practical use. Some of the main techniques we covered were:\n * Reducing graph-breaks and recompilations\n * Tuning compilation settings to maximize performance gains\n * Effective use of the PyTorch logs\n * Top-down vs. bottom-up debugging strategies\n * Modular application of torch.compile\n * Reducing the duration of compilation warmup\n\n\nPyTorch compilation is a complex and nuanced topic. In this post we have covered just some of its many features. For more on the topic, be refer to the [official documentation](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html).\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657094227_7ymyo0.jpeg)\nWritten By\n## [Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[See all from Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Python](https://towardsdatascience.com/tag/python/)[Pytorch](https://towardsdatascience.com/tag/pytorch/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmaximizing-ai-ml-model-performance-with-pytorch-compilation%2F&title=Maximizing%20AI%2FML%20Model%20Performance%20with%20PyTorch%20Compilation)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmaximizing-ai-ml-model-performance-with-pytorch-compilation%2F&title=Maximizing%20AI%2FML%20Model%20Performance%20with%20PyTorch%20Compilation)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmaximizing-ai-ml-model-performance-with-pytorch-compilation%2F&text=Maximizing%20AI%2FML%20Model%20Performance%20with%20PyTorch%20Compilation)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/1WH6ndVAUD48g2Un06mZF7A.png)](https://towardsdatascience.com/time-series-transformations-selection-using-edaspy-d97e0dc1fcca/)\n## [Time series transformations selection using EDAspy](https://towardsdatascience.com/time-series-transformations-selection-using-edaspy-d97e0dc1fcca/)\n[](https://towardsdatascience.com/maximizing-ai-ml-model-performance-with-pytorch-compilation/)\nEasy way to find the optimum Time series transformations for a forecasting model in a Machine Learning project.\n[Vicente P. Soloviev](https://towardsdatascience.com/author/vicente-perez-soloviev/)\nNovember 25, 20204 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/12xkuAV6TKPdO8sXRPdlG6Q-scaled.jpeg)](https://towardsdatascience.com/want-to-break-into-data-science-start-building-db5ed24a7d77/)\n## [Want to break into data science? Start building](https://towardsdatascience.com/want-to-break-into-data-science-start-building-db5ed24a7d77/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWhy you should build first and ask questions later\n[Russell Pollari](https://towardsdatascience.com/author/russellpollari/)\nOctober 19, 20204 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/0cZC-iSUaaPWFJSDO-scaled.jpg)](https://towardsdatascience.com/how-to-select-loss-function-and-activation-function-for-classification-problems-b7f32c724507/)\n## [How to Select Loss Function and Activation Function for Classification Problems](https://towardsdatascience.com/how-to-select-loss-function-and-activation-function-for-classification-problems-b7f32c724507/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nUnderstanding how to choose activation function in the final layer and loss function for neural network\n[Ibrahim Kovan](https://towardsdatascience.com/author/ibrahimkovan/)\nApril 14, 20226 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/0BtgmFoE9ahrClxPC-scaled.jpg)](https://towardsdatascience.com/three-main-components-of-machine-learning-modeling-c9186658f7fe/)\n## [Three Main Components of Machine Learning Modeling](https://towardsdatascience.com/three-main-components-of-machine-learning-modeling-c9186658f7fe/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThe art of learning through data\n[Abiodun Olaoye](https://towardsdatascience.com/author/aolaoye/)\nOctober 24, 20224 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/11/0JUGJBhaqDV6cSO88-scaled.jpg)](https://towardsdatascience.com/teaching-is-hard-how-to-train-small-models-and-outperforming-large-counterparts-f131f9d463e1/)\n## [Teaching is Hard: How to Train Small Models and Outperforming Large Counterparts](https://towardsdatascience.com/teaching-is-hard-how-to-train-small-models-and-outperforming-large-counterparts-f131f9d463e1/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDistilling the knowledge of a large model is complex but a new method shows incredible performances\n[Salvatore Raieli](https://towardsdatascience.com/author/salvatore-raieli/)\nNovember 11, 202311 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/1XrpyB4mf16KRM7lUPdia9Q.jpeg)](https://towardsdatascience.com/exploring-reddit-wallstreetbets-posts-data-bed613386029/)\n## [Exploring Reddit WallStreetBets Posts Data](https://towardsdatascience.com/exploring-reddit-wallstreetbets-posts-data-bed613386029/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nEDA and Sentiment Analysis of Reddit Data\n[Sadrach Pierre, Ph.D.](https://towardsdatascience.com/author/spierre91/)\nFebruary 8, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Spurious-regression.png)](https://towardsdatascience.com/linear-regression-in-time-series-sources-of-spurious-regression/)\n## [Linear Regression in Time Series: Sources of Spurious Regression](https://towardsdatascience.com/linear-regression-in-time-series-sources-of-spurious-regression/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWhy does the autocorrelation of the errors term matter?\n[JUNIOR JUMBONG](https://towardsdatascience.com/author/jumbongjunior1999/)\nMarch 10, 202513 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1aayydJi_93qXxk7eKd-Znw.png)](https://towardsdatascience.com/how-can-machine-learning-help-visually-impaired-people-4fcdc76816b2/)\n## [How machine learning can help visually impaired people](https://towardsdatascience.com/how-can-machine-learning-help-visually-impaired-people-4fcdc76816b2/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAn outdoor obstacle identification project proposal for blind people with DeepLabv3+\n[Shuyang Xiang](https://towardsdatascience.com/author/vanillaxiangshuyang/)\nJanuary 4, 20226 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/04/1lSA4IYkqrIgpM1SyuQE4Dg.png)](https://towardsdatascience.com/the-end-of-anchors-improving-object-detection-models-and-annotations-73828c7b39f6/)\n## [The End of Anchors - Improving Object Detection Models and Annotations](https://towardsdatascience.com/the-end-of-anchors-improving-object-detection-models-and-annotations-73828c7b39f6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nIf you have ever had to tinker with anchor boxes, you were probably frustrated, confused and saying to yourself, \"There must be another...\n[Anders Christiansen](https://towardsdatascience.com/author/andersasac/)\nApril 2, 20194 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n", "body_base_path": "attachments/cec6d4ebea83/pages/", "body_markdown_path": "attachments/cec6d4ebea83/pages/archived-body.md", "local_html": "attachments/cec6d4ebea83/pages/archived-body.html", "body_source": "prior_capture" }