{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Chaim Rand",
  "title_listed": "Optimizing Data Transfer in AI/ML Workloads",
  "keywords": "",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/optimizing-data-transfer-in-ai-ml-workloads/",
  "publish_date": "2026-01-03",
  "corpus_stage": "",
  "notes": "",
  "id": "4267c8346f9a",
  "collected_at": "2026-10-02T11:03:41",
  "title": "Optimizing Data Transfer in AI/ML Workloads | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Optimizing Data Transfer in AI/ML Workloads\nA deep dive on data transfer bottlenecks, their identification, and their resolution with the help of NVIDIA Nsight™ Systems\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nJanuary 3, 202614 min read\n![](../images/000.jpeg)\n[Photo](https://unsplash.com/photos/green-trees-beside-river-during-daytime-ZPQdp0jSaP8) by Daniel Seßler on Unsplash\nIn a typical AI/ML workload, a deep learning model is executed on a dedicated GPU accelerator using input data batches it receives from a CPU host. Ideally, the GPU — the more expensive resource — should be maximally utilized, with minimal periods of idle time. In particular, this means that every time it completes its execution on a batch, the subsequent batch will be “ripe and ready” for processing. When this does not happen, the GPU idles while waiting for input data — a common performance bottleneck often referred to as GPU starvation.\nIn previous posts, (e.g., see [A Caching Strategy for Identifying Bottlenecks on the Data Input Pipeline](https://towardsdatascience.com/a-caching-strategy-for-identifying-bottlenecks-on-the-data-input-pipeline/)), we discussed common causes of this issue, including: inefficient storage retrieval, CPU resource exhaustion, and host-to-device transfer bottlenecks. In this post, we zoom in on data transfer bottlenecks and revisit their identification and resolution — this time with the help of [NVIDIA Nsight™ Systems (nsys)](https://docs.nvidia.com/nsight-systems/index.html), a performance profiler designed for analyzing the system-wide activity of workloads running on NVIDIA GPUs.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## NVIDIA Nsight vs. PyTorch Profiler\nReaders familiar with our work may be surprised at the mention of NVIDIA Nsight profiler rather than [PyTorch Profiler](https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html). In our previous posts we have advocated strongly for the use of PyTorch Profiler in AI/ML model development as a tool for identifying and optimizing runtime performance. Time and again, we have demonstrated its application to a wide variety of performance issues. Its use does not require any special installations and can be run without special OS permissions. NVIDIA Nsight profiler, on the other hand, requires a [dedicated system setup](https://docs.nvidia.com/nsight-systems/InstallationGuide/) (or [a dedicated NVIDIA container](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/devtools/containers/nsight-systems-cli)) and — for some of its features — elevated permissions, making its use less accessible and more complicated than PyTorch Profiler.\nThe two profilers differ in their focus: PyTorch profiler is a  _framework_ profiler tightly coupled with PyTorch and heavily focused on how models use the PyTorch software stack and supporting libraries. NVIDIA Nsight profiler is a  _system_ -level profiler; it does not know the details of the model being run or which framework is being used, but rather how the components of the entire system are being used and utilized. While PyTorch Profiler excels at tracing the low-level operations of a PyTorch model execution, nsys provides a detailed view of the activities of the entire system (GPU hardware, CUDA streams, OS interrupts, Network, PCIe, etc.). For many performance issues PyTorch profiler is sufficient for identifying and solving the source of the bottleneck; But some situations call for nsys profiler, the “big guns”, for deriving deeper insights into the inner workings of the underlying system.\nIn this post we intend to demonstrate some of the unique capabilities of nsys profiler and their application to the common data-transfer bottleneck.\n## Outline\nTo facilitate our discussion we will define a toy ML workload with a data-transfer performance bottleneck and proceed to introduce a number of successive optimizations in an attempt to solve it. Throughout the process, we will use the nsys profiler in order to analyze the system performance and assess the impact of the code modifications.\n## Setup\nWe will run our experiments on an [Amazon EC2 g6e.2xlarge](https://aws.amazon.com/ec2/instance-types/g6e/) instance with an [NVIDIA L40S](https://www.nvidia.com/en-eu/data-center/l40s/) GPU running an [AWS Deep Learning (Ubuntu 24.04) AMI](https://docs.aws.amazon.com/dlami/latest/devguide/appendix-ami-release-notes.html) with [PyTorch (2.8)](https://pytorch.org/get-started/locally/). To install the nsys-cli profiler (version 2025.6.1) we follow the [official NVIDIA guidelines](https://developer.nvidia.com/nsight-systems/get-started):\nbash\n\n```\nwget https://developer.nvidia.com/downloads/assets/tools/secure/nsight-systems/2025_6/NsightSystems-linux-cli-public-2025.6.1.190-3689520.deb\nsudo apt install ./NsightSystems-linux-cli-public-2025.6.1.190-3689520.deb\n```\n\nThe [NVIDIA Tools Extension](https://docs.nvidia.com/tools-extension/index.html) (NVTX) library allows us to annotate our code with human-readable labels to increase the readability and comprehension of the performance trace. While PyTorch offers built-in NVTX support via its [torch.cuda.nvtx](https://docs.pytorch.org/docs/stable/cuda.html#nvidia-tools-extension-nvtx) APIs, we will use the standalone [nvtx](https://pypi.org/project/nvtx/) package (version 0.2.14) which supports color-coding the trace timeline for better visual analysis:\nbash\n\n```\npip install nvtx\n```\n\n## Disclaimers\nThe code we will share is intended for demonstrative purposes; please do not rely on its correctness or optimality. Please do not interpret our use of any library, tool, or platform, as an endorsement of its use. The impact of the optimizations we will cover can vary greatly based on the details of the model and the runtime environment. Please be sure to assess their effect on your own use case before integrating their use.\nMany thanks to [Yitzhak Levi](https://www.linkedin.com/in/yitzhak-levi-49a217201/) and [Gilad Wasserman](https://www.linkedin.com/in/gilad-wasserman-992530268/?originalSubdomain=il) for their contributions to this post.\n## A Toy PyTorch Model\nWe introduce a training script intentionally designed to consist of a bottleneck on the data-input pipeline.\nIn the code block below we define a simple image classification model with a [ResNet-18](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18) backbone.\npython\n\n```\nimport time, torch, torchvision\n\nDEVICE = \"cuda\"\nmodel = torchvision.models.resnet18().to(DEVICE).train()\noptimizer = torch.optim.Adam(model.parameters())\n```\n\nNext, we define a synthetic dataset which we will use to train our toy model.\npython\n\n```\nfrom torch.utils.data import Dataset, DataLoader\n\nWARMUP_STEPS = 10\nPROFILE_STEPS = 3\nCOOLDOWN_STEPS = 1\nTOTAL_STEPS = WARMUP_STEPS + PROFILE_STEPS + COOLDOWN_STEPS\nBATCH_SIZE = 64\nTOTAL_SAMPLES = TOTAL_STEPS * BATCH_SIZE\nIMG_SIZE = 512\n\n# A synthetic Dataset with random images and labels\nclass FakeDataset(Dataset):\n\n    def __len__(self):\n        return TOTAL_SAMPLES\n\n    def __getitem__(self, index):\n        img = torch.randn((3, IMG_SIZE, IMG_SIZE))\n        label = torch.tensor(index % 10)\n        return img, label\n\ntrain_loader = DataLoader(\n    FakeDataset(),\n    batch_size=BATCH_SIZE\n)\n```\n\nLastly, we define a standard training step programmed to run nsys-profiler for three steps using the [torch.cuda.profiler.start](https://github.com/pytorch/pytorch/blob/v2.9.0/torch/cuda/profiler.py#L50) and [stop](https://github.com/pytorch/pytorch/blob/v2.9.0/torch/cuda/profiler.py#L59) commands — intended for use in conjunction with the [nsys cli](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#profiling-from-the-cli). We highlight the components of the training step using the [nvtx.annotate](https://nvtx.readthedocs.io/en/latest/index.html) utility. Please refer to the official documentation for more details on [profiling with nsys in PyTorch](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#pytorch-profiling).\npython\n\n```\nimport nvtx\nfrom torch.cuda import profiler\n\ndef copy_data(batch):\n    data, targets = batch\n    data_gpu = data.to(DEVICE)\n    targets_gpu = targets.to(DEVICE)\n    return data_gpu, targets_gpu\n\n\ndef compute_step(model, batch, optimizer):\n    data, targets = batch\n    output = model(data)\n    loss = torch.nn.functional.cross_entropy(output, targets)\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n    return loss\n\n\ndata_iter = iter(train_loader)\n\nfor i in range(TOTAL_STEPS):\n\n    if i == WARMUP_STEPS:\n        # start nsys profiler\n        torch.cuda.synchronize()\n        start_time = time.perf_counter()\n        profiler.start()\n    elif i == WARMUP_STEPS + PROFILE_STEPS:\n        # stop nsys profiler\n        torch.cuda.synchronize()\n        profiler.stop()\n        end_time = time.perf_counter()\n\n    with nvtx.annotate(f\"Batch {i}\", color=\"blue\"):\n        with nvtx.annotate(\"get batch\", color=\"red\"):\n            batch = next(data_iter)\n        with nvtx.annotate(\"copy batch\", color=\"yellow\"):\n            batch = copy_data(batch)\n        with nvtx.annotate(\"Compute\", color=\"green\"):\n            compute_step(model, batch, optimizer)\n\ntotal_time = end_time - start_time\nthroughput = PROFILE_STEPS / total_time\nprint(f\"Throughput: {throughput:.2f} steps/sec\")\n```\n\nWe run our script using the  _cudaProfilerApi_ option to start and stop the profiler programmatically. Please see the official documentation for full details on [profiling from the nsys cli](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#profiling-from-the-cli).\nbash\n\n```\nnsys profile \\\n  --capture-range=cudaProfilerApi \\\n  --trace=cuda,nvtx,osrt \\\n  --output=baseline \\\n  python train.py\n```\n\nThis results in a  _baseline.nsys-rep_ trace file that we copy over to our development machine for analysis.\nIn order to draw a comparison to PyTorch profiler, we define an alternative training loop programmed with [PyTorch Profiler](https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) and annotated with the [torch.profiler.record_function](https://pytorch.org/tutorials/beginner/profiler.html#performance-debugging-using-profiler) utility:\npython\n\n```\nfrom torch.profiler import (\n    profile, record_function, schedule, tensorboard_trace_handler\n)\n\nwith profile(\n    schedule=schedule(wait=0, warmup=WARMUP_STEPS, \n                      active=PROFILE_STEPS, repeat=1),\n    on_trace_ready=tensorboard_trace_handler('./baseline'),\n    record_shapes=True,\n    with_stack=True\n) as prof:\n    for i in range(TOTAL_STEPS):\n        with record_function(\"get batch\"):\n            batch = next(data_iter)\n        with record_function(\"copy batch\"):\n            batch = copy_data(batch)\n        with record_function(\"compute\"):\n            compute_step(model, batch, optimizer)\n        prof.step()\n```\n\nThe throughput of our baseline experiment is 2.97 steps-per-second. In the next sections we will use the profile traces to identify performance bottlenecks in our training step and try to improve on this result.\n## Baseline Performance Analysis\nTo analyze the resultant nsys trace file, we open it in the [Nsight Systems GUI application](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#opening-command-line-results-files-for-visualization). In the image below we zoom in on the timeline of two of the training steps captured by the profiler:\n![](../images/001.webp)\nBaseline Nsight Systems Profiler Trace (by Author)\nThe trace contains a wealth of information, just a subset of which we will touch on in this post. Please see the nsys documentation for additional functionalities and features.\nThe timeline is divided into two parts: the CUDA section which reports GPU activity and the threads section which reports the CPU activity. The CUDA section makes a clear distinction between the GPU kernel (compute) activity (90.9%) and memory activity (9.1%). The top bars in each section report the utilization of each of the resources and both sections include an NVTX section with the colored annotations we included in our training step. We note the following observations:\n  1. The GPU is idle for roughly 50% of each training step. This can be seen by the portion of time taken by each batch (in blue) in the GPU NVTX bar and the large blocks of whitespace in between them.\n  2. The GPU activity for each batch starts immediately after the “get batch” activity has completed on the CPU. It starts with the host-to-device memory copy, marked in light green and continues with the kernel computations, marked in light blue.\n  3. Once the CPU has launched the GPU memory and compute commands for batch  _N_ , it proceeds to the next batch in the training loop — leading to a partial overlap of batch  _N+1_ on the CPU with batch  _N_ on the GPU.\n  4. The vast majority of the CPU thread is spent on the “get batch” activity. This constitutes the primary bottleneck in our baseline experiment.\n\n\nThe profiling trace points to a clear culprit — the dataloader. By default, PyTorch performs [single process data loading](https://docs.pytorch.org/docs/stable/data.html#single-process-data-loading-default) — a single CPU process is used to load the next data input batch, copy it to the GPU, and launch the compute kernels — all in a sequential manner. This typically results in severe under-utilization of the CPU resources by: 1) limiting dataloading to just a single process, and 2) making the loading of the next batch contingent on the completion of the CPU processing (i.e., kernel loading) of the previous batch. Our irresponsible use of our CPU resources has resulted in our GPU being starved for input data.\nThe same conclusion could have been reached using PyTorch Profiler trace shown below:\n![](../images/body-0f5ff7cf8a0e6914397a.webp)\nBaseline PyTorch Profiler Trace (by Author)\nHere too, we can see long periods of GPU underutilization that are caused by the long “get batch” blocks on the CPU side.\n### Optimization 1: Multi-Process Data Loading\nThe first step is to modify the data input pipeline to use [multi-process data loading](https://docs.pytorch.org/docs/stable/data.html#multi-process-data-loading). We set the number of workers to match the 8 vCPUs available on our [Amazon EC2 g6e.2xlarge](https://aws.amazon.com/ec2/instance-types/g6e/) instance. In a real-world scenario, this value should be tuned for optimal throughput:\npython\n\n```\nNUM_WORKERS = 8\n\ntrain_loader = DataLoader(\n    FakeDataset(),\n    batch_size=BATCH_SIZE,\n    num_workers=NUM_WORKERS\n)\n```\n\nFollowing this change our throughput jumps to 4.81 steps per second — a 62% improvement over our baseline result. The corresponding nsys profiler trace is shown below:\n![](../images/body-472a8930f1a689e226e4.webp)\nMultiproc Dataloading Nsight Systems Profiler Timeline (by Author)\nNote that the red “get batch” segment has become just a tiny sliver of each step in the NVTX bar. In its place, the yellow “copy batch” block now takes center stage. As a result of our use of multi-process dataloading, there is now always a new batch ready for processing — but can we do better?\nTaking a closer look at the GPU section we see that there is still a significant portion (~290 milliseconds) of idle time in between the memory operation and the kernel compute. This idle time is perfectly aligned with an “munmap” operation in the OS runtime bar. The “munmap” block is a CPU-side memory cleanup operation performed just after the CUDA memory copy is complete. It occurs at the tail-end of the long yellow “copy batch” operation. The compute kernels are launched onto the GPU only after the memory cleanup has completed. This is a clear pattern of synchronous host-to-device memory copy: The CPU cannot proceed with kernel loading until the data copy operation has been fully completed and the GPU stays idle until the CPU loads the kernels.\nThe PyTorch profiler trace shows the same GPU idle time but it does not provide the same “munmap” hint. This is our first example of the advantage of the system-wide visibility of the nsys profiler.\n![](../images/body-1fa661cc2f05293e8831.webp)\nMultiproc Dataloading PyTorch Profiler Trace (by Author)\nWith our finding of the data-copy performance bottleneck in hand, we proceed to our next optimization.\n### Optimization 2: Asynchronous Data Transfer\nThe solution to the bottleneck we have found is to program our training step to load [data asynchronously](https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html#enable-asynchronous-data-loading-and-augmentation). This enables the CPU to launch the compute kernels immediately after sending the memory copy command — without waiting for the memory copy to be completed. This way the GPU can begin processing the kernels as soon as the CUDA memory copy is done. Enabling asynchronous data copy requires two changes: First we must program the dataloader to use [pinned memory](https://docs.pytorch.org/docs/stable/data.html#memory-pinning) (instead of pageable memory), and second, we must pass `non_blocking=True` argument to the [to()](https://docs.pytorch.org/docs/stable/generated/torch.Tensor.to.html#torch.Tensor.to) operations:\npython\n\n```\nNUM_WORKERS = 8\nASYNC_DATATRANSFER = True\n\n\ntrain_loader = DataLoader(\n    FakeDataset(),\n    batch_size=BATCH_SIZE,\n    num_workers=NUM_WORKERS,\n    pin_memory=ASYNC_DATATRANSFER\n)\n\ndef copy_data(batch):\n    data, targets = batch\n    data_gpu = data.to(DEVICE, non_blocking=ASYNC_DATATRANSFER)\n    targets_gpu = targets.to(DEVICE, non_blocking=ASYNC_DATATRANSFER)\n    return data_gpu, targets_gpu\n```\n\nUsing asynchronous dataloading results in a throughput of 5.91 steps per second — an additional 23% improvement and 99% improvement overall. The resultant profiling trace is shown below:\n![](../images/body-692f5fb276203f1c9eb8.webp)\nAsync Dataloading Nsight Systems Profiler Timeline (by Author)\nWe now see all of the CPU operations bunched together at the beginning of the trace. We have removed all performance obstacles on the CPU side allowing it to freely load the data and kernels to the GPU. In the GPU section, we see continuous activity without any idle time. We do, however, see a clear separation between CUDA memory activities (in light green) and CUDA kernel activities (in light blue). PyTorch profiler, in contrast, does not make this distinction clear. This is another advantage of the hardware-centric profiler and, in the case of our toy experiment, is what informs the next steps of our optimization.\n![](../images/body-7ad5d8766df1d72fff6b.webp)\nAsync Dataloading PyTorch Profiler Trace (by Author)\n### Optimization 3: Pipelining With CUDA Streams\nOur final optimizations derive from the fact that modern GPUs, such as the [NVIDIA L40S](https://www.nvidia.com/en-eu/data-center/l40s/), use independent engines for copying memory (the DMA) and executing compute kernels (the SMs). We can take advantage of this by parallelizing the distinct memory and kernel activities we saw in the nsys profiler trace. We will program this through the use of CUDA streams.\nIn a [previous post](https://towardsdatascience.com/pipelining-ai-ml-training-workloads-with-cuda-streams/), we expanded on the opportunity for optimizing AI/ML workloads using CUDA Streams. Here, we apply a similar pipelining strategy: We define two distinct “copy” and “compute” CUDA streams and program the “copy” stream to copy batch  _N+1_ at the same time that the “compute” stream is processing batch  _N_ :\npython\n\n```\n# define two CUDA streams\ncompute_stream = torch.cuda.Stream()\ncopy_stream = torch.cuda.Stream()\n\n\n# extract first batch\nnext_batch = next(data_iter)\nwith torch.cuda.stream(copy_stream):\n    next_batch = copy_data(next_batch)\n\nfor i in range(TOTAL_STEPS):\n\n    if i == WARMUP_STEPS:\n        torch.cuda.synchronize()\n        start_time = time.perf_counter()\n        profiler.start()\n    elif i == WARMUP_STEPS + PROFILE_STEPS:\n        torch.cuda.synchronize()\n        profiler.stop()\n        end_time = time.perf_counter()\n\n    with nvtx.annotate(f\"Batch {i}\", color=\"blue\"):\n        # wait for copy stream to complete copy of batch N\n        compute_stream.wait_stream(copy_stream)\n        batch = next_batch\n\n        # execute model on batch N+1 compute stream\n        try:\n            with nvtx.annotate(\"get batch\", color=\"red\"):\n                next_batch = next(data_iter)\n            with torch.cuda.stream(copy_stream):\n                with nvtx.annotate(\"copy batch\", color=\"yellow\"):\n                    next_batch = copy_data(next_batch)\n        except:\n            # reached end of dataset\n            next_batch = None\n\n        # execute model on batch N compute stream\n        with torch.cuda.stream(compute_stream):\n            with nvtx.annotate(\"Compute\", color=\"green\"):\n                compute_step(model, batch, optimizer)\n\ntotal_time = end_time - start_time\nthroughput = PROFILE_STEPS / total_time\nprint(f\"Throughput: {throughput:.2f} steps/sec\")\n```\n\nThis optimization results in a throughput of 6.44 steps per second — a 9% improvement over our previous experiment. We note that the impact of this optimization is capped by the duration of the longer of the two operation types. In our previous profile trace, the memory block took 15.5 milliseconds and the kernel block took 155 milliseconds. In the current profile trace, the entire GPU steps takes 155 milliseconds, which means that the memory copy time is completed hidden by the kernel compute time and that our optimization reaches the maximum possible result.\nThe use of the CUDA streams and its impact on GPU utilization can be seen in the traces of both profilers:\n![](../images/body-b7be80f5210035a22279.webp)\nPipelined Nsight Systems Profiler Timeline (by Author)\n![](../images/body-253a8af7cce44419b648.webp)\nPipelined PyTorch Profiler Trace (by Author)\n### Optimization 4: Prefetching to CUDA\nFor our final step, we move the data copying from the main training loop process to the data loading process: Rather than explicitly calling the copy function inside the training loop, we assume that the batches returned from the data iterator are already placed on the GPU.\nIn the code block below, we wrap our dataloader with a CUDA-prefetching iterator class. Note, that this is a simplified implementation intended for the purposes of demonstration. More work may be required for more complex scenarios (e.g., DDP training). Alternatively, you may consider a third-party implementation such as [torchtnt.utils.data.data_prefetcher.CudaDataPrefetcher](https://github.com/meta-pytorch/tnt/blob/master/torchtnt/utils/data/data_prefetcher.py#L17):\npython\n\n```\nclass DataPrefetcher:\n    def __init__(self, loader):\n        self.loader = iter(loader)\n        self.stream = torch.cuda.Stream()\n        self.next_batch = None\n        self.preload()\n\n    def preload(self):\n        try:\n            data, targets = next(self.loader)\n\n            with torch.cuda.stream(self.stream):\n                with nvtx.annotate(\"copy batch\", color=\"yellow\"):\n                    next_data = data.to(DEVICE, non_blocking=True)\n                    next_targets = targets.to(DEVICE, non_blocking=True)\n            self.next_batch = (next_data, next_targets)        \n        except:\n            self.next_batch = (None, None)\n\n    def __iter__(self):\n        return self\n\n    def __next__(self):\n        torch.cuda.current_stream().wait_stream(self.stream)\n        data, targets = self.next_batch\n        self.preload()\n        return data, targets\n\n\ndata_iter = DataPrefetcher(train_loader)\n\nfor i in range(TOTAL_STEPS):\n    if i == WARMUP_STEPS:\n        torch.cuda.synchronize()\n        start_time = time.perf_counter()\n        profiler.start()\n    elif i == WARMUP_STEPS + PROFILE_STEPS:\n        torch.cuda.synchronize()\n        profiler.stop()\n        end_time = time.perf_counter()\n\n    with nvtx.annotate(f\"Batch {i}\", color=\"blue\"):\n        with nvtx.annotate(\"get batch\", color=\"red\"):\n            batch = next(data_iter)\n        with nvtx.annotate(\"Compute\", color=\"green\"):\n            loss = compute_step(model, batch, optimizer)\n\ntotal_time = end_time - start_time\nthroughput = PROFILE_STEPS / total_time\nprint(f\"Throughput: {throughput:.2f} steps/sec\")\n```\n\nThis optimization results in a throughput of 6.44 steps per second — the same as our previous experiment. This should not surprise us since we have already seen that the throughput is bound by the 155 millisecond GPU compute and our optimization has not done anything to reduce the kernel compute time.\nMore generally, despite the removal of the copy call from the main loop, you may have a hard time finding a situation where this will have a meaningful impact on performance since the call is already being called asynchronously. However, given the minimal changes to the training loop, you may find this solution to be cleaner and/or to be more applicable for use with high-level libraries that do not enable fine-grained control of the training loop.\nUnsurprisingly, the profile traces for this experiment appear nearly identical to the previous ones. The main difference is the placement of the yellow “copy data” block in the NVTX row of the CPU section.\n![](../images/body-71547445de4e25630af3.webp)\nData Prefetching Nsight Systems Profiler Timeline (by Author)\n![](../images/body-33ff356513ed06bb2ed2.webp)\nData Prefetching PyTorch Profiler Trace (by Author)\n## Results\nThe table below summarizes the results of our experiments:\n![](../images/body-883c166e9dc24f315f91.webp)\nExperiment Results (by Author)\nThe optimizations, which were driven by the use of Nsight Systems profiler, resulted in an overall increase of  _2.17X_ to the runtime performance.\n### Summary\nGPU starvation is a common performance bottleneck that can have a devastating impact on the efficiency and costs of AI/ML workloads. In this post, we demonstrated how to use the Nsight Systems profiler to study the causes of the performance bottleneck and take informed steps towards their resolution. Along the way, we emphasized the unique capabilities of Nsight Systems profiler when compared to the built-in framework-centric PyTorch Profiler — specifically its deep system-level visibility.\nOur focus, in this post has been on the host-to-device data copy that typically occurs at the beginning of the training step. However, data-transfer bottlenecks can appear at different stages of training. In a sequel to this post we intend to repeat our nsys profiling analysis on data copies going in the opposite direction — from the device to the host. Stay tuned!\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-97f5483d57a9b8741146.jpeg)\nWritten By\n## [Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[See all from Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Model Optimization](https://towardsdatascience.com/tag/model-optimization/)[Pytorch](https://towardsdatascience.com/tag/pytorch/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-data-transfer-in-ai-ml-workloads%2F&title=Optimizing%20Data%20Transfer%20in%20AI%2FML%20Workloads)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-data-transfer-in-ai-ml-workloads%2F&title=Optimizing%20Data%20Transfer%20in%20AI%2FML%20Workloads)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-data-transfer-in-ai-ml-workloads%2F&text=Optimizing%20Data%20Transfer%20in%20AI%2FML%20Workloads)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/002.jpg)](https://towardsdatascience.com/paper-walkthrough-u-net-98877a2cd33c/)\n## [Paper Walkthrough: U-Net](https://towardsdatascience.com/paper-walkthrough-u-net-98877a2cd33c/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA PyTorch implementation on one of the most popular semantic segmentation models.\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nSeptember 20, 202418 min read\n  * [![](../images/003.jpeg)](https://towardsdatascience.com/simple-mathematical-derivation-of-bias-variance-error-4ab223f28791/)\n## [Simple mathematical derivation of bias-variance error](https://towardsdatascience.com/simple-mathematical-derivation-of-bias-variance-error-4ab223f28791/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n1. Introduction\n[Damian Ejlli](https://towardsdatascience.com/author/damian.ejlli/)\nJuly 29, 202110 min read\n  * [![](../images/004.jpg)](https://towardsdatascience.com/hybrid-discrete-continuous-geometric-deep-learning-2e5871293184/)\n## [Hybrid Discrete-Continuous Geometric Deep Learning](https://towardsdatascience.com/hybrid-discrete-continuous-geometric-deep-learning-2e5871293184/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nScalable and equivariant spherical CNNs by DISCO convolutions\n[Jason McEwen](https://towardsdatascience.com/author/jasonmcewen/)\nMarch 15, 20236 min read\n  * [![](../images/005.jpg)](https://towardsdatascience.com/overcoming-ml-data-preprocessing-bottlenecks-with-grpc-ca30fdc01bee/)\n## [Overcoming ML Data Preprocessing Bottlenecks With Auxiliary Workers and gRPC](https://towardsdatascience.com/overcoming-ml-data-preprocessing-bottlenecks-with-grpc-ca30fdc01bee/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA Simple Solution for Increasing Resource Utilization and Accelerating Training\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nMay 31, 202214 min read\n  * [![](../images/006.jpg)](https://towardsdatascience.com/how-to-differentiate-gradient-descent-objective-function-in-3-simple-steps-b9d58567d387/)\n## [How to Differentiate Gradient Descent Objective Function in 3 simple steps](https://towardsdatascience.com/how-to-differentiate-gradient-descent-objective-function-in-3-simple-steps-b9d58567d387/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nGrab a pen and paper\n[Łukasz Gebel](https://towardsdatascience.com/author/luk.gebel/)\nAugust 14, 20205 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/)\n## [Stop Returning Flat Text from a PDF: The Relational Tables RAG Needs](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nEnterprise Document Intelligence [Vol.1 #5B] - One PDF in, a relational set of DataFrames out: lines, pages, TOC, images, cross-references, captions, spans, and a parsing summary\n[Kezhan Shi](https://towardsdatascience.com/author/kezhanshi/)\nJune 11, 202625 min read\n  * [![](../images/008.jpeg)](https://towardsdatascience.com/the-best-data-science-bootcamp-for-startups-in-the-world-e9ae2b4ce6e6/)\n## [The Best Data Science Bootcamp for Startups in the World](https://towardsdatascience.com/the-best-data-science-bootcamp-for-startups-in-the-world-e9ae2b4ce6e6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhy I am joining Le Wagon, the best Data Science bootcamp for startups.\n[Renato Boemer](https://towardsdatascience.com/author/boemer/)\nFebruary 11, 20216 min read\n  * [![](../images/009.png)](https://towardsdatascience.com/complex-valued-cnns-for-medical-image-denoising-12a4262c6ef6/)\n## [Complex-Valued CNNs for Medical Image Denoising](https://towardsdatascience.com/complex-valued-cnns-for-medical-image-denoising-12a4262c6ef6/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA novel approach for denoising medical images\n[Shubhankar Rawat](https://towardsdatascience.com/author/shubhankarrawatsam.1999/)\nOctober 14, 20225 min read\n  * [![](../images/010.jpeg)](https://towardsdatascience.com/linear-regression-with-bootstrapping-4924c05d2a9/)\n## [Linear Regression With Bootstrapping](https://towardsdatascience.com/linear-regression-with-bootstrapping-4924c05d2a9/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThere were others who had forced their way to the top from the lowest rung by the aid of their bootstraps\n[James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\nMarch 13, 20219 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n",
  "metadata": {
    "title": "Optimizing Data Transfer in AI/ML Workloads | Towards Data Science",
    "description": "A deep dive on data transfer bottlenecks, their identification, and their resolution with the help of NVIDIA Nsight™ Systems",
    "keywords": null,
    "author": "Chaim Rand",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Optimizing Data Transfer in AI/ML Workloads",
    "og:description": "A deep dive on data transfer bottlenecks, their identification, and their resolution with the help of NVIDIA Nsight™ Systems",
    "og:url": "https://towardsdatascience.com/optimizing-data-transfer-in-ai-ml-workloads/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/01/1-4.jpeg",
    "og:image:width": "828",
    "og:image:height": "553",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Chaim Rand",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "14 min read",
    "article:published_time": "2026-01-03T10:00:07.000Z",
    "article:modified_time": "2026-01-03T10:00:07.000Z",
    "article:author": "https://towardsdatascience.com/author/chaimrand/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_000.txt",
      "chars": 237
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_001.txt",
      "chars": 17
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_002.txt",
      "chars": 155
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_003.txt",
      "chars": 599
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_004.txt",
      "chars": 1290
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_005.txt",
      "chars": 118
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_006.txt",
      "chars": 638
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_007.txt",
      "chars": 120
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_008.txt",
      "chars": 398
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_009.txt",
      "chars": 1467
    },
    {
      "language": "txt",
      "file": "attachments/4267c8346f9a/code/block_010.txt",
      "chars": 1587
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:10:56Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:10:56Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:10:56Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:10:56Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/chaimrand/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/a-caching-strategy-for-identifying-bottlenecks-on-the-data-input-pipeline/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/pipelining-ai-ml-training-workloads-with-cuda-streams/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/model-optimization/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/pytorch/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/paper-walkthrough-u-net-98877a2cd33c/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/muhammad_ardi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/simple-mathematical-derivation-of-bias-variance-error-4ab223f28791/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/damian.ejlli/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/photos/green-trees-beside-river-during-daytime-ZPQdp0jSaP8"
    },
    {
      "type": "external",
      "href": "https://docs.nvidia.com/nsight-systems/index.html"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html"
    },
    {
      "type": "external",
      "href": "https://docs.nvidia.com/nsight-systems/InstallationGuide/"
    },
    {
      "type": "external",
      "href": "https://catalog.ngc.nvidia.com/orgs/nvidia/teams/devtools/containers/nsight-systems-cli"
    },
    {
      "type": "external",
      "href": "https://aws.amazon.com/ec2/instance-types/g6e/"
    },
    {
      "type": "external",
      "href": "https://www.nvidia.com/en-eu/data-center/l40s/"
    },
    {
      "type": "external",
      "href": "https://docs.aws.amazon.com/dlami/latest/devguide/appendix-ami-release-notes.html"
    },
    {
      "type": "external",
      "href": "https://pytorch.org/get-started/locally/"
    },
    {
      "type": "external",
      "href": "https://developer.nvidia.com/nsight-systems/get-started"
    },
    {
      "type": "external",
      "href": "https://docs.nvidia.com/tools-extension/index.html"
    },
    {
      "type": "external",
      "href": "https://docs.pytorch.org/docs/stable/cuda.html"
    },
    {
      "type": "external",
      "href": "https://pypi.org/project/nvtx/"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/in/yitzhak-levi-49a217201/"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/in/gilad-wasserman-992530268/?originalSubdomain=il"
    },
    {
      "type": "external",
      "href": "https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:34:50Z",
  "source_url": "https://towardsdatascience.com/optimizing-data-transfer-in-ai-ml-workloads/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Optimizing Data Transfer in AI/ML Workloads\nA deep dive on data transfer bottlenecks, their identification, and their resolution with the help of NVIDIA Nsight™ Systems\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nJanuary 3, 202614 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/01/1-4.jpeg)\n[Photo](https://unsplash.com/photos/green-trees-beside-river-during-daytime-ZPQdp0jSaP8) by Daniel Seßler on Unsplash\nIn a typical AI/ML workload, a deep learning model is executed on a dedicated GPU accelerator using input data batches it receives from a CPU host. Ideally, the GPU — the more expensive resource — should be maximally utilized, with minimal periods of idle time. In particular, this means that every time it completes its execution on a batch, the subsequent batch will be “ripe and ready” for processing. When this does not happen, the GPU idles while waiting for input data — a common performance bottleneck often referred to as GPU starvation.\nIn previous posts, (e.g., see [A Caching Strategy for Identifying Bottlenecks on the Data Input Pipeline](https://towardsdatascience.com/a-caching-strategy-for-identifying-bottlenecks-on-the-data-input-pipeline/)), we discussed common causes of this issue, including: inefficient storage retrieval, CPU resource exhaustion, and host-to-device transfer bottlenecks. In this post, we zoom in on data transfer bottlenecks and revisit their identification and resolution — this time with the help of [NVIDIA Nsight™ Systems (nsys)](https://docs.nvidia.com/nsight-systems/index.html), a performance profiler designed for analyzing the system-wide activity of workloads running on NVIDIA GPUs.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## NVIDIA Nsight vs. PyTorch Profiler\nReaders familiar with our work may be surprised at the mention of NVIDIA Nsight profiler rather than [PyTorch Profiler](https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html). In our previous posts we have advocated strongly for the use of PyTorch Profiler in AI/ML model development as a tool for identifying and optimizing runtime performance. Time and again, we have demonstrated its application to a wide variety of performance issues. Its use does not require any special installations and can be run without special OS permissions. NVIDIA Nsight profiler, on the other hand, requires a [dedicated system setup](https://docs.nvidia.com/nsight-systems/InstallationGuide/) (or [a dedicated NVIDIA container](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/devtools/containers/nsight-systems-cli)) and — for some of its features — elevated permissions, making its use less accessible and more complicated than PyTorch Profiler.\nThe two profilers differ in their focus: PyTorch profiler is a  _framework_ profiler tightly coupled with PyTorch and heavily focused on how models use the PyTorch software stack and supporting libraries. NVIDIA Nsight profiler is a  _system_ -level profiler; it does not know the details of the model being run or which framework is being used, but rather how the components of the entire system are being used and utilized. While PyTorch Profiler excels at tracing the low-level operations of a PyTorch model execution, nsys provides a detailed view of the activities of the entire system (GPU hardware, CUDA streams, OS interrupts, Network, PCIe, etc.). For many performance issues PyTorch profiler is sufficient for identifying and solving the source of the bottleneck; But some situations call for nsys profiler, the “big guns”, for deriving deeper insights into the inner workings of the underlying system.\nIn this post we intend to demonstrate some of the unique capabilities of nsys profiler and their application to the common data-transfer bottleneck.\n## Outline\nTo facilitate our discussion we will define a toy ML workload with a data-transfer performance bottleneck and proceed to introduce a number of successive optimizations in an attempt to solve it. Throughout the process, we will use the nsys profiler in order to analyze the system performance and assess the impact of the code modifications.\n## Setup\nWe will run our experiments on an [Amazon EC2 g6e.2xlarge](https://aws.amazon.com/ec2/instance-types/g6e/) instance with an [NVIDIA L40S](https://www.nvidia.com/en-eu/data-center/l40s/) GPU running an [AWS Deep Learning (Ubuntu 24.04) AMI](https://docs.aws.amazon.com/dlami/latest/devguide/appendix-ami-release-notes.html) with [PyTorch (2.8)](https://pytorch.org/get-started/locally/). To install the nsys-cli profiler (version 2025.6.1) we follow the [official NVIDIA guidelines](https://developer.nvidia.com/nsight-systems/get-started):\nbash\n\n```\nwget https://developer.nvidia.com/downloads/assets/tools/secure/nsight-systems/2025_6/NsightSystems-linux-cli-public-2025.6.1.190-3689520.deb\nsudo apt install ./NsightSystems-linux-cli-public-2025.6.1.190-3689520.deb\n```\n\nThe [NVIDIA Tools Extension](https://docs.nvidia.com/tools-extension/index.html) (NVTX) library allows us to annotate our code with human-readable labels to increase the readability and comprehension of the performance trace. While PyTorch offers built-in NVTX support via its [torch.cuda.nvtx](https://docs.pytorch.org/docs/stable/cuda.html#nvidia-tools-extension-nvtx) APIs, we will use the standalone [nvtx](https://pypi.org/project/nvtx/) package (version 0.2.14) which supports color-coding the trace timeline for better visual analysis:\nbash\n\n```\npip install nvtx\n```\n\n## Disclaimers\nThe code we will share is intended for demonstrative purposes; please do not rely on its correctness or optimality. Please do not interpret our use of any library, tool, or platform, as an endorsement of its use. The impact of the optimizations we will cover can vary greatly based on the details of the model and the runtime environment. Please be sure to assess their effect on your own use case before integrating their use.\nMany thanks to [Yitzhak Levi](https://www.linkedin.com/in/yitzhak-levi-49a217201/) and [Gilad Wasserman](https://www.linkedin.com/in/gilad-wasserman-992530268/?originalSubdomain=il) for their contributions to this post.\n## A Toy PyTorch Model\nWe introduce a training script intentionally designed to consist of a bottleneck on the data-input pipeline.\nIn the code block below we define a simple image classification model with a [ResNet-18](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18) backbone.\npython\n\n```\nimport time, torch, torchvision\n\nDEVICE = \"cuda\"\nmodel = torchvision.models.resnet18().to(DEVICE).train()\noptimizer = torch.optim.Adam(model.parameters())\n```\n\nNext, we define a synthetic dataset which we will use to train our toy model.\npython\n\n```\nfrom torch.utils.data import Dataset, DataLoader\n\nWARMUP_STEPS = 10\nPROFILE_STEPS = 3\nCOOLDOWN_STEPS = 1\nTOTAL_STEPS = WARMUP_STEPS + PROFILE_STEPS + COOLDOWN_STEPS\nBATCH_SIZE = 64\nTOTAL_SAMPLES = TOTAL_STEPS * BATCH_SIZE\nIMG_SIZE = 512\n\n# A synthetic Dataset with random images and labels\nclass FakeDataset(Dataset):\n\n    def __len__(self):\n        return TOTAL_SAMPLES\n\n    def __getitem__(self, index):\n        img = torch.randn((3, IMG_SIZE, IMG_SIZE))\n        label = torch.tensor(index % 10)\n        return img, label\n\ntrain_loader = DataLoader(\n    FakeDataset(),\n    batch_size=BATCH_SIZE\n)\n```\n\nLastly, we define a standard training step programmed to run nsys-profiler for three steps using the [torch.cuda.profiler.start](https://github.com/pytorch/pytorch/blob/v2.9.0/torch/cuda/profiler.py#L50) and [stop](https://github.com/pytorch/pytorch/blob/v2.9.0/torch/cuda/profiler.py#L59) commands — intended for use in conjunction with the [nsys cli](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#profiling-from-the-cli). We highlight the components of the training step using the [nvtx.annotate](https://nvtx.readthedocs.io/en/latest/index.html) utility. Please refer to the official documentation for more details on [profiling with nsys in PyTorch](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#pytorch-profiling).\npython\n\n```\nimport nvtx\nfrom torch.cuda import profiler\n\ndef copy_data(batch):\n    data, targets = batch\n    data_gpu = data.to(DEVICE)\n    targets_gpu = targets.to(DEVICE)\n    return data_gpu, targets_gpu\n\n\ndef compute_step(model, batch, optimizer):\n    data, targets = batch\n    output = model(data)\n    loss = torch.nn.functional.cross_entropy(output, targets)\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n    return loss\n\n\ndata_iter = iter(train_loader)\n\nfor i in range(TOTAL_STEPS):\n\n    if i == WARMUP_STEPS:\n        # start nsys profiler\n        torch.cuda.synchronize()\n        start_time = time.perf_counter()\n        profiler.start()\n    elif i == WARMUP_STEPS + PROFILE_STEPS:\n        # stop nsys profiler\n        torch.cuda.synchronize()\n        profiler.stop()\n        end_time = time.perf_counter()\n\n    with nvtx.annotate(f\"Batch {i}\", color=\"blue\"):\n        with nvtx.annotate(\"get batch\", color=\"red\"):\n            batch = next(data_iter)\n        with nvtx.annotate(\"copy batch\", color=\"yellow\"):\n            batch = copy_data(batch)\n        with nvtx.annotate(\"Compute\", color=\"green\"):\n            compute_step(model, batch, optimizer)\n\ntotal_time = end_time - start_time\nthroughput = PROFILE_STEPS / total_time\nprint(f\"Throughput: {throughput:.2f} steps/sec\")\n```\n\nWe run our script using the  _cudaProfilerApi_ option to start and stop the profiler programmatically. Please see the official documentation for full details on [profiling from the nsys cli](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#profiling-from-the-cli).\nbash\n\n```\nnsys profile \\\n  --capture-range=cudaProfilerApi \\\n  --trace=cuda,nvtx,osrt \\\n  --output=baseline \\\n  python train.py\n```\n\nThis results in a  _baseline.nsys-rep_ trace file that we copy over to our development machine for analysis.\nIn order to draw a comparison to PyTorch profiler, we define an alternative training loop programmed with [PyTorch Profiler](https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) and annotated with the [torch.profiler.record_function](https://pytorch.org/tutorials/beginner/profiler.html#performance-debugging-using-profiler) utility:\npython\n\n```\nfrom torch.profiler import (\n    profile, record_function, schedule, tensorboard_trace_handler\n)\n\nwith profile(\n    schedule=schedule(wait=0, warmup=WARMUP_STEPS, \n                      active=PROFILE_STEPS, repeat=1),\n    on_trace_ready=tensorboard_trace_handler('./baseline'),\n    record_shapes=True,\n    with_stack=True\n) as prof:\n    for i in range(TOTAL_STEPS):\n        with record_function(\"get batch\"):\n            batch = next(data_iter)\n        with record_function(\"copy batch\"):\n            batch = copy_data(batch)\n        with record_function(\"compute\"):\n            compute_step(model, batch, optimizer)\n        prof.step()\n```\n\nThe throughput of our baseline experiment is 2.97 steps-per-second. In the next sections we will use the profile traces to identify performance bottlenecks in our training step and try to improve on this result.\n## Baseline Performance Analysis\nTo analyze the resultant nsys trace file, we open it in the [Nsight Systems GUI application](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#opening-command-line-results-files-for-visualization). In the image below we zoom in on the timeline of two of the training steps captured by the profiler:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/2.webp)\nBaseline Nsight Systems Profiler Trace (by Author)\nThe trace contains a wealth of information, just a subset of which we will touch on in this post. Please see the nsys documentation for additional functionalities and features.\nThe timeline is divided into two parts: the CUDA section which reports GPU activity and the threads section which reports the CPU activity. The CUDA section makes a clear distinction between the GPU kernel (compute) activity (90.9%) and memory activity (9.1%). The top bars in each section report the utilization of each of the resources and both sections include an NVTX section with the colored annotations we included in our training step. We note the following observations:\n  1. The GPU is idle for roughly 50% of each training step. This can be seen by the portion of time taken by each batch (in blue) in the GPU NVTX bar and the large blocks of whitespace in between them.\n  2. The GPU activity for each batch starts immediately after the “get batch” activity has completed on the CPU. It starts with the host-to-device memory copy, marked in light green and continues with the kernel computations, marked in light blue.\n  3. Once the CPU has launched the GPU memory and compute commands for batch  _N_ , it proceeds to the next batch in the training loop — leading to a partial overlap of batch  _N+1_ on the CPU with batch  _N_ on the GPU.\n  4. The vast majority of the CPU thread is spent on the “get batch” activity. This constitutes the primary bottleneck in our baseline experiment.\n\n\nThe profiling trace points to a clear culprit — the dataloader. By default, PyTorch performs [single process data loading](https://docs.pytorch.org/docs/stable/data.html#single-process-data-loading-default) — a single CPU process is used to load the next data input batch, copy it to the GPU, and launch the compute kernels — all in a sequential manner. This typically results in severe under-utilization of the CPU resources by: 1) limiting dataloading to just a single process, and 2) making the loading of the next batch contingent on the completion of the CPU processing (i.e., kernel loading) of the previous batch. Our irresponsible use of our CPU resources has resulted in our GPU being starved for input data.\nThe same conclusion could have been reached using PyTorch Profiler trace shown below:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/3.webp)\nBaseline PyTorch Profiler Trace (by Author)\nHere too, we can see long periods of GPU underutilization that are caused by the long “get batch” blocks on the CPU side.\n### Optimization 1: Multi-Process Data Loading\nThe first step is to modify the data input pipeline to use [multi-process data loading](https://docs.pytorch.org/docs/stable/data.html#multi-process-data-loading). We set the number of workers to match the 8 vCPUs available on our [Amazon EC2 g6e.2xlarge](https://aws.amazon.com/ec2/instance-types/g6e/) instance. In a real-world scenario, this value should be tuned for optimal throughput:\npython\n\n```\nNUM_WORKERS = 8\n\ntrain_loader = DataLoader(\n    FakeDataset(),\n    batch_size=BATCH_SIZE,\n    num_workers=NUM_WORKERS\n)\n```\n\nFollowing this change our throughput jumps to 4.81 steps per second — a 62% improvement over our baseline result. The corresponding nsys profiler trace is shown below:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/4.webp)\nMultiproc Dataloading Nsight Systems Profiler Timeline (by Author)\nNote that the red “get batch” segment has become just a tiny sliver of each step in the NVTX bar. In its place, the yellow “copy batch” block now takes center stage. As a result of our use of multi-process dataloading, there is now always a new batch ready for processing — but can we do better?\nTaking a closer look at the GPU section we see that there is still a significant portion (~290 milliseconds) of idle time in between the memory operation and the kernel compute. This idle time is perfectly aligned with an “munmap” operation in the OS runtime bar. The “munmap” block is a CPU-side memory cleanup operation performed just after the CUDA memory copy is complete. It occurs at the tail-end of the long yellow “copy batch” operation. The compute kernels are launched onto the GPU only after the memory cleanup has completed. This is a clear pattern of synchronous host-to-device memory copy: The CPU cannot proceed with kernel loading until the data copy operation has been fully completed and the GPU stays idle until the CPU loads the kernels.\nThe PyTorch profiler trace shows the same GPU idle time but it does not provide the same “munmap” hint. This is our first example of the advantage of the system-wide visibility of the nsys profiler.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/5.webp)\nMultiproc Dataloading PyTorch Profiler Trace (by Author)\nWith our finding of the data-copy performance bottleneck in hand, we proceed to our next optimization.\n### Optimization 2: Asynchronous Data Transfer\nThe solution to the bottleneck we have found is to program our training step to load [data asynchronously](https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html#enable-asynchronous-data-loading-and-augmentation). This enables the CPU to launch the compute kernels immediately after sending the memory copy command — without waiting for the memory copy to be completed. This way the GPU can begin processing the kernels as soon as the CUDA memory copy is done. Enabling asynchronous data copy requires two changes: First we must program the dataloader to use [pinned memory](https://docs.pytorch.org/docs/stable/data.html#memory-pinning) (instead of pageable memory), and second, we must pass `non_blocking=True` argument to the [to()](https://docs.pytorch.org/docs/stable/generated/torch.Tensor.to.html#torch.Tensor.to) operations:\npython\n\n```\nNUM_WORKERS = 8\nASYNC_DATATRANSFER = True\n\n\ntrain_loader = DataLoader(\n    FakeDataset(),\n    batch_size=BATCH_SIZE,\n    num_workers=NUM_WORKERS,\n    pin_memory=ASYNC_DATATRANSFER\n)\n\ndef copy_data(batch):\n    data, targets = batch\n    data_gpu = data.to(DEVICE, non_blocking=ASYNC_DATATRANSFER)\n    targets_gpu = targets.to(DEVICE, non_blocking=ASYNC_DATATRANSFER)\n    return data_gpu, targets_gpu\n```\n\nUsing asynchronous dataloading results in a throughput of 5.91 steps per second — an additional 23% improvement and 99% improvement overall. The resultant profiling trace is shown below:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/6.webp)\nAsync Dataloading Nsight Systems Profiler Timeline (by Author)\nWe now see all of the CPU operations bunched together at the beginning of the trace. We have removed all performance obstacles on the CPU side allowing it to freely load the data and kernels to the GPU. In the GPU section, we see continuous activity without any idle time. We do, however, see a clear separation between CUDA memory activities (in light green) and CUDA kernel activities (in light blue). PyTorch profiler, in contrast, does not make this distinction clear. This is another advantage of the hardware-centric profiler and, in the case of our toy experiment, is what informs the next steps of our optimization.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/7.webp)\nAsync Dataloading PyTorch Profiler Trace (by Author)\n### Optimization 3: Pipelining With CUDA Streams\nOur final optimizations derive from the fact that modern GPUs, such as the [NVIDIA L40S](https://www.nvidia.com/en-eu/data-center/l40s/), use independent engines for copying memory (the DMA) and executing compute kernels (the SMs). We can take advantage of this by parallelizing the distinct memory and kernel activities we saw in the nsys profiler trace. We will program this through the use of CUDA streams.\nIn a [previous post](https://towardsdatascience.com/pipelining-ai-ml-training-workloads-with-cuda-streams/), we expanded on the opportunity for optimizing AI/ML workloads using CUDA Streams. Here, we apply a similar pipelining strategy: We define two distinct “copy” and “compute” CUDA streams and program the “copy” stream to copy batch  _N+1_ at the same time that the “compute” stream is processing batch  _N_ :\npython\n\n```\n# define two CUDA streams\ncompute_stream = torch.cuda.Stream()\ncopy_stream = torch.cuda.Stream()\n\n\n# extract first batch\nnext_batch = next(data_iter)\nwith torch.cuda.stream(copy_stream):\n    next_batch = copy_data(next_batch)\n\nfor i in range(TOTAL_STEPS):\n\n    if i == WARMUP_STEPS:\n        torch.cuda.synchronize()\n        start_time = time.perf_counter()\n        profiler.start()\n    elif i == WARMUP_STEPS + PROFILE_STEPS:\n        torch.cuda.synchronize()\n        profiler.stop()\n        end_time = time.perf_counter()\n\n    with nvtx.annotate(f\"Batch {i}\", color=\"blue\"):\n        # wait for copy stream to complete copy of batch N\n        compute_stream.wait_stream(copy_stream)\n        batch = next_batch\n\n        # execute model on batch N+1 compute stream\n        try:\n            with nvtx.annotate(\"get batch\", color=\"red\"):\n                next_batch = next(data_iter)\n            with torch.cuda.stream(copy_stream):\n                with nvtx.annotate(\"copy batch\", color=\"yellow\"):\n                    next_batch = copy_data(next_batch)\n        except:\n            # reached end of dataset\n            next_batch = None\n\n        # execute model on batch N compute stream\n        with torch.cuda.stream(compute_stream):\n            with nvtx.annotate(\"Compute\", color=\"green\"):\n                compute_step(model, batch, optimizer)\n\ntotal_time = end_time - start_time\nthroughput = PROFILE_STEPS / total_time\nprint(f\"Throughput: {throughput:.2f} steps/sec\")\n```\n\nThis optimization results in a throughput of 6.44 steps per second — a 9% improvement over our previous experiment. We note that the impact of this optimization is capped by the duration of the longer of the two operation types. In our previous profile trace, the memory block took 15.5 milliseconds and the kernel block took 155 milliseconds. In the current profile trace, the entire GPU steps takes 155 milliseconds, which means that the memory copy time is completed hidden by the kernel compute time and that our optimization reaches the maximum possible result.\nThe use of the CUDA streams and its impact on GPU utilization can be seen in the traces of both profilers:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/8.webp)\nPipelined Nsight Systems Profiler Timeline (by Author)\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/9.webp)\nPipelined PyTorch Profiler Trace (by Author)\n### Optimization 4: Prefetching to CUDA\nFor our final step, we move the data copying from the main training loop process to the data loading process: Rather than explicitly calling the copy function inside the training loop, we assume that the batches returned from the data iterator are already placed on the GPU.\nIn the code block below, we wrap our dataloader with a CUDA-prefetching iterator class. Note, that this is a simplified implementation intended for the purposes of demonstration. More work may be required for more complex scenarios (e.g., DDP training). Alternatively, you may consider a third-party implementation such as [torchtnt.utils.data.data_prefetcher.CudaDataPrefetcher](https://github.com/meta-pytorch/tnt/blob/master/torchtnt/utils/data/data_prefetcher.py#L17):\npython\n\n```\nclass DataPrefetcher:\n    def __init__(self, loader):\n        self.loader = iter(loader)\n        self.stream = torch.cuda.Stream()\n        self.next_batch = None\n        self.preload()\n\n    def preload(self):\n        try:\n            data, targets = next(self.loader)\n\n            with torch.cuda.stream(self.stream):\n                with nvtx.annotate(\"copy batch\", color=\"yellow\"):\n                    next_data = data.to(DEVICE, non_blocking=True)\n                    next_targets = targets.to(DEVICE, non_blocking=True)\n            self.next_batch = (next_data, next_targets)        \n        except:\n            self.next_batch = (None, None)\n\n    def __iter__(self):\n        return self\n\n    def __next__(self):\n        torch.cuda.current_stream().wait_stream(self.stream)\n        data, targets = self.next_batch\n        self.preload()\n        return data, targets\n\n\ndata_iter = DataPrefetcher(train_loader)\n\nfor i in range(TOTAL_STEPS):\n    if i == WARMUP_STEPS:\n        torch.cuda.synchronize()\n        start_time = time.perf_counter()\n        profiler.start()\n    elif i == WARMUP_STEPS + PROFILE_STEPS:\n        torch.cuda.synchronize()\n        profiler.stop()\n        end_time = time.perf_counter()\n\n    with nvtx.annotate(f\"Batch {i}\", color=\"blue\"):\n        with nvtx.annotate(\"get batch\", color=\"red\"):\n            batch = next(data_iter)\n        with nvtx.annotate(\"Compute\", color=\"green\"):\n            loss = compute_step(model, batch, optimizer)\n\ntotal_time = end_time - start_time\nthroughput = PROFILE_STEPS / total_time\nprint(f\"Throughput: {throughput:.2f} steps/sec\")\n```\n\nThis optimization results in a throughput of 6.44 steps per second — the same as our previous experiment. This should not surprise us since we have already seen that the throughput is bound by the 155 millisecond GPU compute and our optimization has not done anything to reduce the kernel compute time.\nMore generally, despite the removal of the copy call from the main loop, you may have a hard time finding a situation where this will have a meaningful impact on performance since the call is already being called asynchronously. However, given the minimal changes to the training loop, you may find this solution to be cleaner and/or to be more applicable for use with high-level libraries that do not enable fine-grained control of the training loop.\nUnsurprisingly, the profile traces for this experiment appear nearly identical to the previous ones. The main difference is the placement of the yellow “copy data” block in the NVTX row of the CPU section.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/10.webp)\nData Prefetching Nsight Systems Profiler Timeline (by Author)\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/11.webp)\nData Prefetching PyTorch Profiler Trace (by Author)\n## Results\nThe table below summarizes the results of our experiments:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/12.webp)\nExperiment Results (by Author)\nThe optimizations, which were driven by the use of Nsight Systems profiler, resulted in an overall increase of  _2.17X_ to the runtime performance.\n### Summary\nGPU starvation is a common performance bottleneck that can have a devastating impact on the efficiency and costs of AI/ML workloads. In this post, we demonstrated how to use the Nsight Systems profiler to study the causes of the performance bottleneck and take informed steps towards their resolution. Along the way, we emphasized the unique capabilities of Nsight Systems profiler when compared to the built-in framework-centric PyTorch Profiler — specifically its deep system-level visibility.\nOur focus, in this post has been on the host-to-device data copy that typically occurs at the beginning of the training step. However, data-transfer bottlenecks can appear at different stages of training. In a sequel to this post we intend to repeat our nsys profiling analysis on data copies going in the opposite direction — from the device to the host. Stay tuned!\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657094227_7ymyo0.jpeg)\nWritten By\n## [Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[See all from Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Model Optimization](https://towardsdatascience.com/tag/model-optimization/)[Pytorch](https://towardsdatascience.com/tag/pytorch/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-data-transfer-in-ai-ml-workloads%2F&title=Optimizing%20Data%20Transfer%20in%20AI%2FML%20Workloads)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-data-transfer-in-ai-ml-workloads%2F&title=Optimizing%20Data%20Transfer%20in%20AI%2FML%20Workloads)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Foptimizing-data-transfer-in-ai-ml-workloads%2F&text=Optimizing%20Data%20Transfer%20in%20AI%2FML%20Workloads)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/09/0vxhpyHFBz7MPenI1-scaled.jpg)](https://towardsdatascience.com/paper-walkthrough-u-net-98877a2cd33c/)\n## [Paper Walkthrough: U-Net](https://towardsdatascience.com/paper-walkthrough-u-net-98877a2cd33c/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA PyTorch implementation on one of the most popular semantic segmentation models.\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nSeptember 20, 202418 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1M3zrZC-nja6pJipgMPjQ4A.jpeg)](https://towardsdatascience.com/simple-mathematical-derivation-of-bias-variance-error-4ab223f28791/)\n## [Simple mathematical derivation of bias-variance error](https://towardsdatascience.com/simple-mathematical-derivation-of-bias-variance-error-4ab223f28791/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n1. Introduction\n[Damian Ejlli](https://towardsdatascience.com/author/damian.ejlli/)\nJuly 29, 202110 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/03/0vwu90_NtUGs4q5zJ-scaled.jpg)](https://towardsdatascience.com/hybrid-discrete-continuous-geometric-deep-learning-2e5871293184/)\n## [Hybrid Discrete-Continuous Geometric Deep Learning](https://towardsdatascience.com/hybrid-discrete-continuous-geometric-deep-learning-2e5871293184/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nScalable and equivariant spherical CNNs by DISCO convolutions\n[Jason McEwen](https://towardsdatascience.com/author/jasonmcewen/)\nMarch 15, 20236 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/05/0gOTEEsL3_z44p1dv-scaled.jpg)](https://towardsdatascience.com/overcoming-ml-data-preprocessing-bottlenecks-with-grpc-ca30fdc01bee/)\n## [Overcoming ML Data Preprocessing Bottlenecks With Auxiliary Workers and gRPC](https://towardsdatascience.com/overcoming-ml-data-preprocessing-bottlenecks-with-grpc-ca30fdc01bee/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA Simple Solution for Increasing Resource Utilization and Accelerating Training\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nMay 31, 202214 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/0VJdv4cH0Xl_mQz7v-scaled.jpg)](https://towardsdatascience.com/how-to-differentiate-gradient-descent-objective-function-in-3-simple-steps-b9d58567d387/)\n## [How to Differentiate Gradient Descent Objective Function in 3 simple steps](https://towardsdatascience.com/how-to-differentiate-gradient-descent-objective-function-in-3-simple-steps-b9d58567d387/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nGrab a pen and paper\n[Łukasz Gebel](https://towardsdatascience.com/author/luk.gebel/)\nAugust 14, 20205 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/Untitled.jpg)](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/)\n## [Stop Returning Flat Text from a PDF: The Relational Tables RAG Needs](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nEnterprise Document Intelligence [Vol.1 #5B] - One PDF in, a relational set of DataFrames out: lines, pages, TOC, images, cross-references, captions, spans, and a parsing summary\n[Kezhan Shi](https://towardsdatascience.com/author/kezhanshi/)\nJune 11, 202625 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/1VpQ-DaQF1YRmQ7Xhp4jVsg-scaled.jpeg)](https://towardsdatascience.com/the-best-data-science-bootcamp-for-startups-in-the-world-e9ae2b4ce6e6/)\n## [The Best Data Science Bootcamp for Startups in the World](https://towardsdatascience.com/the-best-data-science-bootcamp-for-startups-in-the-world-e9ae2b4ce6e6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhy I am joining Le Wagon, the best Data Science bootcamp for startups.\n[Renato Boemer](https://towardsdatascience.com/author/boemer/)\nFebruary 11, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/1b8vOEp2DL8YAShpFhhucdg.png)](https://towardsdatascience.com/complex-valued-cnns-for-medical-image-denoising-12a4262c6ef6/)\n## [Complex-Valued CNNs for Medical Image Denoising](https://towardsdatascience.com/complex-valued-cnns-for-medical-image-denoising-12a4262c6ef6/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA novel approach for denoising medical images\n[Shubhankar Rawat](https://towardsdatascience.com/author/shubhankarrawatsam.1999/)\nOctober 14, 20225 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1fxuQFyLEUl-XJxxFZZR_2g.jpeg)](https://towardsdatascience.com/linear-regression-with-bootstrapping-4924c05d2a9/)\n## [Linear Regression With Bootstrapping](https://towardsdatascience.com/linear-regression-with-bootstrapping-4924c05d2a9/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThere were others who had forced their way to the top from the lowest rung by the aid of their bootstraps\n[James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\nMarch 13, 20219 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n",
  "body_base_path": "attachments/4267c8346f9a/pages/",
  "body_markdown_path": "attachments/4267c8346f9a/pages/archived-body.md",
  "local_html": "attachments/4267c8346f9a/pages/archived-body.html",
  "body_source": "prior_capture"
}