{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Shawhin Talebi",
  "title_listed": "How to Build Data Pipelines for Machine Learning",
  "keywords": "数据管道,ML",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5/",
  "publish_date": "2024-05",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "0b0b6c3fbcb4",
  "collected_at": "2026-10-01T22:50:33",
  "title": "How to Build Data Pipelines for Machine Learning | Towards Data Science",
  "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# How to Build Data Pipelines for Machine Learning\nA beginner-friendly introduction with Python code\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nMay 2, 202410 min read\nThis is the 3rd article in a [larger series](https://shawhin.medium.com/list/full-stack-data-science-f0910c75d006) on Full Stack Data Science (FSDS). In the [previous post](https://towardsdatascience.com/data-science-project-management-e8787d818ad0), I introduced a 5-step project management framework for building machine learning (ML) solutions. While ML may bring to mind fancy algorithms and technologies, the **quality of an ML solution is determined by the** **quality of the available data**. This raises the need for data engineering (DE) skills in FSDS. This article will discuss the most critical DE skills in this context and walk through a real-world example.\n![Photo by the blowup on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/067YBe2ZDTrnUqJSw-scaled.jpg)\nPhoto by [the blowup](https://unsplash.com/@theblowup?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n* * *\nFull Stack Data Science (FSDS) involves managing and implementing ML solutions end-to-end. **Data engineering** is critical to this process i.e. **making data readily available for analytics and ML applications** [1].\nWhile this can involve a wide range of tasks (e.g., data modeling, designing schema, managing distributed systems), in the context of FSDS, data engineering comes down to one key thing—building data pipelines.\nA **data pipeline** **gets data from point A to point B**. For example, scraping a webpage, reformating the data, and loading it into a database. Data pipelines consist of 3 steps - extract (E), transform (T), and load (L), which can be combined in two ways - ETL or ELT.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist), [Machine Learning](https://roadmap.sh/machine-learning) and [Data Engineer](https://roadmap.sh/data-engineer) roadmaps.\nPlease accept cookies to access this content\n## **ETL vs ELT**\nWhile the difference between these two pipeline types may sound subtle, they look very different in practice [2].\n**ETL (extract, transform, load)** involves extracting data from a source, transforming it into a useful form, and storing it in a fixed format (typically a table in a database or data warehouse). This is great for data that can be represented as **rows and columns** and **serves a narrow set of downstream applications** , such as customer-level sales data, for training a propensity model.\n**ELT (extract, load, transform)** , on the other hand, extracts data and loads it in its raw format (typically into a data warehouse or data lake). This centralizes an organization's various data sources in a way that provides **more flexibility in downstream use cases**. Additionally, ELT supports unstructured and semi-structured data (i.e. data not represented as rows and columns).\n## **ETL (for FSDS)**\nAlthough ELT is becoming the norm across enterprises [2], full-stack data scientists typically build **product-focused data pipelines** (i.e. pipelines built for particular use cases). Thus, the ETL paradigm may be a natural choice for most ML projects.\nLet's walk through each stage of this type of pipeline.\n## **Extract**\nThe first step of any data pipeline is **extracting data from its source**. This could be leads data from a CRM, sales data from an e-commerce platform, marketing data from social media, etc. While enterprises can have hundreds (if not thousands) of data sources [2], most ML projects will involve only a few key data sources.\n**Most data sources for most businesses** **run through 3rd parties** , which can (ideally) be accessed via official APIs, e.g. HubSpot's API, Shopify's API, and Meta's APIs.\nHowever, data from some sources may not be readily available and require custom software to extract the necessary information. This may involve scraping public web pages, gathering data from a self-hosted system, collecting raw sensor data, or extracting documents from a file system.\n## **Transform**\nAt extraction, data are typically semi-structured (e.g., JSON, .csv) or unstructured (documents, images, or other binary files). The transform step aims to **translate these data into a format suitable for downstream tasks**.\nThis can consist of a wide range of operations. Some examples are given below.\n  * Managing data types and ranges\n  * Deduplication\n  * Imputing missing values\n  * Handling special characters and values\n  * Feature engineering\n  * Converting data into structured format (i.e. table for a database or data warehouse).\n\n\nAs the list above suggests, the transform step requires full-stack data scientists to wear their **data engineer and data scientist hats**. Additionally, this step may be revisited multiple times during an ML project, as new insights are gained during model development.\n## **Load**\nOnce transformed, data must be **made readily available for downstream tasks**. Data warehouses and data lakes have become standard across enterprises because they are built to accommodate petabyte-scale data needs.\nHowever, many ML systems can be developed using GB-scale data. For example, an FP64 array with 1M rows and 1K columns will require ~10GB of storage. In these cases, a relational database (e.g., MySQL, Postgres SQL) or a simpler data store (e.g., S3, Google Drive, Dropbox, a system's file directory) may be sufficient.\nWhile the best storage solution will depend on the details of the specific use case, I believe in **keeping things as simple as possible** and only seeking more sophisticated solutions once things start to break. Here's a coarse-grained guide I like to use as a starting point.\n  * **MB-scale, few sources** : Keep it in the project directory\n  * **GB-scale, few sources** : Simple storage (e.g. S3, Google Drive, Dropbox)\n  * **GB-scale, many sources** : Relational database (e.g. MySQL, PostgreSQL)\n  * **TB-scale, many sources** : Data Warehouse (e.g. Amazon Redshift, Google BigQuery, Snowflake)\n  * **PB-scale, many sources** : Data Lake (e.g. Databricks Delta Lake, Snowflake, AWS Lake Formation)\n\n\n## **Orchestration**\nAs data sources and processes grow, implementing and managing data pipelines becomes challenging. This is where orchestration tools (like Airflow) are helpful.\n**Airflow** is popular among data engineers because it provides a fully Python way to **schedule and manage batch workflows**. This means data pipelines can be set to run at specific time intervals or upon specific events.\n![A DAG for a simple data pipeline. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/155fe8zq2SJhJPVvh5f9aZQ.png)\nA DAG for a simple data pipeline. Image by author.\n## **Data Management**\nOnce a data pipeline moves out of the development environment, it's important to implement processes for **managing and monitoring the data effectively**. While the degree of management will depend on the volume and complexity of the data pipeline, here are a few key things to consider.\n  * **Data Dictionaries** : Human readable descriptions of the data at both table and column levels (or folder and file levels).\n  * **Data Versioning & Backups**: Keep track of multiple versions of data and ensure proper backups. This comes built-in with many data platforms.\n  * **Data Observability** : Setting up monitoring systems to track performance and send alerts when issues arise.\n  * **Data Governance** : Defining rules around data to ensure quality, security, compliance, track linage, and define ownership. This becomes critical when juggling multiple use cases and data sources.\n\n\n## **Example Code: ETL of YouTube Video Transcripts**\nNow that we have a basic understanding of data pipelines let's walk through a simple yet real-world example of building one. Here, I build upon the example case study from the [previous post](https://towardsdatascience.com/data-science-project-management-e8787d818ad0) and implement an ETL pipeline to curate automatically generated captions from all my YouTube videos.\n🔗 [GitHub Repo](https://github.com/ShawhinT/YouTube-Blog/tree/main/full-stack-data-science/data-engineering) | [Full Stack Project Repo](https://github.com/ShawhinT/yt-search)\n* * *\nWe start by importing a few libraries and a secret YouTube API key. If you do not have an API key, you can create one following [this guide](https://developers.google.com/youtube/v3/getting-started).\npython\n\n```\nimport requests\nimport json\nimport polars as pl\nfrom my_sk import my_key\n\nfrom youtube_transcript_api import YouTubeTranscriptApi\n```\n\n## Extract\nNext, we will define variables to help us extract video data from the YouTube API. Here, I specify the ID of my YouTube channel and the API URL, initialize _page _token_ , and create a list for storing video data.\npython\n\n```\n# define channel ID\nchannel_id = 'UCa9gErQ9AE5jT2DZLjXBIdA'\n\n# define url for API\nurl = 'https://www.googleapis.com/youtube/v3/search'\n\n# initialize page token\npage_token = None\n\n# intialize list to store video data\nvideo_record_list = []\n```\n\nThe next chunk of code might be scary, so I will explain what's happening first. We will perform GET requests to YouTube's search API. This is just like searching for videos on YouTube, but instead of using the UI, we will perform searches programmatically.\nSince search results are limited to 50 per page, we need to recursively perform searches to return every video that matches the search criteria. Here's what that looks like in Python code.\npython\n\n```\n# extract video data across multiple search result pages\n\nwhile page_token != 0:\n    # define parameters for API call\n    params = {'key': my_key, 'channelId': channel_id, \n              'part': [\"snippet\",\"id\"], 'order': \"date\", \n              'maxResults':50, 'pageToken': page_token}\n    # make get request\n    response = requests.get(url, params=params)\n\n    # append video data from page results to list\n    video_record_list += getVideoRecords(response)\n\n    try:\n        # grab next page token\n        page_token = json.loads(response.text)['nextPageToken']\n    except:\n        # if no next page token kill while loop\n        page_token = 0\n```\n\n_getVideoRecords()_ is a user-defined function that extracts the relevant information from an API response.\npython\n\n```\n# extract video data from single search result page\n\ndef getVideoRecords(response: requests.models.Response) -> list:\n    \"\"\"\n        Function to extract YouTube video data from GET request response\n    \"\"\"\n\n    # initialize list to store video data from page results\n    video_record_list = []\n\n    for raw_item in json.loads(response.text)['items']:\n\n        # only execute for youtube videos\n        if raw_item['id']['kind'] != \"youtube#video\":\n            continue\n\n        # extract relevant data\n        video_record = {}\n        video_record['video_id'] = raw_item['id']['videoId']\n        video_record['datetime'] = raw_item['snippet']['publishedAt']\n        video_record['title'] = raw_item['snippet']['title']\n\n        # append record to list\n        video_record_list.append(video_record)\n\n    return video_record_list\n```\n\nNow that we have information about all my YouTube videos let's extract the automatically generated captions. To make the video IDs easier to access, I will store the video data in a Polars dataframe.\npython\n\n```\n# store data in polars dataframe\ndf = pl.DataFrame(video_record_list)\nprint(df.head())\n```\n\n![Head of dataframe. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1afqcojIl5b4UmwdhN838oA.png)\nHead of dataframe. Image by author.\nTo pull the video captions, I will use the _youtube_transcript _api_ Python library. I will loop through each video ID in the dataframe and extract the associated transcript.\npython\n\n```\n# intialize list to store video captions\ntranscript_text_list = []\n\n# loop through each row of dataframe\nfor i in range(len(df)):\n\n    # try to extract captions\n    try:\n        # get transcript\n        transcript = YouTubeTranscriptApi.get_transcript(df['video_id'][i])\n        # extract text transcript\n        transcript_text = extract_text(transcript)\n    # if not captions available set as n/a\n    except:\n        transcript_text = \"n/a\"\n\n    # append transcript text to list\n    transcript_text_list.append(transcript_text)\n```\n\nAgain, I use a user-defined function called _extract _text()_ to extract the necessary information from the API.\npython\n\n```\ndef extract_text(transcript: list) -> str:\n    \"\"\"\n        Function to extract text from transcript dictionary\n    \"\"\"\n\n    text_list = [transcript[i]['text'] for i in range(len(transcript))]\n    return ' '.join(text_list)\n```\n\nThen we can add the transcripts for each video to the dataframe.\npython\n\n```\n# add transcripts to dataframe\ndf = df.with_columns(pl.Series(name=\"transcript\", values=transcript_text_list)) \nprint(df.head())\n```\n\n![Head of dataframe with transcripts. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/11-6NXF1O1KnMTM1lDGJqEg.png)\nHead of dataframe with transcripts. Image by author.\n## Transform\nWith the data extracted, we can transform it so it is ready for the downstream use case. This requires some exploratory data analysis (EDA).\n### Handing duplicates\nA good starting point for EDA is to examine the number of unique rows and elements in each column. Here, we expected each row to be uniquely identified by the video_id. Additionally, each column should not have repeating elements except for videos for which no transcript was available, which we set as \"_n/a_ \".\nHere's some code to probe that information. We can see from the output the data match our expectations.\npython\n\n```\n# shape + unique values\nprint(\"shape:\", df.shape)\nprint(\"n unique rows:\", df.n_unique())\nfor j in range(df.shape[1]):\n    print(\"n unique elements (\" + df.columns[j] + \"):\", df[:,j].n_unique())\n\n### output\n# shape: (84, 4)\n# n unique rows: 84\n# n unique elements (video_id): 84\n# n unique elements (datetime): 84\n# n unique elements (title): 84\n# n unique elements (transcript): 82\n```\n\n### Check dtypes\nNext, we can examine the data types of each column. In the image above, we saw that all columns are strings.\nWhile this is appropriate for the _video _id_ , _title_ , and _transcript_ , this is not a good choice for the _datetime_ column. We can change this type in the following way.\nbash\n\n```\n# change datetime to Datetime dtype\ndf = df.with_columns(pl.col('datetime').cast(pl.Datetime))\nprint(df.head())\n```\n\n![Head of dataframe after updating datetime dtype. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1-trYS_otUkUT_pFvhZ-4dA.png)\nHead of dataframe after updating datetime dtype. Image by author.\n### Handling special characters\nSince we are working with text data, it's important to look out for special character strings. This requires a bit of manual skimming of the text, but after a few minutes, I found 2 special cases: _' → '_ and _& → &_\nIn the code below, I replaced these strings with the appropriate characters and changed \"_sha_ \" to \"_Shaw_ \".\nbash\n\n```\n# list all special strings and their replacements\nspecial_strings = [''', '&', 'sha ']\nspecial_string_replacements = [\"'\", \"&\", \"Shaw \"]\n\n# replace each special string appearing in title and transcript columns\nfor i in range(len(special_strings)):\n    df = df.with_columns(df['title'].str.replace(special_strings[i], \n                          special_string_replacements[i]).alias('title'))\n    df = df.with_columns(df['transcript'].str.replace(special_strings[i], \n                          special_string_replacements[i]).alias('transcript'))\n```\n\n## Load\nSince the dataset here is very small (84 rows and 4 columns, ~900k characters), we can store the data directly in the project directory. This can be done in one line of code using the _write _parquet()_ method in Polars. The final file size is 341 KB.\nbash\n\n```\n# write data to file\ndf.write_parquet('data/video-transcripts.parquet')\n```\n\n> [**YouTube-Blog/full-stack-data-science/data-engineering at main · ShawhinT/YouTube-Blog**](https://github.com/ShawhinT/YouTube-Blog/tree/main/full-stack-data-science/data-engineering)\n## What's Next?\nHere, we discussed the basics of building data pipelines in the context of Full Stack Data Science and walked through a concrete example. Although the example showed the guts of a real-world data pipeline, it isn't clear how we might automate this process.\nIn the [next article](https://towardsdatascience.com/automating-data-pipelines-with-python-github-actions-c19e2ef9ca90) of this series, I will show how we can use GitHub actions to automatically run this data pipeline every day so that fresh data are available to downstream ML tasks.\nMore in this series 👇\n> [**Full Stack Data Science**](https://shawhin.medium.com/list/f0910c75d006)\n* * *\n## Resources\n**Connect** : [My website](https://shawhintalebi.com/) | [Book a call](https://calendly.com/shawhintalebi)\n**Socials** : [YouTube 🎥 ](https://www.youtube.com/channel/UCa9gErQ9AE5jT2DZLjXBIdA) | [LinkedIn](https://www.linkedin.com/in/shawhintalebi/) | [Twitter](https://twitter.com/ShawhinT)\n**Support** : [Buy me a coffee](https://www.buymeacoffee.com/shawhint) ☕️\n> [**Get FREE access to every new story I write**](https://shawhin.medium.com/subscribe)\n* * *\n[1] [How Data Engineering Works](https://www.youtube.com/watch?v=qWru-b6m030&list=WL&index=9)\n[2] [ETL vs ELT](https://aws.amazon.com/compare/the-difference-between-etl-and-elt/)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[Data Engineering](https://towardsdatascience.com/tag/data-engineering/)[Data Pipeline](https://towardsdatascience.com/tag/data-pipeline/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Getting Started](https://towardsdatascience.com/tag/getting-started/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-build-data-pipelines-for-machine-learning-b97bbef050a5%2F&title=How%20to%20Build%20Data%20Pipelines%20for%20Machine%20Learning)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-build-data-pipelines-for-machine-learning-b97bbef050a5%2F&title=How%20to%20Build%20Data%20Pipelines%20for%20Machine%20Learning)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-build-data-pipelines-for-machine-learning-b97bbef050a5%2F&text=How%20to%20Build%20Data%20Pipelines%20for%20Machine%20Learning)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/05/1RX5HU3xUBWB7T6gyzQQpTQ.png)](https://towardsdatascience.com/after-doing-your-research-its-time-to-trust-your-instinct-252412610fc9/)\n## [After Doing Your Research, It's Time to Trust Your Instinct](https://towardsdatascience.com/after-doing-your-research-its-time-to-trust-your-instinct-252412610fc9/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nOn the challenges of data science education, and making complex topics accessible.\n[TDS Editors](https://towardsdatascience.com/author/towardsdatascience/)\nMay 24, 20227 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/0AUAyXBE417VV9rfP-scaled.jpg)](https://towardsdatascience.com/try-this-simple-trick-to-improve-your-clustering-b2d5d502039b/)\n## [Try this simple trick to improve your clustering](https://towardsdatascience.com/try-this-simple-trick-to-improve-your-clustering-b2d5d502039b/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nThe k-means++ algorithm to kick start your initialization\n[Martin Helm](https://towardsdatascience.com/author/martin_87901/)\nJuly 28, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1kLUC6uNCwj5ZnMILtvHPpw.png)](https://towardsdatascience.com/frequentist-a-b-testing-explained-90b9ce8c28b9/)\n## [Frequentist A/B Testing Explained](https://towardsdatascience.com/frequentist-a-b-testing-explained-90b9ce8c28b9/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nThis article will explain the frequentist approach to A/B testing and provide an example with code of when and how to use it\n[Vatsal](https://towardsdatascience.com/author/vatsal12-p/)\nMarch 29, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1p5SAw7kckaClkvwDu8BxoA.jpeg)](https://towardsdatascience.com/best-library-to-simplify-math-for-machine-learning-ed64cbe536ac/)\n## [Best Library To Simplify Math For Machine Learning!](https://towardsdatascience.com/best-library-to-simplify-math-for-machine-learning-ed64cbe536ac/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUsing the sympy Python library to simplify differential and integral calculus for Machine Learning\n[Bharath K](https://towardsdatascience.com/author/bharath-k1297/)\nApril 21, 20217 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/06/0NIF9k1TS8SrVuYO2-scaled.jpg)](https://towardsdatascience.com/spatial-data-science-with-postgresql-geometries-c00387755700/)\n## [Spatial Data Science with PostgreSQL: Geometries](https://towardsdatascience.com/spatial-data-science-with-postgresql-geometries-c00387755700/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA beginners walkthrough in PostGIS Geometry types and functions.\n[Abdishakur](https://towardsdatascience.com/author/shakasom/)\nJune 23, 20205 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1ER0odqnFHLsg2NQvw5WVng.jpeg)](https://towardsdatascience.com/creating-openai-gpts-for-lots-of-fun-and-a-little-profit-331471490ccf/)\n## [Creating OpenAI GPTs for (lots of) Fun and (a little) Profit](https://towardsdatascience.com/creating-openai-gpts-for-lots-of-fun-and-a-little-profit-331471490ccf/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow I created custom GPTs for creative endeavors that draw from works in the public domain\n[Robert A. Gonsalves](https://towardsdatascience.com/author/robgon/)\nMarch 18, 202419 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1qbz9qHMvvf1GC8sw0vH0MA-scaled.jpeg)](https://towardsdatascience.com/evaluation-metrics-for-clustering-models-5dde821dd6cd/)\n## [Evaluation Metrics for Clustering Models](https://towardsdatascience.com/evaluation-metrics-for-clustering-models-5dde821dd6cd/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n3 different metrics for clustering explained\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJanuary 10, 20215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/0S1LtxPOlTKHXele5-scaled.jpg)](https://towardsdatascience.com/public-datasets-for-machine-learning-in-geoscience-cf880862300a/)\n## [Public Datasets for Machine Learning in Geoscience](https://towardsdatascience.com/public-datasets-for-machine-learning-in-geoscience-cf880862300a/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThe best open source websites for applying data science, machine learning and petrophysical techniques\n[Andy McDonald](https://towardsdatascience.com/author/andymcdonaldgeo/)\nApril 17, 20227 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1v7yiaOn7i6tXH8zEgoiTog.png)](https://towardsdatascience.com/machine-learning-code-reproducibility-using-versioning-9a7ec373173a/)\n## [Machine Learning code reproducibility using versioning](https://towardsdatascience.com/machine-learning-code-reproducibility-using-versioning-9a7ec373173a/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nSoftware dependency and version control is a hard and tedious process. But as such, they are the ideal target for automatization.\n[Fran Pérez](https://towardsdatascience.com/author/franperez/)\nJanuary 12, 202111 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "metadata": {
    "title": "How to Build Data Pipelines for Machine Learning | Towards Data Science",
    "description": "A beginner-friendly introduction with Python code",
    "keywords": null,
    "author": "Shaw Talebi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "How to Build Data Pipelines for Machine Learning",
    "og:description": "A beginner-friendly introduction with Python code",
    "og:url": "https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/067YBe2ZDTrnUqJSw-scaled.jpg",
    "og:image:width": "2560",
    "og:image:height": "1920",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Shaw Talebi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "10 min read",
    "article:published_time": "2024-05-02T19:20:10.000Z",
    "article:modified_time": "2024-05-02T19:20:10.000Z",
    "article:author": "https://towardsdatascience.com/author/shawhin/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_000.txt",
      "chars": 130
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_001.txt",
      "chars": 239
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_002.txt",
      "chars": 648
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_003.txt",
      "chars": 831
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_004.txt",
      "chars": 87
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_005.txt",
      "chars": 530
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_006.txt",
      "chars": 223
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_007.txt",
      "chars": 129
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_008.txt",
      "chars": 382
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_009.txt",
      "chars": 112
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_010.txt",
      "chars": 566
    },
    {
      "language": "txt",
      "file": "attachments/0b0b6c3fbcb4/code/block_011.txt",
      "chars": 72
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/067YBe2ZDTrnUqJSw-scaled.jpg",
      "alt": "Photo by the blowup on Unsplash",
      "local_path": "attachments/0b0b6c3fbcb4/images/000.jpg",
      "bytes": 417179,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/155fe8zq2SJhJPVvh5f9aZQ.png",
      "alt": "A DAG for a simple data pipeline. Image by author.",
      "local_path": "attachments/0b0b6c3fbcb4/images/001.png",
      "bytes": 32592,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1afqcojIl5b4UmwdhN838oA.png",
      "alt": "Head of dataframe. Image by author.",
      "local_path": "attachments/0b0b6c3fbcb4/images/002.png",
      "bytes": 19845,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/11-6NXF1O1KnMTM1lDGJqEg.png",
      "alt": "Head of dataframe with transcripts. Image by author.",
      "local_path": "attachments/0b0b6c3fbcb4/images/003.png",
      "bytes": 20494,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1-trYS_otUkUT_pFvhZ-4dA.png",
      "alt": "Head of dataframe after updating datetime dtype. Image by author.",
      "local_path": "attachments/0b0b6c3fbcb4/images/004.png",
      "bytes": 21178,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/05/1RX5HU3xUBWB7T6gyzQQpTQ.png",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/005.png",
      "bytes": 1360330,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/0AUAyXBE417VV9rfP-scaled.jpg",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/006.jpg",
      "bytes": 745025,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1kLUC6uNCwj5ZnMILtvHPpw.png",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/007.png",
      "bytes": 1328480,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1p5SAw7kckaClkvwDu8BxoA.jpeg",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/008.jpeg",
      "bytes": 288329,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/06/0NIF9k1TS8SrVuYO2-scaled.jpg",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/009.jpg",
      "bytes": 727460,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1ER0odqnFHLsg2NQvw5WVng.jpeg",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/010.jpeg",
      "bytes": 291509,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1qbz9qHMvvf1GC8sw0vH0MA-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/011.jpeg",
      "bytes": 737161,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/0S1LtxPOlTKHXele5-scaled.jpg",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/012.jpg",
      "bytes": 603679,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1v7yiaOn7i6tXH8zEgoiTog.png",
      "alt": "",
      "local_path": "attachments/0b0b6c3fbcb4/images/013.png",
      "bytes": 23772,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/channel/UCa9gErQ9AE5jT2DZLjXBIdA",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    },
    {
      "src": "https://www.youtube.com/watch?v=qWru-b6m030&list=WL&index=9",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/shawhin/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/data-science-project-management-e8787d818ad0"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/automating-data-pipelines-with-python-github-actions-c19e2ef9ca90"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-engineering/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-pipeline/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/getting-started/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/after-doing-your-research-its-time-to-trust-your-instinct-252412610fc9/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/towardsdatascience/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/try-this-simple-trick-to-improve-your-clustering-b2d5d502039b/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/martin_87901/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/frequentist-a-b-testing-explained-90b9ce8c28b9/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/full-stack-data-science-f0910c75d006"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/@theblowup"
    },
    {
      "type": "external",
      "href": "https://unsplash.com"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-data-scientist"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/data-engineer"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/YouTube-Blog/tree/main/full-stack-data-science/data-engineering"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/yt-search"
    },
    {
      "type": "external",
      "href": "https://developers.google.com/youtube/v3/getting-started"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/f0910c75d006"
    },
    {
      "type": "external",
      "href": "https://shawhintalebi.com/"
    },
    {
      "type": "external",
      "href": "https://calendly.com/shawhintalebi"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/channel/UCa9gErQ9AE5jT2DZLjXBIdA"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/in/shawhintalebi/"
    },
    {
      "type": "external",
      "href": "https://twitter.com/ShawhinT"
    },
    {
      "type": "external",
      "href": "https://www.buymeacoffee.com/shawhint"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/subscribe"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:49Z",
  "source_url": "https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}