{ "source_file": "0930", "source_platform": "Medium", "author": "Shawhin Talebi", "title_listed": "Automating Data Pipelines with Python & GitHub Actions", "keywords": "数据管道,GitHub Actions", "domain": "数据处理", "content_type": "深度技术解析", "is_paid_flag": "部分免费/会员墙", "url": "https://towardsdatascience.com/automating-data-pipelines-with-python-github-actions-c19e2ef9ca90/", "publish_date": "2024-05", "corpus_stage": "预训练/SFT", "notes": "", "id": "e4081a595d86", "collected_at": "2026-10-01T22:44:45", "title": "Automating Data Pipelines with Python & GitHub Actions | Towards Data Science", "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# Automating Data Pipelines with Python & GitHub Actions\nA simple (and free) way to run data workflows\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nMay 30, 20249 min read\nThis is the 4th article in a [larger series](https://shawhin.medium.com/list/full-stack-data-science-f0910c75d006) on Full Stack Data Science (FSDS). In the [last post](https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5), I shared a concrete example of how to build data pipelines for machine learning projects. One limitation of the example, however, was that the data pipeline had to be run manually. While this might be fine for some applications, more often than not, it's better to automate than delegate this process to a person. In this article, I will walk through a simple way to do this using Python and GitHub Actions.\n![Photo by Chen Mizrach on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/0rrbrX7Jv6VlDdkNE-scaled.jpg)\nPhoto by [Chen Mizrach](https://unsplash.com/@chenhanozel?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n* * *\nA few years ago, Andrej Karpathy gave a talk describing \"Operation Vacation\" [1]. This was what Tesla's full self-driving engineering team called their goal of completely automating the improvement of the self-driving model.\nAlthough this goal was somewhat facetious, it illustrates an aspiration I've seen in most data scientists and engineers: **the desire to build a system that can operate autonomously** (so they can go on vacation).\nHere, I will discuss how to automate a key element of any machine learning system—the data pipeline.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Data Engineer](https://roadmap.sh/data-engineer) roadmaps.\nPlease accept cookies to access this content\n## **2 Ways to Automate Data Pipelines**\nWhile there are countless ways to build and automate data pipelines, here I'll categorize the approaches into two buckets: using an orchestration tool and not using an orchestration tool.\n## **Way 1: Orchestration Tool**\nOrchestration tools allow developers to manage workflows with hundreds (and even thousands) steps.\n### **Airflow**\nOne of the most popular orchestration tools is Airflow, which can manage complex workflows using Python. This has made it a standard among data engineers managing enterprise data pipelines.\nA downside of Airflow, however, is that its setup and maintenance can be complicated. Consequently, it requires a strong technical understanding of how it works, which can take time to develop.\n### **Airflow Wrappers**\nThe complexity of Airflow has led to the rise of Airflow wrappers, which make its core functionality easier to use. Popular examples of these tools include [Dagster](https://docs.dagster.io/getting-started), [Mage](https://docs.mage.ai/introduction/overview), [Astronomer](https://www.astronomer.io/docs/), and [Prefect](https://www.prefect.io/opensource) (_not a wrapper, but comparable_).\nWhile all these tools provide scalable and reliable ways to manage sophisticated data workflows, **they might be overkill for many ML applications**. So, let's take a step back and ask how we can build pipelines without orchestration tools.\n## **Way 2: Python + Triggers**\nAs discussed in the [previous article](https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5), data pipelines consist of **three basic tasks: extraction, transformation, and loading**. All of these can be implemented using Python.\nFor example, if we wanted to pull data from a single data source and store it in a single database, the workflow would look like this.\n![A simple data pipeline. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1s-qAvBPOBXaYFZLPIUHfCQ.png)\nA simple data pipeline. Image by author.\nWhile we could surely use an orchestration tool for this, nothing stops us from consolidating the ETL scripts into a single Python file and running that.\n![Consolidated data pipeline. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1HyhIF4jUCamsU4Yikv2CUw.png)\nConsolidated data pipeline. Image by author.\n### **Automating with Triggers**\nThe above abstraction simplifies the execution of this pipeline, but it's still not automated since we manually have to run the _etl.py_ script. To take the next step, we need to introduce a trigger.\nA **trigger** runs a command when a specific criterion is satisfied. For example, the time is 12:00 AM, or a new file appears in a directory.\nThis final piece allows us to fully automate the data pipeline so we can spend our time doing more productive things e.g. read Medium articles 😉 .\nHere's what our example pipeline would look like if running every day at midnight.\n![Consolidated data pipeline running every day at midnight via cron. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1NFNUDbl9frRFzSfufJqWAA.png)\nConsolidated data pipeline running every day at midnight via cron. Image by author.\n## **GitHub Actions**\nWe can implement the simple workflow described above via **GitHub Actions (GA)**. GA is a **CI/CD (continuous integration, continuous delivery) platform** , which is a fancy way of saying it helps you automate software testing and updating.\nWhile data may traditionally be seen to sit outside of software, these are inseparable when it comes to machine learning. This is because **ML uses data to \"write\" software** (i.e. train models).\nThe biggest upside of using GA is GitHub provides free computing to run actions for public repositories, which is great for poor developers (like me) and simple proof-of-concept projects.\n## **Example Code: Automating ETL Pipeline for YouTube Video Transcripts**\nLet's walk through a concrete example of automating a simple ETL people using GitHub Actions. I'll do this for the example code from the [previous article](https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5) in this series, where I extracted transcripts from all the videos from my YouTube channel.\nExample code is freely available at the [GitHub repository](https://github.com/ShawhinT/data-pipeline-example).\n* * *\n### **Create ETL Python Script**\nThe first step is consolidating the entire ETL (Extract, Transform, Load) pipeline into a single Python script.\nI do that here by defining three files: _functions.py, data _pipeline.py_ , and _requirements.txt_. In _**functions.py**_ , each step of the pipeline is defined as a function. These are then called in sequential order in _**data_pipeline.py** _._**requirements.txt** _ lists all the Python dependencies for the pipeline.\n![Directory tree. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1wgBRc7fKxf2dB6_fpsDd-Q.png)\nDirectory tree. Image by author.\npython\n\n```\n# data_pipeline.py\n\nfrom functions import *\nimport time\nimport datetime\n\nprint(\"Starting data pipeline at \", datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))\nprint(\"----------------------------------------------\")\n\n# Step 1: extract video IDs\nt0 = time.time()\ngetVideoIDs()\nt1 = time.time()\nprint(\"Step 1: Done\")\nprint(\"---> Video IDs downloaded in\", str(t1-t0), \"seconds\", \"n\")\n\n# Step 2: extract transcripts for videos\nt0 = time.time()\ngetVideoTranscripts()\nt1 = time.time()\nprint(\"Step 2: Done\")\nprint(\"---> Transcripts downloaded in\", str(t1-t0), \"seconds\", \"n\")\n\n# Step 3: Transform data\nt0 = time.time()\ntransformData()\nt1 = time.time()\nprint(\"Step 3: Done\")\nprint(\"---> Data transformed in\", str(t1-t0), \"seconds\", \"n\")\n\n# Step 4: Generate text emebeddings\nt0 = time.time()\ncreateTextEmbeddings()\nt1 = time.time()\nprint(\"Step 4: Done\")\nprint(\"---> Embeddings generated in\", str(t1-t0), \"seconds\", \"n\")\n```\n\nI added time printouts for each step in the pipeline to help with code observability and debugging. I won't get into the guts of each step here since those were described in the [previous article](https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5) and [this video](https://youtu.be/6qCrvlHRhcM?si=MYHlNOuO-Q19zsuf&t=2054), but those interested can review the code on [GitHub](https://github.com/ShawhinT/data-pipeline-example).\n### **Create GitHub Repo**\nNext, we create a GitHub repo. You can do this from the command line or the web interface. I'll use the latter.\nTo do this, I go to my GitHub repositories, hit \"New\", and fill out the following fields.\n![Creating GitHub repository. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1W3CXtIutP_SJCXOOnHKfwQ.png)\nCreating GitHub repository. Image by author.\nThen, we can clone the repo locally via the following command.\nbash\n\n```\n>> git clone https://github.com/ShawhinT/data-pipeline-example.git\n```\n\n### **Create Workflow .yml File**\nWe can now create our workflow via a .yml file. To do that, we will create a new folder called .github/workflows and a new file called _data-pipeline.yml_.\n![New directory tree. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1EEvKm2Z7GGedCPU8NneLTw.png)\nNew directory tree. Image by author.\nThe .yml file looks like this.\nyaml\n\n```\nname: data-pipeline-workflow\n\non:\n push: # run on push\n schedule:\n - cron: \"35 0 * * *\" # run every day at 12:35AM\n workflow_dispatch: # manual triggers\n\njobs:\n run-data-pipeline:\n runs-on: ubuntu-latest\n steps:\n - name: Checkout repo content\n uses: actions/checkout@v4\n with:\n token: ${{ secrets.PERSONAL_ACCESS_TOKEN }} # Use the PAT instead of the default GITHUB_TOKEN\n - name: Setup python\n uses: actions/setup-python@v5\n with:\n python-version: '3.9'\n cache: 'pip'\n - name: Install dependencies\n run: pip install -r requirements.txt\n - name: Run data pipeline\n env:\n YT_API_KEY: ${{ secrets.YT_API_KEY }} # import API key\n run: python data_pipeline.py # run data pipeline\n - name: Check for changes # create env variable indicating if any changes were made\n id: git-check\n run: |\n git config user.name 'github-actions'\n git config user.email 'github-actions@github.com'\n git add .\n git diff --staged --quiet || echo \"changes=true\" >> $GITHUB_ENV \n - name: Commit and push if changes\n if: env.changes == 'true' # if changes made push new data to repo\n run: |\n git commit -m \"updated video index\"\n git push\n```\n\nWhile this may seem overwhelming to those new to GitHub Actions, it consists of **4 key elements: name, triggers, jobs, and steps**.\nStarting with the **name** , this is the name of our workflow. Here, I call it _\"data-pipeline-workflow\"_.\nNext, we define **triggers** for the workflow using the \"_on:_ \" syntax. Here, I define three separate triggers. **1) on push** , meaning the workflow will execute when new code is pushed to the repo. **2) on schedule** , meaning it will execute via a cron job. **3) on workflow_dispatch** , this will create a button in our repo we can manually click to run the workflow.\nWorkflows are made up of **jobs**. In this example, we have a single job called \"_run-data-pipeline_ \". We can set the operating system for the job (I use Ubuntu here).\nJobs are then made up of **steps**. Here, our job consists of 6 steps, which are listed below.\n 1. **Checkout repo content** : uses the [pre-built checkout action](https://github.com/actions/checkout) to pull code from the repo. I also provide a Personal Access Token (PAT) to allow the action to push code to the repo, which we will create momentarily.\n 2. **Setup Python** : install the desired Python version with the [pre-built action](https://github.com/actions/setup-python).\n 3. **Install dependencies** : install Python libraries listed in _requirements.txt_.\n 4. **Run data pipeline** : executes the _data _pipeline.py_ script. I add a secret environment variable to allow the action to use my YouTube API token without exposing it on a public repo. I will show you how to do this next.\n 5. **Check for changes** : checks if any changes were made to the repo. This is necessary because if we try to push code to the repo when there are no differences, it will throw an error, and the job will fail to run.\n 6. **Commit and push if changes** : if there are changes to the repo, they will be committed and pushed.\n\n\nOne last thing I'll do is create a folder called \"data\" where we can save the final data files.\n![Updated directory tree with data folder. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1rHO7HhcvIIT7MlNgGvg1zA.png)\nUpdated directory tree with data folder. Image by author.\n### Add Repo Secrets\nNotice in the data-pipeline.yml file I referenced two strange-looking variables, e.g., _${{ secrets.PERSONAL_ACCESS _TOKEN }}_ and _${{ secrets.YT_API _KEY }}._\nThese are **repository secrets** that are accessible to GitHub Actions as environment variables. To create them, we go to our repository settings, click Secrets and Variables, select Actions, and click \"New repository secrets\".\n![Creating a new repository secret. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1mGBdM2iuPSWyuLR8H73Qkw.png)\nCreating a new repository secret. Image by author.\nTo create the _PERSONAL_ACCESS _TOKEN_ variable, click on your profile icon in the top right-hand corner > open Settings in a new tab > scroll to the bottom, and select \"Developer settings\" > click \"Personal access tokens\" > click \"Generate new token\" (classic). This will allow you to create a token that gives the Actions write access to your repo.\nHere are the details I used.\n * **Name** : data-pipeline-example-PAT\n * **Expiration** : No expiration (feel free to set this to end at some point)\n * **Select scopes** : repo\n\n\nThen click \"Generate token\" at the bottom of the page. This will display a long string of text, which you can copy and paste into your GitHub repository secret.\n![Creating _PERSONAL_ACCESSTOKEN. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1OeYhQSEDkgRX0bmjL-eexA.png)\nCreating _PERSONAL_ACCESS _TOKEN. Image by author._\nI then create another secret variable called _YT_API _KEY_ similarly _._\n### Push and commit\nWith our secrets in place, we can commit and push our code.\nbash\n\n```\n>> git add .\n>> git commit -m \"adding data pipeline code\"\n>> git push\n```\n\nOnce pushed, we can go to the repo's \"Actions\" tab and watch the workflow run!\n> [**GitHub - ShawhinT/data-pipeline-example: Example data pipeline automation with GitHub Actions**](https://github.com/ShawhinT/data-pipeline-example)\n## What's next?\nWhile orchestration tools are commonplace in data engineering, these might be overkill for some ML use cases. Here, we saw a free and simple way to automate a data pipeline using Python and GitHub Actions.\nIn the [next article](https://towardsdatascience.com/how-to-deploy-ml-solutions-with-fastapi-docker-and-gcp-de1bb8bfc59a) of this series, we will continue going down the data science tech stack and discuss how we can integrate this data pipeline into a semantic search system for my YouTube videos.\nMore in this series 👇\n> [**Full Stack Data Science**](https://shawhin.medium.com/list/f0910c75d006)\n* * *\n## Resources\n**Connect** : [My website](https://shawhintalebi.com/) | [Book a call](https://calendly.com/shawhintalebi)\n**Socials** : [YouTube 🎥 ](https://www.youtube.com/channel/UCa9gErQ9AE5jT2DZLjXBIdA) | [LinkedIn](https://www.linkedin.com/in/shawhintalebi/) | [Twitter](https://twitter.com/ShawhinT)\n**Support** : [Buy me a coffee](https://www.buymeacoffee.com/shawhint) ☕️\n> [**Get FREE access to every new story I write**](https://shawhin.medium.com/subscribe)\n* * *\n[1] [PyTorch at Tesla - Andrej Karpathy](https://youtu.be/oBklltKXtDE?si=a7aQZ903orc0nqmM)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[Data Engineering](https://towardsdatascience.com/tag/data-engineering/)[Data Science](https://towardsdatascience.com/tag/data-science/)[github-actions](https://towardsdatascience.com/tag/github-actions/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fautomating-data-pipelines-with-python-github-actions-c19e2ef9ca90%2F&title=Automating%20Data%20Pipelines%20with%20Python%20%26%20GitHub%20Actions)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fautomating-data-pipelines-with-python-github-actions-c19e2ef9ca90%2F&title=Automating%20Data%20Pipelines%20with%20Python%20%26%20GitHub%20Actions)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fautomating-data-pipelines-with-python-github-actions-c19e2ef9ca90%2F&text=Automating%20Data%20Pipelines%20with%20Python%20%26%20GitHub%20Actions)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/10kFCtjIpEF6AwlQZg-WG4Q.jpeg)](https://towardsdatascience.com/identifying-signs-of-tiredness-using-sequence-classification-f312e0adb6c/)\n## [Identifying signs of tiredness using sequence classification](https://towardsdatascience.com/identifying-signs-of-tiredness-using-sequence-classification-f312e0adb6c/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nTraining a model to classify tiredness using wearable health data\n[Elliot Humphrey](https://towardsdatascience.com/author/elz1582/)\nAugust 31, 20219 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/1mSmWZ-4zRDIhuiaC3UgkZQ.jpeg)](https://towardsdatascience.com/the-data-analyst-skill-you-should-be-looking-for-f9cf2cd584cc/)\n## [The data analyst skill you should be looking for](https://towardsdatascience.com/the-data-analyst-skill-you-should-be-looking-for-f9cf2cd584cc/)\n[Programming](https://towardsdatascience.com/category/programming/)\nAs an analyst, if you can't write, you can't share. If you can't share, your analysis has no value.\n[Brandon Burns](https://towardsdatascience.com/author/datadfw/)\nSeptember 28, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/0eJB1AceSjATGPbfC.jpg)](https://towardsdatascience.com/dynamic-typing-in-python-307f7c22b24e/)\n## [Dynamic Typing in Python](https://towardsdatascience.com/dynamic-typing-in-python-307f7c22b24e/)\n[Programming](https://towardsdatascience.com/category/programming/)\nExploring How Object References Work in Python\n[Giorgos Myrianthous](https://towardsdatascience.com/author/gmyrianthous/)\nJanuary 10, 202110 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0LshlcePMBU29OyoO-scaled.jpg)](https://towardsdatascience.com/guide-to-implement-linear-regression-in-pyspark-and-r-26a94fe938a3/)\n## [Guide to implement Linear Regression in Pyspark and R](https://towardsdatascience.com/guide-to-implement-linear-regression-in-pyspark-and-r-26a94fe938a3/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWhich programming tool should I use to build my Linear Regression model? Is there any differences in terms of the prediction accuracy...\n[Kieran Tan Kah Wang](https://towardsdatascience.com/author/kierantan/)\nOctober 7, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/09/1BTfjPiIvLkfX9Q-kpPVoSw.png)](https://towardsdatascience.com/why-to-use-seeded-topic-models-in-your-next-project-and-how-to-implement-them-in-r-8502d15d6e8d/)\n## [Seeded Topic Models as a Yard Stick: Implement them in R with keyATM](https://towardsdatascience.com/why-to-use-seeded-topic-models-in-your-next-project-and-how-to-implement-them-in-r-8502d15d6e8d/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nModelling geopolitical risk based on UK parliament transcripts\n[Finn-Ole Höner](https://towardsdatascience.com/author/ds-econ/)\nSeptember 7, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/06/1yrkZTQ2FTXmS4hynR2Snrw-scaled.jpeg)](https://towardsdatascience.com/a-practical-guide-on-missing-values-with-pandas-8fb3e0b46c24/)\n## [A Practical Guide on Missing Values with Pandas](https://towardsdatascience.com/a-practical-guide-on-missing-values-with-pandas-8fb3e0b46c24/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA comprehensive guide to handle missing\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJune 13, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/0aaIHnCSlRFq1NzRd-scaled.jpg)](https://towardsdatascience.com/applying-data-science-in-the-life-insurance-industry-a-perspective-from-a-qualified-actuary-698c197e1826/)\n## [Applying data science in the life insurance industry - a perspective from a qualified actuary](https://towardsdatascience.com/applying-data-science-in-the-life-insurance-industry-a-perspective-from-a-qualified-actuary-698c197e1826/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow data science is changing the traditional landscape for life insurance actuaries (explained with a few use cases)\n[Jin Cui](https://towardsdatascience.com/author/jin-cui/)\nDecember 25, 202113 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/08/14CAEr1mRC1Jv88vZaj_iHw.jpeg)](https://towardsdatascience.com/how-autoencoders-outperform-pca-in-dimensionality-reduction-1ae44c68b42f/)\n## [How Autoencoders Outperform PCA in Dimensionality Reduction](https://towardsdatascience.com/how-autoencoders-outperform-pca-in-dimensionality-reduction-1ae44c68b42f/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDimensionality reduction with autoencoders using non-linear data\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nAugust 19, 20224 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0bT739oxAVSCFDAlp-scaled.jpg)](https://towardsdatascience.com/basic-projects-to-practice-data-science-1e93d7b60799/)\n## [Basic Projects to Practice Data Science](https://towardsdatascience.com/basic-projects-to-practice-data-science-1e93d7b60799/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nStart Your Data Science Journey With These 6 Basic Data Science Projects\n[Przemyslaw Jarzynski](https://towardsdatascience.com/author/pjwebdev/)\nOctober 23, 20208 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "metadata": { "title": "Automating Data Pipelines with Python & GitHub Actions | Towards Data Science", "description": "A simple (and free) way to run data workflows", "keywords": null, "author": "Shaw Talebi", "og:locale": "en_US", "og:type": "article", "og:title": "Automating Data Pipelines with Python & GitHub Actions", "og:description": "A simple (and free) way to run data workflows", "og:url": "https://towardsdatascience.com/automating-data-pipelines-with-python-github-actions-c19e2ef9ca90/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/0rrbrX7Jv6VlDdkNE-scaled.jpg", "og:image:width": "2560", "og:image:height": "1444", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Shaw Talebi", "twitter:label2": "Est. reading time", "twitter:data2": "9 min read", "article:published_time": "2024-05-30T17:51:56.000Z", "article:modified_time": "2024-05-30T17:51:56.000Z", "article:author": "https://towardsdatascience.com/author/shawhin/" }, "paywall": { "listed_paid": true, "detected": false, "marker": "", "public_only": true, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 202, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/e4081a595d86/code/block_000.txt", "chars": 915 }, { "language": "txt", "file": "attachments/e4081a595d86/code/block_001.txt", "chars": 67 }, { "language": "txt", "file": "attachments/e4081a595d86/code/block_002.txt", "chars": 1321 }, { "language": "txt", "file": "attachments/e4081a595d86/code/block_003.txt", "chars": 70 } ], "images": [ { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/0rrbrX7Jv6VlDdkNE-scaled.jpg", "alt": "Photo by Chen Mizrach on Unsplash", "local_path": "attachments/e4081a595d86/images/000.jpg", "bytes": 166714, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1s-qAvBPOBXaYFZLPIUHfCQ.png", "alt": "A simple data pipeline. Image by author.", "local_path": "attachments/e4081a595d86/images/001.png", "bytes": 28786, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1HyhIF4jUCamsU4Yikv2CUw.png", "alt": "Consolidated data pipeline. Image by author.", "local_path": "attachments/e4081a595d86/images/002.png", "bytes": 17341, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1NFNUDbl9frRFzSfufJqWAA.png", "alt": "Consolidated data pipeline running every day at midnight via cron. Image by author.", "local_path": "attachments/e4081a595d86/images/003.png", "bytes": 26582, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1wgBRc7fKxf2dB6_fpsDd-Q.png", "alt": "Directory tree. Image by author.", "local_path": "attachments/e4081a595d86/images/004.png", "bytes": 4104, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1W3CXtIutP_SJCXOOnHKfwQ.png", "alt": "Creating GitHub repository. Image by author.", "local_path": "attachments/e4081a595d86/images/005.png", "bytes": 88958, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1EEvKm2Z7GGedCPU8NneLTw.png", "alt": "New directory tree. Image by author.", "local_path": "attachments/e4081a595d86/images/006.png", "bytes": 7191, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1rHO7HhcvIIT7MlNgGvg1zA.png", "alt": "Updated directory tree with data folder. Image by author.", "local_path": "attachments/e4081a595d86/images/007.png", "bytes": 7586, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1mGBdM2iuPSWyuLR8H73Qkw.png", "alt": "Creating a new repository secret. Image by author.", "local_path": "attachments/e4081a595d86/images/008.png", "bytes": 71874, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1OeYhQSEDkgRX0bmjL-eexA.png", "alt": "Creating _PERSONAL_ACCESSTOKEN. Image by author.", "local_path": "attachments/e4081a595d86/images/009.png", "bytes": 22329, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/10kFCtjIpEF6AwlQZg-WG4Q.jpeg", "alt": "", "local_path": "attachments/e4081a595d86/images/010.jpeg", "bytes": 346905, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/1mSmWZ-4zRDIhuiaC3UgkZQ.jpeg", "alt": "", "local_path": "attachments/e4081a595d86/images/011.jpeg", "bytes": 189068, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/0eJB1AceSjATGPbfC.jpg", "alt": "", "local_path": "attachments/e4081a595d86/images/012.jpg", "bytes": 50036, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0LshlcePMBU29OyoO-scaled.jpg", "alt": "", "local_path": "attachments/e4081a595d86/images/013.jpg", "bytes": 545667, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/09/1BTfjPiIvLkfX9Q-kpPVoSw.png", "alt": "", "local_path": "attachments/e4081a595d86/images/014.png", "bytes": 568946, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/06/1yrkZTQ2FTXmS4hynR2Snrw-scaled.jpeg", "alt": "", "local_path": "attachments/e4081a595d86/images/015.jpeg", "bytes": 100801, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/0aaIHnCSlRFq1NzRd-scaled.jpg", "alt": "", "local_path": "attachments/e4081a595d86/images/016.jpg", "bytes": 378604, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/08/14CAEr1mRC1Jv88vZaj_iHw.jpeg", "alt": "", "local_path": "attachments/e4081a595d86/images/017.jpeg", "bytes": 59724, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0bT739oxAVSCFDAlp-scaled.jpg", "alt": "", "local_path": "attachments/e4081a595d86/images/018.jpg", "bytes": 476715, "downloaded": true } ], "audio": [], "videos": [ { "src": "https://youtu.be/6qCrvlHRhcM?si=MYHlNOuO-Q19zsuf&t=2054", "quality": "unavailable", "local_path": "", "note": "无1080p且无720p，未交付更低清晰度" }, { "src": "https://www.youtube.com/channel/UCa9gErQ9AE5jT2DZLjXBIdA", "quality": "unavailable", "local_path": "", "note": "无1080p且无720p，未交付更低清晰度" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/shawhin/" }, { "type": "internal", "href": "https://towardsdatascience.com/how-to-build-data-pipelines-for-machine-learning-b97bbef050a5" }, { "type": "internal", "href": "https://towardsdatascience.com/how-to-deploy-ml-solutions-with-fastapi-docker-and-gcp-de1bb8bfc59a" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/data-engineering/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/github-actions/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/hands-on-tutorials/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/python/" }, { "type": "internal", "href": "https://towardsdatascience.com/identifying-signs-of-tiredness-using-sequence-classification-f312e0adb6c/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/elz1582/" }, { "type": "internal", "href": "https://towardsdatascience.com/the-data-analyst-skill-you-should-be-looking-for-f9cf2cd584cc/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/programming/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://shawhin.medium.com/list/full-stack-data-science-f0910c75d006" }, { "type": "external", "href": "https://unsplash.com/@chenhanozel" }, { "type": "external", "href": "https://unsplash.com" }, { "type": "external", "href": "https://roadmap.sh/ai-data-scientist" }, { "type": "external", "href": "https://roadmap.sh/data-engineer" }, { "type": "external", "href": "https://docs.dagster.io/getting-started" }, { "type": "external", "href": "https://docs.mage.ai/introduction/overview" }, { "type": "external", "href": "https://www.astronomer.io/docs/" }, { "type": "external", "href": "https://www.prefect.io/opensource" }, { "type": "external", "href": "https://github.com/ShawhinT/data-pipeline-example" }, { "type": "external", "href": "https://youtu.be/6qCrvlHRhcM?si=MYHlNOuO-Q19zsuf&t=2054" }, { "type": "external", "href": "https://github.com/actions/checkout" }, { "type": "external", "href": "https://github.com/actions/setup-python" }, { "type": "external", "href": "https://shawhin.medium.com/list/f0910c75d006" }, { "type": "external", "href": "https://shawhintalebi.com/" }, { "type": "external", "href": "https://calendly.com/shawhintalebi" }, { "type": "external", "href": "https://www.youtube.com/channel/UCa9gErQ9AE5jT2DZLjXBIdA" } ], "repair_version": 2, "repaired_at": "2026-10-14T11:11:49Z", "source_url": "https://towardsdatascience.com/automating-data-pipelines-with-python-github-actions-c19e2ef9ca90/", "delivery_status": "excluded_paid", "discovery_status": "excluded_paid", "missing_reason": "按原约定排除付费/会员全文" }