保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Engineering

I Tried to Schedule My ETL Pipeline. Here’s What I Didn’t Expect.

What I thought was a scheduling problem turned out to be a portability problem first Ibrahim Salami June 19, 20267 min read Generated by Gemini AI In my last article, I mentioned that scheduling is the next wall I'll be walking toward. So I guess, here I am, walking towards it But before I get into what happened, let me give some context for anyone stumbling on this for the first time. I am a systems analyst who decided to transition into data engineering. Instead of just taking courses and collecting certificates, I decided to learn by building and writing about it publicly. Every article in this series documents something I actually built, the decisions I made, the things that broke, and what I learned from it. The first article was my 12-month self-study roadmap, where I laid out the plan for how I was going to approach this transition. The second was me building my first ETL pipeline from scratch using the GitHub API, as a complete beginner. In the third, I took that same pipeline and made it more production-ready by adding SQLite storage, idempotency handling, and Google Drive persistence, all inside Google Colab. This article is the fourth. And it picks up exactly where the last one ended. I expected to spend most of my time picking a scheduling tool and configuring it. What I didn't expect was that before I could even think about scheduling, I had to deal with something more fundamental. My pipeline couldn't run outside of Google Colab. And until that changed, no scheduler in the world could help me. This is the story of what actually happened. Learn this step by step with the interactive Data Engineer roadmap.

The First Wall: My Pipeline Lived in Colab

Before I even got to scheduling, I wanted to understand what it would actually take to run my pipeline automatically. So I looked at my code properly for the first time with that question in mind. Here's what the load section looked like: python

conn = sqlite3.connect('/content/drive/MyDrive/github_repos.db')

That path, /content/drive/MyDrive/, only exists inside Google Colab. It's the mounted Google Drive path that Colab gives you when you connect your Drive to a notebook. Outside Colab, that path doesn't exist. If any scheduler tried to run this script, it would crash right there. The interesting thing is that my code had no google.colab imports. No Colab-specific libraries. Just one hardcoded path that I had been typing without really thinking about it. That path was the dependency, not the code. This was the first thing I didn't expect. I thought the challenge would be learning a scheduling tool. Instead, the first lesson was that my environment was part of my pipeline, and I hadn't noticed. The fix was simple. Instead of hardcoding the Colab path, I made the database path configurable through an environment variable: python

import os

DB_PATH = os.environ.get('DB_PATH', 'github_repos.db')
conn = sqlite3.connect(DB_PATH)

Now the script uses whatever path is set in the environment. If nothing is set, it falls back to creating a local github_repos.db file in the same folder. One change, and the pipeline was no longer tied to Colab.

Running It Outside Colab for the First Time

Before setting up any scheduler, I wanted to confirm the script actually worked on its own. So I saved it as pipeline.py, created a requirements.txt with the two libraries it needs: text

requests
pandas

And ran it from my terminal: It printed: Pipeline complete. Duplicates handled. And a file called github_repos.db appeared in my folder. The same pipeline I had been running in Colab was now running as a plain Python script, anywhere. That felt like a bigger deal than I expected. Not because the change was complex, it wasn't. But because I realized I had been thinking of my pipeline as a notebook, when what I actually had was a script that happened to live inside one.

Choosing a Scheduling Tool

At this point I had a standalone script. Now I needed something to run it on a schedule. I looked at a few options. APScheduler lets you define schedules inside your Python code, which works while a session is running but stops the moment you close your terminal. That's not really scheduling, that's just a loop. Airflow is the industry standard for orchestrating pipelines, but it requires running a server, a metadata database, and a web interface. That's a lot of infrastructure for where I am right now. GitHub Actions sat in the middle. It's free, it runs on GitHub's servers, the schedule is defined in code, and it doesn't require me to maintain any infrastructure. The tradeoff is that it's designed for CI/CD workflows, not pipeline orchestration, so it has limits around complex dependencies and monitoring. But for a pipeline at my stage, it's a practical choice. I also want to be honest: tools like Airflow exist for a reason. When a pipeline grows, when you have dependencies between tasks, when you need visibility into what ran and what failed, you need proper orchestration. GitHub Actions is not that. But it's a good first step, and understanding why it's limited is part of learning what those more serious tools are actually solving.

Setting Up GitHub Actions

GitHub Actions works through workflow files, which are YAML files you place in a specific folder in your repository. The folder structure looks like this: text

github-etl/
├── .github/
│   └── workflows/
│       └── schedule.yml
├── pipeline.py
└── requirements.txt

Here's the full workflow file I created: yaml

name: Run ETL Pipeline

on:
  schedule:
    - cron: '0 9 * * *'
  workflow_dispatch:

jobs:
  run-pipeline:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run pipeline
        run: python pipeline.py

Let me walk through what each part is doing.

The steps are straightforward. Checkout pulls your code from the repository into the runner. Setup Python installs the version you specify. Install dependencies runs pip install -r requirements.txt. And then Run pipeline executes your script.

What Happened When I Ran It

After pushing the workflow file to GitHub, I went to the Actions tab in my repository and triggered it manually using the workflow_dispatch button. It ran. Twenty-seven seconds from start to finish. The pipeline pulled data from the GitHub API, transformed it, and loaded it into SQLite, all on a GitHub server, without me doing anything after clicking the button. I did get one warning on the first run: text

Node.js 20 actions are deprecated...

This was because I had used older versions of the checkout and setup-python actions. The fix was updating actions/checkout@v3 to actions/checkout@v4 and actions/setup-python@v4 to actions/setup-python@v5. After that, the workflow ran clean.

What I Actually Learned

Going into this, I thought scheduling was about picking the right tool. What I found was that scheduling forced me to think about something I hadn't thought carefully about before: portability. A pipeline that only runs in one specific environment isn't really a pipeline. It's a script tied to a platform. Making it schedulable meant making it portable first, and making it portable meant understanding what it actually depended on. The hardcoded path was a small thing. But catching it changed how I think about writing pipeline code going forward. Every time I write a path or a credential or an environment-specific value, I now ask whether that thing will exist outside the context I'm building in. The other thing I learned is that scheduling and orchestration are different problems. GitHub Actions handles scheduling well. It doesn't handle things like retrying failed runs with backoff, alerting when something goes wrong, visualizing pipeline dependencies, or managing multiple pipelines that depend on each other. Those are orchestration problems, and they're what tools like Airflow are built to solve. I'm not there yet. But I understand now why those tools exist in a way I didn't before.

What's Next

The pipeline is now running every day at 9am UTC. Data is being collected. And I'm starting to notice something: when you have a pipeline running daily, you start caring about the data it produces in a different way. Are all the records clean? Are there repos slipping through with missing fields? Is the viral flag actually meaningful, or did I define it in a way that makes almost everything "No"? Those are data quality questions. And they're the next wall I'm walking toward. This is part of my ongoing series documenting my transition from systems analyst to data engineer. If you've been following along, thank you. If this is your first article in the series, the earlier ones are linked below. From Data Analyst to Data Engineer: My 12-Month Self-Study Roadmap I Built My First ETL Pipeline as a Complete Beginner. Here’s How. I Thought Data Engineering Was Just Writing Scripts. I Was Wrong. Connect with me on LinkedIn ,YouTube , and Twitter .

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Ibrahim Salami

See all from Ibrahim Salami Data EngineerData PipelineData ScienceEtlGoogle Colab Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

MapReduce: How It Powers Scalable Data Processing

Data Engineering An overview of the MapReduce programming model and how it can be used to optimize large-scale data processing. Jimin Kang April 22, 202512 min read

How to Expose Delta Tables via REST APIs

Data Engineering Three architectures discussed and tested to serve delta tables René Bremer May 6, 20247 min read

5 Python Mistakes You Should Avoid

Programming Some of the most common bugs are hard to find Pranjal Saxena September 2, 20224 min read

Getting Started with Cron Job in the Linux Server: A Complete Tutorial for Beginner

Programming Automate your regular job using cron Audhi Aprilliant November 11, 20205 min read

Streaming in Data Engineering

Data Science Streaming data pipelines and real-time analytics Mike Shakhomirov December 12, 20239 min read

Data: Where Engineering and Science Meet

Data Science Our weekly selection of must-read Editors' Picks and original features TDS Editors September 29, 20223 min read

Feature Engineering with Microsoft Fabric and PySpark

Data Science Fabric Madness part 2 Roger Noble April 8, 202413 min read

Data Engineering: A Formula 1-inspired Guide for Beginners

Data Engineering A Glossary with Use Cases for First-Timers in Data Engineering Matteo Consoli December 4, 20236 min read

2020 NFL Postseason Predictions from Machine Learning Model - Super Bowl

Machine Learning Chiefs are 1-2 point favorite, but model might be undervaluing Chiefs offense Nasir Bhanpuri, PhD February 6, 20214 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close