保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Engineering

I Built My First ETL Pipeline as a Complete Beginner. Here’s How.

A beginner's honest walkthrough of Extract, Transform, Load using the GitHub API Ibrahim Salami May 25, 20266 min read Generated with Gemini This is part two of my data engineering journey series. In part one, I shared my 12-month roadmap for transitioning from data analyst to data engineer . This is where the actual building begins.

When I published my first article documenting my data engineering journey, something unexpected happened. People resonated with it. I had strangers reaching out saying they were excited to follow along. That felt good. But it also came with pressure. Suddenly this wasn't just a personal goal I could quietly abandon if things got hard. People were watching. People were in the same boat. And that accountability, honestly, is part of why you're reading this right now. So I had to move. And like anyone starting a new skill, the first thing I did was look for resources. There are countless tutorials on the internet for data engineering. YouTube videos, courses, written guides. More than you could ever finish. But I couldn't bring myself to just consume theory. I needed to build something. Something real, with real data, that actually worked at the end. So I closed the tutorials and opened a Google Colab notebook instead. I found the GitHub API documentation and decided I was going to build my first ETL pipeline from scratch. No hand-holding. Just me, some Python, and a goal. This article is that experience documented in full. The code, the confusion, the small wins, and what I actually learned by doing it. Learn this step by step with the interactive Data Engineer roadmap.

First, what is ETL?

Before I get into what I built, let me quickly explain what ETL actually means because I had to look this up myself not too long ago. ETL stands for Extract, Transform, Load. It's one of the most fundamental concepts in data engineering.

That's it. Those three steps, done in sequence, are what a data pipeline is. Everything else in data engineering, Airflow, Spark, Databricks, is just more sophisticated ways of doing those same three things at scale. I'm at the beginning of my roadmap, so I kept it simple. Pure Python, no orchestration tools yet. But the shape of the problem is the same.

What I built

I extracted data from the GitHub API, specifically the most starred Python repositories created in the last 30 days. I then cleaned it, added a new column, and saved the output as a CSV file.
Simple. Real. Entirely mine. Here's how it went.

Step 1: Extract

The first thing I had to do was figure out how to talk to the GitHub API. An API is basically a door that a company or platform opens so that developers can request data from it programmatically, without having to manually copy and paste anything. GitHub has a free, public API. No account or paid plan needed for basic searches. Here's the code I wrote to extract the data: python

import requests

url = "https://api.github.com/search/repositories"

params = {
    "q": "language:python created:>2025-04-22",
    "sort": "stars",
    "order": "desc",
    "per_page": 30
}

response = requests.get(url, params=params)
data = response.json()

print(response.status_code)
print(data.keys())

I'll be honest. This block confused me at first. The requests library was new to me. The params dictionary with that q syntax felt alien. I didn't immediately know what .json() was doing or why I needed it. Let me break it down simply.

When I ran it, I got this: pascal

200 
dict_keys(['total_count', 'incomplete_results', 'items'])

The 200 means success. That's the internet's way of saying "your request worked." If you see 403 or 404, something went wrong.
The dictionary has three keys. total_count tells you how many repos matched the search. incomplete_results tells you if GitHub had to cut anything short. And items is where the actual data lives. I then ran a second block to peek inside: python

print("Total matches on GitHub:", data['total_count'])
print("Repos returned:", len(data['items']))

first_repo = data['items'][0]
print("\nFirst repo name:", first_repo['name'])
print("Stars:", first_repo['stargazers_count'])
print("Language:", first_repo['language'])
print("URL:", first_repo['html_url'])

Output: markup

Total matches on GitHub: 9228201Repos returned: 30First repo name: skillsStars: 139136Language: PythonURL: https://github.com/anthropics/skills

The first result was an Anthropic repo with 139k stars. Real data. Live. Pulled by code I wrote. That's Extract done.

Step 2: Transform

Now I had 30 repos sitting in a Python list, each one a nested dictionary with dozens of fields. Most of which I didn't need. The Transform step is where you take that raw, messy data and shape it into something clean and purposeful. First I pulled out only the fields I cared about and loaded them into a Pandas dataframe: python

import pandas as pd

repos = []

for repo in data['items']:
    repos.append({
        "name": repo['name'],
        "owner": repo['owner']['login'],
        "stars": repo['stargazers_count'],
        "forks": repo['forks_count'],
        "language": repo['language'],
        "description": repo['description'],
        "url": repo['html_url'],
        "created_at": repo['created_at']
    })

df = pd.DataFrame(repos)
df.head()

Seeing that dataframe appear was a proper "wow" moment. I went from a wall of JSON to a clean, readable table with labelled columns in a few lines. Then I did three transformations: python

# Drop rows where description is missing
df_clean = df.dropna(subset=['description'])

# Add a viral flag for repos with over 50k stars
df_clean = df_clean.copy()
df_clean['viral'] = df_clean['stars'].apply(lambda x: 'Yes' if x > 50000 else 'No')

# Sort by stars descending
df_clean = df_clean.sort_values('stars', ascending=False).reset_index(drop=True)

print("Before cleaning:", len(df))
print("After cleaning:", len(df_clean))

Output: markup

Before cleaning: 30 After cleaning: 29

One repo had no description and got dropped. The viral column showed up cleanly. The data was now sorted and structured.
That's Transform done.

Step 3: Load

The final step. Take the clean data and save it somewhere. I kept this simple and loaded it into a CSV file: python

df_clean.to_csv('github_trending_repos.csv', index=False)

print("Pipeline complete. File saved.")
print(f"{len(df_clean)} repos loaded into github_trending_repos.csv")

Output: markup

Pipeline complete. File saved.29 repos loaded into github_trending_repos.csv

I downloaded the file and opened it. A clean spreadsheet with 29 rows and 9 columns. Real GitHub data, shaped and saved by a pipeline I built from scratch. That's Load done.

What this actually felt like

Before this, whenever I wanted data to work with, I'd go looking for a public dataset someone had already cleaned and uploaded. Kaggle, Google Dataset Search, wherever. I was always a consumer of data that someone else had prepared. This changed something for me. The moment I realised I could just point Python at an API I was curious about and extract live data myself, the possibilities felt completely different. I'm not limited to datasets that already exist. I can build the pipeline that creates the dataset. That's a different kind of power. And it's one of the things that drew me toward data engineering in the first place.

What's next

This pipeline is simple by design. I'm at the start of my roadmap and I'm not going to pretend I'm using Airflow or Spark yet. But the foundation is real. Extract, Transform, Load. It works. I built it. I understand it. The next step is to make it more robust. Schedule it to run daily. Store the output in a SQLite database instead of a flat CSV. Start tracking how repos trend over time. And eventually, orchestrate the whole thing with Airflow. But that's a future article. For now, the most important thing I proved to myself is that building teaches you things that watching never will. I spent weeks in tutorial land and barely moved. I spent one afternoon actually building, and I understand ETL better than any video made it feel. Stop watching. Start building.

This is part two of my ongoing data engineering series. Follow along as I document every step of the journey, including the parts that don't go smoothly. Feel free to check out my more in-depth ETL take on my YouTube channel below. Connect with me on LinkedIn ,YouTube , and Twitter .

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Ibrahim Salami

See all from Ibrahim Salami Data PipelineData ScienceEtlExtract Transform LoadGithub Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Explore Fireballs with the NASA Fireball Data API, Python, and Tableau

Data Science Fireballs are brighter-than-usual meteors that become visible at about 100 km above sea level Randy Runtsch April 16, 20225 min read

Read This If You're Considering a Career in Cloud Computing

Programming A complete career guide covering the roles and responsibilities, technical skills, general salary range, and relevant certifications Arunn Thevapalan June 28, 20217 min read

Introducing k-Nearest Neighbors

Machine Learning Understanding the fundamentals of this classification model Peter Grant July 21, 20194 min read

Combine and Preprocess Your Heterogeneous Data for Analytics with Apache Flink

Tired of accessing multiple systems for your analytics? This architecture cleanses and merges real-time and historical data in one place Philip Singer December 16, 202110 min read

How to Prepare your Development Environment to Rank on Kaggle

Data Science Best Practices to Rank on Kaggle with PyTorch Lightning and Grid Spot Instances (Part 1/5) イルカ Borovec August 25, 20215 min read

Reduced-Rank Vector Autoregressive Model for High-Dimensional Time Series Forecasting

Machine Learning Introduction Xinyu Chen (陈新宇) October 16, 20215 min read

The Evolution of SQL

Data Science Unlocking the power of large language models Mike Shakhomirov August 18, 202412 min read

A Simple Guide To Pandas DataFrames

Programming How to create, store, and manipulate data using Python's Pandas library Sam Starkman October 1, 202110 min read

Productionize LLM RAG App in Django - Part I: Celery

Data Science Automate Pinecone Daily Upsert Task with Celery and Slack monitoring Wen Yang April 15, 20249 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close