保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Programming

Pandas Isn’t Going Anywhere: Why It’s Still My Go-To for Data Wrangling

Billions of rows might be the exception, but for everything else, Pandas is still a highly reliable tool. Soner Yıldırım May 17, 20265 min read When I first started learning data science in 2020, Pandas was one of the most popular tools. Although new tools focus on improving Pandas' weaknesses in handling very large datasets, I still use Pandas for many data cleaning, processing, and analysis tasks. Yes, Pandas gives me a hard time when working with billions of rows, but it is definitely more than enough for working with anything below that. I see Pandas being used in not only for EDA or in notebooks but also in production systems. In this article, I'll go over some data cleaning and processing operations to demonstrate how capable Pandas is. Let's start with the dataset, which contains stock keeping units (SKUs) and a search API responses for these SKUs. python

import pandas as pd

search_results = pd.read_csv("search_results.csv")

search_results.head()

Search result is a list of dictionaries and looks like this: python

search_results.loc[0, "search_result"]

"[{'my_id': 'HBCV00007F5Y2B', 'distance': 1.0, 'entity': {}}, 
{'my_id': 'HBCV00007UPQBM', 'distance': 1.0, 'entity': {}}, 
{'my_id': 'HBCV00008I29IH', 'distance': 1.0, 'entity': {}}, 
{'my_id': 'HBCV00006U3ZYB', 'distance': 0.8961254358291626, 'entity': {}}, 
{'my_id': 'HBCV0000AFA4H6', 'distance': 0.8702399730682373, 'entity': {}}, 
{'my_id': 'HBCV00009CDGD4', 'distance': 0.86175537109375, 'entity': {}}, 
{'my_id': 'HBCV000046336T', 'distance': 0.8594968318939209, 'entity': {}}, 
{'my_id': 'HBCV00009QDZRT', 'distance': 0.8572311997413635, 'entity': {}}, 
{'my_id': 'HBCV00008E11P3', 'distance': 0.8553324937820435, 'entity': {}}, 
{'my_id': 'HBV00000C4IY6', 'distance': 0.8539167642593384, 'entity': {}}] 
... and 5 entities remaining"

As we see in the output, it's not a proper list of dictionary format because of the last part ("... and 5 entities remaining"). Also, it's saved as a single string. In order to make better use of it, we need to convert it to a proper list of dictionaries. The following line of code removes the last part by splitting the string at "..." and takes the first split. python

search_results.loc[0, "search_result"].split("...")[0].strip()

However, the output is still a single string. We can use the built-in ast module of Python to convert it to a list: python

import ast

res = ast.literal_eval(search_results.loc[0, "search_result"].split("...")[0].strip())

res

[{'my_id': 'HBCV00007F5Y2B', 'distance': 1.0, 'entity': {}},
 {'my_id': 'HBCV00007UPQBM', 'distance': 1.0, 'entity': {}},
 {'my_id': 'HBCV00008I29IH', 'distance': 1.0, 'entity': {}},
 {'my_id': 'HBCV00006U3ZYB', 'distance': 0.8961254358291626, 'entity': {}},
 {'my_id': 'HBCV0000AFA4H6', 'distance': 0.8702399730682373, 'entity': {}},
 {'my_id': 'HBCV00009CDGD4', 'distance': 0.86175537109375, 'entity': {}},
 {'my_id': 'HBCV000046336T', 'distance': 0.8594968318939209, 'entity': {}},
 {'my_id': 'HBCV00009QDZRT', 'distance': 0.8572311997413635, 'entity': {}},
 {'my_id': 'HBCV00008E11P3', 'distance': 0.8553324937820435, 'entity': {}},
 {'my_id': 'HBV00000C4IY6', 'distance': 0.8539167642593384, 'entity': {}}]

We now have the search results as a proper list of dictionaries. This was only for a single row. We need to apply the same operation to all SKUs (i.e. entire SKU column). One option is to go over all the rows in a for loop and perform the same operation. However, this is not the best option. We should prefer vectorized operations when we can. A vectorized operation basically means executing the code on all rows at once. On a single row, I used splitting to get rid of the last part of the string but it did not work in a vectorized operation. A more robust option seems to be using a regex. python

search_results.loc[:, 'search_result'] = search_results['search_result'].str.replace(r"\.\.\..*", "", regex=True).str.strip()

This code selects "..." and everything that comes after it and replaces them with nothing. In other words, it removes "... and 5 entities remaining" part. We now have all the rows in the search results column as a proper list of dictionaries. python

search_results.loc[10, "search_result"]

"[{'my_id': 'HBCV00007F5Y2B', 'distance': 1.0, 'entity': {}},
 {'my_id': 'HBCV00007UPQBM', 'distance': 1.0, 'entity': {}},
 {'my_id': 'HBCV00008I29IH', 'distance': 1.0, 'entity': {}},
 {'my_id': 'HBCV00006U3ZYB', 'distance': 0.8961254358291626, 'entity': {}},
 {'my_id': 'HBCV0000AFA4H6', 'distance': 0.8702399730682373, 'entity': {}},
 {'my_id': 'HBCV00009CDGD4', 'distance': 0.86175537109375, 'entity': {}},
 {'my_id': 'HBCV000046336T', 'distance': 0.8594968318939209, 'entity': {}},
 {'my_id': 'HBCV00009QDZRT', 'distance': 0.8572311997413635, 'entity': {}},
 {'my_id': 'HBCV00008E11P3', 'distance': 0.8553324937820435, 'entity': {}},
 {'my_id': 'HBV00000C4IY6', 'distance': 0.8539167642593384, 'entity': {}}]"

They're still saved as a string but I can easily convert them to a list using the ast module, which I will do in the next step. What I'm interested in is the SKUs returned in the search results. I'll create a new column by extracting the SKUs in the dictionaries. I can access them using the "my_id" key of the dictionary. There are 3 parts of this operation:

We can do all these operations by applying a lambda function to all rows as follows: python

search_results.loc[:, "result_skus"] = \
search_results["search_result"].apply(lambda x: [item['my_id'] for item in ast.literal_eval(x)])

search_results.head()

Each row in the result_skus column contains a list of 10 SKUs. Let's say I need to have these 10 SKUs in different rows. For each row in the sku column, there will be 10 rows created from the list in the result_skus column. There is a very simple way of doing this in Pandas, which is the explode function. python

data = search_results[["sku", "result_skus"]].explode("result_skus", ignore_index=True)

data.head()

We created a new dataframe with sku and result_skus column. The drawing below demonstrates what the explode function does: Consider the opposite. We have a dataframe as shown above but want to have all results for an sku in a single row. We can use the groupby function to group the rows by sku and then apply the list function on the result_skus column: python

new_data = data.groupby("sku", as_index=False)["result_skus"].apply(list)

new_data.head()

This will get us back to the previous step: Using the explode function, we created a dataframe with a separate row for each sku in the result_skus column. What if we need to have them separated to different columns instead of rows? One option is to apply the pd.Series function to the result_skus column and concatenate the resulting columns to the original dataframe. python

new_cols = new_data["result_skus"].apply(pd.Series)

new_data = pd.concat([new_data, new_cols], axis=1)

new_data.head()

Columns from 0 to 9 contains the 10 SKUs in the result_skus column. This code using the apply function is not a vectorized operation. We have another option, which is vectorized and much faster. python

new_cols = pd.DataFrame(new_data["result_skus"].tolist())

new_data = pd.concat([new_data, new_cols], axis=1)

This code will give us the same dataframe as above but much faster. I demonstrated a typical data cleaning and processing task a data scientist or analyst may encounter in their job. I've been in the field for over 5 years and Pandas has always been enough to do what I need except for when working very large datasets (e.g. billions of rows). The tools that are better fit for such large datasets have similar syntax to Pandas. For example, PySpark is kind of a mixture of Pandas and SQL. Polars is very similar to Pandas in terms of syntax. Thus, learning and practicind Pandas is still a highly valuable skill for anyone working in the data science and AI domain. Thank you for reading.

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Soner Yıldırım

See all from Soner Yıldırım Data AnalysisData ScienceMachine LearningPandasPython Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

How To Perform Feature Selection for Regression Problems

Machine Learning In this article I explain what feature selection is and how to perform it before training a regression model in Python. Serafeim Loukas, PhD August 8, 20206 min read

Pandas: Combining Data

Machine Learning PyTrix #5: More Data Manipulation with Pandas Kurtis Pykes July 13, 20207 min read

Utilization of Twitter's API with R - Advanced Techniques.

Programming The maddening adventures of extracting millions of tweets. Justin Cocco November 21, 202025 min read

An Introduction to Nine Essential Machine Learning Algorithms

Artificial Intelligence Intuitive explanations of the most popular machine learning models. Terence Shin, MSc, MBA April 2, 20207 min read

An Intuitive Guide to Auto-Encoders: Theory, Code and Visualization

Machine Learning Using Auto-Encoders to gain insight into data Victor Sim September 10, 20205 min read

Topic Modeling and Sentiment Analysis on Amazon Alexa Reviews - Part II

Data Science Follow up to "Topic Modeling and Sentiment Analysis on Amazon Alexa Reviews" analyzing and comparing different Echo models. Muriel Kosaka December 15, 20206 min read

Okay, You've Trained the Best Machine Learning Model. What's Next?

Machine Learning An MLOps project beyond modeling in Jupyter Notebook Albers Uzila June 4, 20233 min read

Reducing memory usage in pandas with smaller datatypes

Programming Optimising pandas memory useage by the effective use of datatypes Parul Pandey March 15, 20210 min read

Create Powerful Model Explanations for Classification Problems with Logistic Regression

Programming A practitioner's guide, with a demonstration using the IBM Telco Churn dataset Jin Cui December 30, 20229 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close