保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Polars + NVIDIA GPU Tutorial

Using Polars with NVIDIA GPU can speed up your data pipelines Ivo Bernardo September 17, 20244 min read Image by noaa @ Unsplash.com Image by noaa @ Unsplash.com Dealing with massive datasets inside Python has always been a challenge. The language is not tailored for handling huge amounts of data as native SQL systems or spark. The most famous library for handling 2-D datasets within Python is, without any question, pandas. Although easy to use and used by every data scientist, Pandas is written in Python and C, making it a bit combersume and slow to perform operations on large data. If you are a data scientist, you've dealt with the pain of waiting 200 years for a group by to finish. One of the libraries that aims to solve this is polars -an extremely efficient Python package that is able to handle large datasets, mostly for the following reasons:

And.. after today, you can now leverage NVIDIA's hardware to maximize polars ' GPU engine capabilities. In this blog post, we'll see how you can leverage polars+GPU and speed up your data pipelines enormously.

Learn this step by step with the interactive AI and Data Scientist and Machine Learning roadmaps.

Setting Up the Environment

To follow along this blog post, you can check the accompanying notebook here. First, let's install polars with GPU capabilities by using: perl

!pip install polars[gpu] --extra-index-url=https://pypi.nvidia.com

We need to bring out the big guns to test our data pipelines. We'll be using a large dataset of Simulated Transactions - a large dataset representing different monetary transactions of customers. As I'm assuming you are using Google Colab and we want to keep ourselves on the free tier, I'll use a sample of this dataset that only uses 20% of the size: ruby

!wget https://storage.googleapis.com/rapidsai/polars-demo/transactions-t4-20.parquet -O transactions.parquet

But, if you have your own system and resources that can handle the size of the total data, feel free to download the original file: ruby

!wget https://storage.googleapis.com/rapidsai/polars-demo/transactions.parquet -O transactions.parquet

Preview of Transactions Table - Image by Author Preview of Transactions Table - Image by Author Finally, let's load polars : javascript

import polars as pl
from polars.testing import assert_frame_equal

Setup is done, time to play around with our dataset!

Simple Calculations

We need to load our dataset into our polarsengine by leveraging the scan_parquet method: ini

transactions = pl.scan_parquet("transactions.parquet")

We can take a peak at our first 5 transactions using head and collect : scss

transactions.head(5).collect()

Transactions Table in Pandas - Image by Author Transactions Table in Pandas - Image by Author The simples use case we can think of is to perform a simple aggregation of a numerical column. Let's sum the total amount of the transactions on our table using polarssyntax: less

%%time
transactions.select(pl.col("AMOUNT").sum()).collect()

Aggregate Execution - Image by Author Aggregate Execution - Image by Author CPU time for this operation was 1.36s on the 52 million rows. We'll keep track of the execution time of our code throughout our experiments for benchmarking purposes. Let's see how this operation works on the gpu_engine : graphql

gpu_engine = pl.GPUEngine(
    device=0,
    raise_on_fail=True,
)

less

%%time
transactions.select(pl.col("AMOUNT").sum()).collect(engine=gpu_engine)

Aggregate Execution with gpu_engine - Image by Author Aggregate Execution with gpu_engine - Image by Author Nice improvement! CPU time was ~4 times faster. Will this stand for complex calculations?

Using Group By + Aggregations

One common operation we do with 2 dimensional structures is performing group by + any type of aggregation. These types of aggregations typically show up when quick answers are needed for specific business questions or during particular steps in the data wrangling process. For example, which customers are the highest spenders in our dataset? Let's calculate that using the CPU: sql

%%time

high_spenders_cpu = (
    transactions
    .group_by("CUST_ID")
    .agg(pl.col("AMOUNT").sum())
    .sort(by="AMOUNT", descending=True)
    .head()
    .collect()
)
high_spenders_cpu

The cpu_time for this operation was (drumroll): Group By Aggregation Operation - Image by Author Group By Aggregation Operation - Image by Author 4.27 seconds, not bad. But what if we do this operation on our GPU? sql

%%time

high_spenders_gpu = (
    transactions
    .group_by("CUST_ID")
    .agg(pl.col("AMOUNT").sum())
    .sort(by="AMOUNT", descending=True)
    .head()
    .collect(engine=gpu_engine)
)
high_spenders_gpu

Group By Aggregation Operation on GPU Engine— Image by Author Group By Aggregation Operation on GPU Engine— Image by Author 345 ms for an aggregation on a 52 million row dataframe. That's actually a great result! And what's nice is that we can also rely on polars SQL interface to achieve similar results: python

sql_query = """
SELECT CUST_ID, SUM(AMOUNT) as sum_amt
FROM transactions
GROUP BY CUST_ID
ORDER BY sum_amt desc
LIMIT 5
"""

%time pl.sql(sql_query).collect()
%time pl.sql(sql_query).collect(engine=gpu_engine)

Group By Aggregation Operation with Polars SQL interface— Image by Author Group By Aggregation Operation with Polars SQL interface— Image by Author

In conclusion, we were able to speed up our execution time in 5x (up to 10x) in aggregation operations by using the GPU capabilities of polars and it was awesome. Standard polars already shows extremely good results in other benchmarks, but prepping it up with GPU capabilities is totally a new game. In a nutshell:

You can also check more examples and experiments by visiting the official repo. Last, but not least, thank you to NVIDIA for giving me early access to the package and being able to experiment with this new accelerated version of polars. Looking forward to new developments! If you want to read / see more content related to AI and DS, subscribe my youtube Channel "The Data Journey ": https://www.youtube.com/@TheDataJourney42 [The dataset used in this blog post is under licence CC0: Public Domain]

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Ivo Bernardo

See all from Ivo Bernardo Data ScienceMachine LearningNvidiaPolars DataframePython Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Ensembles: the almost free Lunch in Machine Learning

Deep Learning Build optimal ensembles of neural networks with PyTorch and NumPy Dmitry Borisenko August 14, 202018 min read

Neural Graph Databases

Artificial Intelligence A new milestone in graph data management Michael Galkin March 28, 202315 min read

A Mathematical Explanation of Support Vector Machines

Artificial Intelligence Develop a deeper understanding of one of the most popular machine learning models Terence Shin, MSc, MBA February 5, 20217 min read

Full Cycle Data Science (FCDS)

Machine Learning Key to Scaling Data-Driven Organizations Daniel Marcous May 22, 202123 min read

Intro to Webhooks and How to Receive them with Python

This tutorial will be an introduction to the concept of webhooks. We will also build a simple Flask server that can receive GitHub... Rahul Banerjee October 27, 20214 min read

Bootstrap a Data Lakehouse in an Afternoon

Data Engineering Using Apache Iceberg on AWS with Athena, Glue/Spark and DuckDB Thomas Reid December 4, 202512 min read

The Polynomial That Fixed 30 Years of Cloth Simulation

Physics How a Japanese e-commerce company published one of the most rigorously engineered cloth simulation papers in recent years, why the logarithmic barrier fails under tight strain limits, and how to replicate the core insight in Python. Ferran Alia June 8, 202613 min read

P-value, Hypothesis Testing and Statistical Significance

Machine Learning How to understand if the difference really matters. Soner Yıldırım May 10, 20206 min read

Practical SQL: Create and Query a Relational Database

Artificial Intelligence Hands-on tutorial for SQL Soner Yıldırım November 28, 20206 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close