{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "How to Work with Parquet Files in Python – A Guide with Examples",
  "keywords": "",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/how-to-work-with-parquet-files-in-python-a-guide-with-examples-16l6",
  "publish_date": "2026-03-05",
  "corpus_stage": "",
  "notes": "",
  "id": "101f6557c995",
  "collected_at": "2026-10-02T14:01:16",
  "title": "How to Work with Parquet Files in Python – A Guide with Examples - DEV Community",
  "body_markdown": "If you've spent time in data engineering or analytics, you've almost certainly run into Parquet files. They show up everywhere — in data lakes, machine learning pipelines, cloud storage buckets, and more.\nBut what makes Parquet special, and how do you actually work with it in Python?\nIn this tutorial, I'll walk you through reading, writing, filtering, and compressing Parquet files using Python. By the end, you'll have a solid foundation for using Parquet in your own data pipelines.\nYou can get the code on GitHub\n.\nTable of Contents\nWhat Is the Parquet File Format?\nPrerequisites\nWriting Parquet Files in Python\nReading Parquet Files in Python\nReading Specific Columns\nWriting Parquet Files with Compression\nWorking with Row Groups and Filtering\nA Practical Example: Analyzing Sales Data\nWhen Should You Use Parquet?\nWhat Is the Parquet File Format?\nParquet is an open-source\ncolumnar storage format\noriginally developed by Cloudera and Twitter, and now part of the Apache ecosystem.\nUnlike CSV or JSON, which store data row by row, Parquet stores data column by column. That might seem like a small implementation detail, but it has a big impact on performance.\nHere's why Parquet is widely used:\nEfficient compression\n— because all values in a column share the same data type, they compress far better than mixed-type rows\nColumn pruning\n— you can read only the columns you need, skipping the rest entirely\nSchema preservation\n— data types are stored in the file, so integers don't become strings on read (and other subtle bugs)\nBroad ecosystem support\n— works natively with Spark, Pandas, DuckDB, BigQuery, Athena, and more\nParquet is often compared to\nORC (or optimized row columnar)\n. Both are columnar formats, but Parquet has wider cross-platform support and tends to be the default choice outside of Hadoop/Hive environments.\nPrerequisites\nBefore we get started, make sure you have:\nPython 3.10 or a later version installed\nBasic familiarity with pandas DataFrames\nSome understanding of file I/O in Python\nYou'll need to install two libraries:\npip\ninstall\npyarrow pandas\nEnter fullscreen mode\nExit fullscreen mode\nWe'll use\nPyArrow\nas the engine for reading and writing Parquet files. You can also use\n[fastparquet](https://fastparquet.readthedocs.io/en/latest/)\nas an alternative engine, but PyArrow is the more commonly recommended choice.\nWriting Parquet Files in Python\nLet's start by creating a Parquet file. We'll build a small product inventory dataset and write it to disk.\nimport\npandas\nas\npd\nimport\npyarrow\nas\npa\nimport\npyarrow.parquet\nas\npq\n# Create a product inventory dataset\ndata\n=\n{\n'\nproduct_id\n'\n:\n[\n1001\n,\n1002\n,\n1003\n,\n1004\n,\n1005\n],\n'\nproduct_name\n'\n:\n[\n'\nWireless Mouse\n'\n,\n'\nMechanical Keyboard\n'\n,\n'\nUSB-C Hub\n'\n,\n'\nMonitor Stand\n'\n,\n'\nWebcam HD\n'\n],\n'\ncategory\n'\n:\n[\n'\nPeripherals\n'\n,\n'\nPeripherals\n'\n,\n'\nAccessories\n'\n,\n'\nAccessories\n'\n,\n'\nPeripherals\n'\n],\n'\nprice\n'\n:\n[\n29.99\n,\n89.99\n,\n49.99\n,\n34.99\n,\n74.99\n],\n'\nstock\n'\n:\n[\n150\n,\n80\n,\n200\n,\n95\n,\n60\n]\n}\ndf\n=\npd\n.\nDataFrame\n(\ndata\n)\n# Convert to PyArrow Table and write as Parquet\ntable\n=\npa\n.\nTable\n.\nfrom_pandas\n(\ndf\n)\npq\n.\nwrite_table\n(\ntable\n,\n'\ninventory.parquet\n'\n)\nprint\n(\n\"\nParquet file created successfully!\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nOutput:\nParquet file created successfully!\nEnter fullscreen mode\nExit fullscreen mode\nHere's what's happening step by step. We start with a standard pandas DataFrame. Then we convert it to a PyArrow Table using\npa.Table.from_pandas()\n. This is PyArrow's in-memory columnar representation. Finally,\npq.write_table()\nserializes it to disk in Parquet format.\nThe conversion step is necessary because Parquet is a columnar format and PyArrow handles the translation between pandas' row-oriented structure and Parquet's column-oriented storage.\nReading Parquet Files in Python\nNow let's read the file back:\n# Read the Parquet file\ntable\n=\npq\n.\nread_table\n(\n'\ninventory.parquet\n'\n)\n# Convert to pandas for easy viewing\ndf_read\n=\ntable\n.\nto_pandas\n()\nprint\n(\ndf_read\n)\nprint\n(\nf\n\"\n\\n\nData types:\n\\n\n{\ndf_read\n.\ndtypes\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nOutput:\nproduct_id         product_name     category  price  stock\n0        1001       Wireless Mouse  Peripherals  29.99    150\n1        1002  Mechanical Keyboard  Peripherals  89.99     80\n2        1003            USB-C Hub  Accessories  49.99    200\n3        1004        Monitor Stand  Accessories  34.99     95\n4        1005            Webcam HD  Peripherals  74.99     60\n\nData types:\nproduct_id        int64\nproduct_name     object\ncategory         object\nprice           float64\nstock             int64\ndtype: object\nEnter fullscreen mode\nExit fullscreen mode\npq.read_table()\nloads the Parquet file into a PyArrow Table. We then call\n.to_pandas()\nto convert it into a familiar DataFrame.\nNotice that data types are preserved correctly —\nprice\nis\nfloat64\n, not a string. This is one of Parquet's practical advantages over CSV, where you often have to explicitly cast types after reading.\nReading Specific Columns\nOne of Parquet's most powerful features is the ability to read only the columns you actually need. This is called\ncolumn pruning\n, and it can substantially reduce memory usage and load time on large files.\n# Read only product name and price\ntable_subset\n=\npq\n.\nread_table\n(\n'\ninventory.parquet\n'\n,\ncolumns\n=\n[\n'\nproduct_name\n'\n,\n'\nprice\n'\n])\ndf_subset\n=\ntable_subset\n.\nto_pandas\n()\nprint\n(\ndf_subset\n)\nEnter fullscreen mode\nExit fullscreen mode\nOutput:\nproduct_name  price\n0       Wireless Mouse  29.99\n1  Mechanical Keyboard  89.99\n2            USB-C Hub  49.99\n3        Monitor Stand  34.99\n4            Webcam HD  74.99\nEnter fullscreen mode\nExit fullscreen mode\nBy passing a\ncolumns\nlist to\nread_table()\n, PyArrow only reads those two columns from disk — the rest are skipped entirely. If your file has 100 columns but your query only needs 4, you're doing roughly 4% of the I/O work. That adds up fast at scale.\nWriting Parquet Files with Compression\nParquet supports several compression codecs. Let's compare them on a larger dataset:\nimport\nos\n# Generate a larger dataset\nlarge_data\n=\n{\n'\ntransaction_id\n'\n:\nrange\n(\n50000\n),\n'\nstore\n'\n:\n[\n'\nDowntown\n'\n,\n'\nUptown\n'\n,\n'\nSuburban\n'\n,\n'\nAirport\n'\n,\n'\nMall\n'\n]\n*\n10000\n,\n'\nproduct\n'\n:\n[\nf\n'\nproduct_\n{\ni\n%\n200\n}\n'\nfor\ni\nin\nrange\n(\n50000\n)],\n'\namount\n'\n:\n[\nround\n(\n10\n+\n(\ni\n%\n500\n)\n*\n0.5\n,\n2\n)\nfor\ni\nin\nrange\n(\n50000\n)]\n}\ndf_large\n=\npd\n.\nDataFrame\n(\nlarge_data\n)\ntable_large\n=\npa\n.\nTable\n.\nfrom_pandas\n(\ndf_large\n)\n# Write with different compression codecs\nfor\ncodec\nin\n[\n'\nNONE\n'\n,\n'\nSNAPPY\n'\n,\n'\nGZIP\n'\n,\n'\nZSTD\n'\n]:\npath\n=\nf\n'\ntransactions_\n{\ncodec\n.\nlower\n()\n}\n.parquet\n'\npq\n.\nwrite_table\n(\ntable_large\n,\npath\n,\ncompression\n=\ncodec\n)\nsize\n=\nos\n.\npath\n.\ngetsize\n(\npath\n)\nprint\n(\nf\n\"\n{\ncodec\n:\n<\n8\n}\n:\n{\nsize\n:\n>\n10\n,\n}\nbytes\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nOutput:\nNONE    :    635,565 bytes\nSNAPPY  :    316,033 bytes\nGZIP    :    177,654 bytes\nZSTD    :    158,082 bytes\nEnter fullscreen mode\nExit fullscreen mode\nEach codec makes a different trade-off between file size and speed:\nNONE\n— no compression, fastest write, largest file\nSNAPPY\n— fast compression and decompression, moderate size reduction, good for hot data\nGZIP\n— better compression than Snappy, slower, good for archival\nZSTD\n— best compression ratio with competitive speed, generally the recommended default\nFor most production use cases, ZSTD is a solid default. It gives you the smallest files without a significant speed penalty.\nWorking with Row Groups and Filtering\nParquet files are internally divided into\nrow groups\n— chunks of rows stored together. This structure enables a powerful optimization called\npredicate pushdown\n, where Parquet skips entire row groups that can't possibly match your filter condition.\nLet's write a file with explicit row group sizing and then filter it:\n# Write with smaller row groups (useful for filtering large files)\npq\n.\nwrite_table\n(\ntable_large\n,\n'\ntransactions_grouped.parquet\n'\n,\ncompression\n=\n'\nZSTD\n'\n,\nrow_group_size\n=\n10000\n)\n# Use PyArrow's dataset API for efficient filtering\nimport\npyarrow.dataset\nas\nds\ndataset\n=\nds\n.\ndataset\n(\n'\ntransactions_grouped.parquet\n'\n,\nformat\n=\n'\nparquet\n'\n)\n# Filter: only transactions from the Downtown store over $150\nfiltered\n=\ndataset\n.\nto_table\n(\nfilter\n=\n(\nds\n.\nfield\n(\n'\nstore\n'\n)\n==\n'\nDowntown\n'\n)\n&\n(\nds\n.\nfield\n(\n'\namount\n'\n)\n>\n150\n)\n)\ndf_filtered\n=\nfiltered\n.\nto_pandas\n()\nprint\n(\nf\n\"\nMatching transactions:\n{\nlen\n(\ndf_filtered\n)\n}\n\"\n)\nprint\n(\ndf_filtered\n.\nhead\n())\nEnter fullscreen mode\nExit fullscreen mode\nOutput:\nMatching transactions: 4300\n   transaction_id     store      product  amount\n0             285  Downtown   product_85   152.5\n1             290  Downtown   product_90   155.0\n2             295  Downtown   product_95   157.5\n3             300  Downtown  product_100   160.0\n4             305  Downtown  product_105   162.5\nEnter fullscreen mode\nExit fullscreen mode\nThe\nds.dataset()\nAPI with a\nfilter\nargument doesn't load the entire file into memory. Instead, it evaluates the filter against each row group's metadata and skips groups that can't contain matching rows. The actual rows are only read when you call\n.to_table()\n. This is particularly useful for large files.\nA Practical Example: Analyzing Sales Data\nLet's put it all together with a better example. Say you receive daily sales exports and need to store them efficiently and query them by region and date.\nfrom\ndatetime\nimport\ndatetime\n,\ntimedelta\nimport\nrandom\n# Simulate 30 days of sales data\nregions\n=\n[\n'\nNorth\n'\n,\n'\nSouth\n'\n,\n'\nEast\n'\n,\n'\nWest\n'\n]\nproducts\n=\n[\n'\nLaptop\n'\n,\n'\nTablet\n'\n,\n'\nHeadphones\n'\n,\n'\nCharger\n'\n,\n'\nCase\n'\n]\nrecords\n=\n[]\nstart\n=\ndatetime\n(\n2025\n,\n1\n,\n1\n)\nfor\nday\nin\nrange\n(\n30\n):\ndate\n=\nstart\n+\ntimedelta\n(\ndays\n=\nday\n)\nfor\n_\nin\nrange\n(\n200\n):\nrecords\n.\nappend\n({\n'\ndate\n'\n:\ndate\n.\nstrftime\n(\n'\n%Y-%m-%d\n'\n),\n'\nregion\n'\n:\nrandom\n.\nchoice\n(\nregions\n),\n'\nproduct\n'\n:\nrandom\n.\nchoice\n(\nproducts\n),\n'\nunits_sold\n'\n:\nrandom\n.\nrandint\n(\n1\n,\n20\n),\n'\nrevenue\n'\n:\nround\n(\nrandom\n.\nuniform\n(\n50\n,\n2000\n),\n2\n)\n})\ndf_sales\n=\npd\n.\nDataFrame\n(\nrecords\n)\ntable_sales\n=\npa\n.\nTable\n.\nfrom_pandas\n(\ndf_sales\n)\n# Save as compressed Parquet\npq\n.\nwrite_table\n(\ntable_sales\n,\n'\nsales_jan2025.parquet\n'\n,\ncompression\n=\n'\nZSTD\n'\n)\nprint\n(\nf\n\"\nSaved\n{\nlen\n(\ndf_sales\n)\n:\n,\n}\nrecords to Parquet\n\"\n)\n# Query: total revenue by region for the first two weeks\ndataset\n=\nds\n.\ndataset\n(\n'\nsales_jan2025.parquet\n'\n,\nformat\n=\n'\nparquet\n'\n)\nfirst_two_weeks\n=\ndataset\n.\nto_table\n(\ncolumns\n=\n[\n'\nregion\n'\n,\n'\nrevenue\n'\n,\n'\ndate\n'\n],\nfilter\n=\nds\n.\nfield\n(\n'\ndate\n'\n)\n<=\n'\n2025-01-14\n'\n).\nto_pandas\n()\nsummary\n=\nfirst_two_weeks\n.\ngroupby\n(\n'\nregion\n'\n)[\n'\nrevenue\n'\n].\nsum\n().\nsort_values\n(\nascending\n=\nFalse\n)\nprint\n(\nf\n\"\n\\n\nRevenue by region (Jan 1–14):\n\"\n)\nprint\n(\nsummary\n.\nround\n(\n2\n))\nEnter fullscreen mode\nExit fullscreen mode\nOutput:\nSaved 6,000 records to Parquet\n\nRevenue by region (Jan 1–14):\nregion\nWest     752919.48\nNorth    739783.22\nSouth    730002.82\nEast     692128.58\nName: revenue, dtype: float64\nEnter fullscreen mode\nExit fullscreen mode\nThis is a pattern you'll see in real data engineering work. Parquet handles the storage efficiently, column pruning limits what gets loaded into memory, and predicate pushdown ensures only the relevant date range is scanned.\nWhen Should You Use Parquet?\nUse Parquet when you:\nWork with analytical or reporting workloads where you query specific columns\nNeed efficient, compressed storage for medium-to-large datasets\nStore data in cloud platforms (S3, GCS, Azure Blob) that are billed by bytes scanned\nUse tools like Spark, DuckDB, Athena, or BigQuery that have native Parquet support\nWant schema enforcement across your pipeline\nDon't use Parquet when you:\nNeed human-readable files; use CSV or JSON for that\nAre doing frequent row-level updates or inserts; Parquet is write-once, read-many\nWork with very small datasets where the overhead isn't worth it\nNeed streaming row-by-row writes; consider Avro instead\nConclusion\nI hope you found this tutorial useful. Parquet's columnar layout, built-in compression, and schema preservation add up to faster queries, smaller files, and fewer type-related bugs.\nIn this tutorial, you learned how to write and read Parquet files, use column pruning to reduce I/O, apply compression codecs, and filter large files efficiently using predicate pushdown. These are the building blocks you'll reach for repeatedly in any data-heavy Python project.\nTry swapping your next CSV export for Parquet and see how it affects your pipeline. Happy coding!",
  "metadata": {
    "title": "How to Work with Parquet Files in Python – A Guide with Examples - DEV Community",
    "description": "If you've spent time in data engineering or analytics, you've almost certainly run into Parquet... Tagged with python.",
    "keywords": "python, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/how-to-work-with-parquet-files-in-python-a-guide-with-examples-16l6",
    "og:title": "How to Work with Parquet Files in Python – A Guide with Examples",
    "og:description": "If you've spent time in data engineering or analytics, you've almost certainly run into Parquet...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1emtcezdgflrjlj8wt38.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "How to Work with Parquet Files in Python – A Guide with Examples",
    "twitter:description": "If you've spent time in data engineering or analytics, you've almost certainly run into Parquet...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1emtcezdgflrjlj8wt38.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/101f6557c995/code/body-000.txt",
      "chars": 27,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-001.txt",
      "chars": 649,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-002.txt",
      "chars": 35,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-003.txt",
      "chars": 191,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-004.txt",
      "chars": 519,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-005.txt",
      "chars": 175,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-006.txt",
      "chars": 180,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-007.txt",
      "chars": 657,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-008.txt",
      "chars": 108,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-009.txt",
      "chars": 623,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-010.txt",
      "chars": 322,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-011.txt",
      "chars": 1265,
      "language": "text"
    },
    {
      "file": "attachments/101f6557c995/code/body-012.txt",
      "chars": 175,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/balapriyac/python-basics/tree/main/parquet",
      "label": "You can get the code on GitHub",
      "type": "a"
    },
    {
      "href": "https://www.freecodecamp.org/news/how-to-work-with-the-orc-file-format-in-python-a-guide-with-examples/",
      "label": "ORC (or optimized row columnar)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:07Z",
  "source_url": "https://dev.to/balapriya/how-to-work-with-parquet-files-in-python-a-guide-with-examples-16l6",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 12572,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/101f6557c995/pages/source.html",
  "local_html": "attachments/101f6557c995/pages/article.html",
  "body_base_path": "attachments/101f6557c995/pages/",
  "body_markdown_path": "attachments/101f6557c995/pages/article.md"
}