{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Wes McKinney",
  "title_listed": "From Arrow to pandas at 10 Gigabytes Per Second",
  "keywords": "Arrow,pandas,性能优化",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://wesmckinney.com/blog/high-perf-arrow-to-pandas/",
  "publish_date": "2017-01",
  "corpus_stage": "数据处理",
  "notes": "Arrow 到 pandas 高速转换",
  "id": "c4b089bd2e2a",
  "collected_at": "2026-10-02T15:15:03",
  "title": "From Arrow to pandas at 10 Gigabytes Per Second – Wes McKinney",
  "body_markdown": "From Arrow to pandas at 10 Gigabytes Per Second\napache arrow\npandas\nPublished\nDecember 27, 2016\nIn this post I discuss some recent work in Apache Arrow to accelerate converting to pandas objects from general Arrow columnar memory.\nChallenges constructing pandas DataFrame objects quickly\nOne of the difficulties in fast construction of pandas DataFrame object is that the “native” internal memory structure is more complex than a dictionary or list of one-dimensional NumPy arrays. I won’t go into the reasons for this complexity, but it’s something we’re hoping to do away with as part of the\npandas 2.0 effort\n. There are two layers of complexity:\npandas’s memory representation for a particular data type may change depending on the presence of null values. Boolean data becomes\ndtype=object\nwhile integer data becomes\ndtype=float64\n.\nUpon calling\npandas.DataFrame\n, internally pandas will “consolidate” the input by copying into its internal two-dimensional\nblock structure\n. Constructing the precise block structure is the only true way to do\nzero-copy\nDataFrame construction.\nTo give you an idea of the overhead introduced by consolidation, let’s look at a benchmark. Consider the setup code, in which we create a dict of 100\nfloat64\narrays constituting a gigabyte of data:\nimport\nnumpy\nas\nnp\nimport\npandas\nas\npd\nimport\npyarrow\nas\npa\ntype_\n=\nnp.dtype(\n'float64'\n)\nDATA_SIZE\n=\n(\n1\n<<\n30\n)\nNCOLS\n=\n100\nNROWS\n=\nDATA_SIZE\n/\nNCOLS\n/\nnp.dtype(type_).itemsize\ndata\n=\n{\n'c'\n+\nstr\n(i): np.random.randn(NROWS)\nfor\ni\nin\nrange\n(NCOLS)\n}\nThen, we create a DataFrame with\npd.DataFrame(data)\n:\n>>> %timeit df = pd.DataFrame(data)\n10 loops, best of 3: 132 ms per loop\n(For those who are counting, that’s\n7.58 GB/second\njust\nto do an internal memory copy.)\nAn important thing to remember is that, here, we have already constructed pandas’s “native” memory representation (nulls would be\nNaN\nin the arrays), but as a collection of 1D arrays.\nConverting from Arrow columnar memory to pandas\nApache Arrow\n, a project I’ve been heavily involved with since its genesis early in 2016, is a language-agnostic in-memory columnar representation and collection of tools for interprocess communication (IPC) . It supports nested JSON-like data and is designed as a building-block for creating fast analytics engines.\nCompared with pandas, Arrow has a more precise representation of null values in a bitmap that is separate from the values. So, zero-copy conversion even to a dict-of-arrays representation suitable for pandas requires more effort.\nOne of my major goals in working on Arrow is to use it as a high-bandwidth IO pipe for the Python ecosystem. We can talk to the JVM, database systems, and many different file formats by using Arrow as the columnar interchange format. For this use case, it’s important to be able to get back a pandas DataFrame as fast as possible.\nOver the last month I’ve done some engineering to construct pandas’s native internal block structure to achieve high bandwidth to Arrow memory. If you’ve been following the\nFeather file format\n, this work is all closely interrelated.\nLet’s return to the same gigabyte of data from above, and be sure to add some nulls:\n>>>\ndf\n=\npd.DataFrame(data)\n>>>\ndf.values[::\n5\n]\n=\nnp.nan\nNow, let’s convert the DataFrame to an Arrow table, which constructs the Arrow columnar representation:\n>>>\ntable\n=\npa.Table.from_pandas(df)\n>>>\ntable\n<\npyarrow.table.Table at\n0x7f18ec65abd0\n>\nTo go back to pandas-land, call the table’s\nto_pandas\nmethod. This supports a multithreaded conversion, so let’s do a single-threaded conversion for comparison:\n>>>\n%\ntimeit df2\n=\ntable.to_pandas(nthreads\n=\n1\n)\n10\nloops, best of\n3\n:\n158\nms per loop\nThis is\n6.33 GB/s\n, or about 20% slower than purely memcpy-based construction. On my desktop, I can use all 4 cores to go even faster:\n>>>\n%\ntimeit df2\n=\ntable.to_pandas(nthreads\n=\n4\n)\n10\nloops, best of\n3\n:\n103\nms per loop\nAt\n9.71 GB/s\n, this is not far from saturating the main memory bandwidth on my consumer desktop hardware (but I am not an expert on this).\nThe performance benefits of multithreading can be more dramatic on other hardware. While the performance ratio on my desktop is only\n1.53\n, on my (also quad-core) laptop it is 3.29.\nNote that numeric data is a best-case scenario; string or binary data would come with additional overhead while pandas continues to use Python objects in its memory representation.\nImplications and roadmap\nSince Arrow arrays, record batches (multiple arrays of the same length), and tables (collections of record batches) can be easily zero-copy constructed from many different sources, it is a flexible and efficient way to move tabular data around between systems. By having high speed conversion to pandas, we can pay a small conversion cost (5-10 GB/s is usually negligible compared with the IO performance of other media) to obtain a fully-fledged pandas DataFrame.\nIn a separate blog post, I’ll go into some of the technical details of Arrow’s low-overhead (and as often as possible: zero-copy) C++ IO subsystem.\nAs we forge ahead on the pandas 2.0 roadmap, we hope to further reduce the overhead (to zero, in some cases) in interacting with columnar memory like Arrow. A simpler memory representation will also make it easier for other applications to interact with pandas at a low-level.",
  "metadata": {
    "title": "From Arrow to pandas at 10 Gigabytes Per Second – Wes McKinney",
    "description": null,
    "keywords": null,
    "author": null,
    "og:title": "From Arrow to pandas at 10 Gigabytes Per Second – Wes McKinney",
    "og:site_name": "Wes McKinney",
    "og:locale": "en_US",
    "twitter:title": "From Arrow to pandas at 10 Gigabytes Per Second – Wes McKinney",
    "twitter:creator": "@wesmckinn",
    "twitter:card": "summary"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/c4b089bd2e2a/code/body-000.txt",
      "chars": 254,
      "language": "text"
    },
    {
      "file": "attachments/c4b089bd2e2a/code/body-001.txt",
      "chars": 72,
      "language": "text"
    },
    {
      "file": "attachments/c4b089bd2e2a/code/body-002.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/c4b089bd2e2a/code/body-003.txt",
      "chars": 86,
      "language": "text"
    },
    {
      "file": "attachments/c4b089bd2e2a/code/body-004.txt",
      "chars": 82,
      "language": "text"
    },
    {
      "file": "attachments/c4b089bd2e2a/code/body-005.txt",
      "chars": 82,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "http://www.slideshare.net/wesm/python-data-wrangling-preparing-for-the-future",
      "label": "pandas 2.0 effort",
      "type": "a"
    },
    {
      "href": "http://arrow.apache.org/",
      "label": "Apache Arrow",
      "type": "a"
    },
    {
      "href": "https://github.com/wesm/feather",
      "label": "Feather file format",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:09Z",
  "source_url": "https://wesmckinney.com/blog/high-perf-arrow-to-pandas/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5314,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c4b089bd2e2a/pages/source.html",
  "local_html": "attachments/c4b089bd2e2a/pages/article.html",
  "body_base_path": "attachments/c4b089bd2e2a/pages/",
  "body_markdown_path": "attachments/c4b089bd2e2a/pages/article.md"
}