{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Wes McKinney",
  "title_listed": "Streaming Columnar Data with Apache Arrow",
  "keywords": "Arrow,列式数据,流式",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://wesmckinney.com/blog/arrow-streaming-columnar/",
  "publish_date": "2017-02",
  "corpus_stage": "数据处理",
  "notes": "Arrow 流式列式数据",
  "id": "e461695e683f",
  "collected_at": "2026-10-02T15:15:04",
  "title": "Streaming Columnar Data with Apache Arrow – Wes McKinney",
  "body_markdown": "Streaming Columnar Data with Apache Arrow\napache arrow\nAuthor\nWes McKinney\nPublished\nJanuary 27, 2017\nOver the past couple weeks,\nNong Li\nand I added a streaming binary format to\nApache Arrow\n, accompanying the existing random access / IPC file format. We have implementations in Java and C++, plus Python bindings. In this post, I explain how the format works and show how you can achieve very high data throughput to pandas DataFrames.\nColumnar streaming data\nA common question I get about using Arrow is the high cost of transposing large tabular datasets from record- or row-oriented format to column-oriented format. For a multi-gigabyte dataset, transposing in memory or on disk may be prohibitive.\nFor streaming data, whether the source data is row-oriented or column-oriented memory layout, one option is to send small batches of rows, each internally having a columnar memory layout.\nIn Apache Arrow, an in-memory columnar array collection representing a chunk of a table is called a\nrecord batch\n. Multiple record batches can be collected to represent a single logical table data structure.\nIn the existing “random access” file format, we write metadata containing the table schema and block locations at the end of the file, enabling you to select any record batch or any column in the dataset very cheaply. In the streaming format, we send a series of messages: the schema followed by one or more record batches.\nThe different formats look roughly like this diagram:\nStreaming data in PyArrow: Usage\nTo show you how this works, I generate an example dataset representing a single streaming chunk:\nimport\ntime\nimport\nnumpy\nas\nnp\nimport\npandas\nas\npd\nimport\npyarrow\nas\npa\ndef\ngenerate_data(total_size, ncols):\nnrows\n=\nint\n(total_size\n/\nncols\n/\nnp.dtype(\n'float64'\n).itemsize)\nreturn\npd.DataFrame({\n'c'\n+\nstr\n(i): np.random.randn(nrows)\nfor\ni\nin\nrange\n(ncols)\n})\nNow, suppose we want to write 1 gigabyte of data composed of chunks that are 1 megabyte each, so 1024 chunks. First, let’s create 1MB DataFrame with 16 columns:\nKILOBYTE\n=\n1\n<<\n10\nMEGABYTE\n=\nKILOBYTE\n*\nKILOBYTE\nDATA_SIZE\n=\n1024\n*\nMEGABYTE\nNCOLS\n=\n16\ndf\n=\ngenerate_data(MEGABYTE, NCOLS)\nThen, I convert this to a\npyarrow.RecordBatch\n:\nbatch = pa.RecordBatch.from_pandas(df)\nNow, I create an output stream that writes to RAM and create a\nStreamWriter\n:\nsink\n=\npa.InMemoryOutputStream()\nstream_writer\n=\npa.StreamWriter(sink, batch.schema)\nThen, we write the 1024 chunks composing the 1 GB dataset:\nfor\ni\nin\nrange\n(DATA_SIZE\n//\nMEGABYTE):\nstream_writer.write_batch(batch)\nSince we wrote to RAM, we can get the entire stream as a single buffer:\nIn [\n13\n]: source\n=\nsink.get_result()\nIn [\n14\n]: source\nOut[\n14\n]:\n<\npyarrow.io.Buffer at\n0x7f2df7118f80\n>\nIn [\n15\n]: source.size\nOut[\n15\n]:\n1074750744\nSince this data is in memory, reading back Arrow record batches is a zero-copy operation. I open a\nStreamReader\n, read back the data as a\npyarrow.Table\n, and then convert to a pandas DataFrame:\nIn [\n16\n]: reader\n=\npa.StreamReader(source)\nIn [\n17\n]: table\n=\nreader.read_all()\nIn [\n18\n]: table\nOut[\n18\n]:\n<\npyarrow.table.Table at\n0x7fae8281f6f0\n>\nIn [\n19\n]: df\n=\ntable.to_pandas()\nIn [\n20\n]: df.memory_usage().\nsum\n()\nOut[\n20\n]:\n1073741904\nThis is all very nice, but you may have some questions. How fast is it? How does the stream chunk size affect the absolute performance to obtain the pandas DataFrame?\nStreaming data performance\nAs the streaming chunksize grows smaller, the cost to reconstruct a contiguous columnar pandas DataFrame increases because of cache-inefficient memory access patterns. There is also some overhead from manipulating the C++ container data structures around the arrays and their memory buffers.\nWith a 1 MB as above, on my laptop (Quad-core Xeon E3-1505M) I have:\nIn [\n20\n]:\n%\ntimeit pa.StreamReader(source).read_all().to_pandas()\n10\nloops, best of\n3\n:\n129\nms per loop\nThis is an effective throughput of\n7.75 GB/s\nto reconstruct a 1GB DataFrame from 1024 1MB chunks. What happens when we use larger and smaller chunks? Here are the results\nThe performance degrades significantly from 256K to 64K chunks. I was surprised to see that 1MB chunks were faster than 16MB ones; it would be worth a more thorough investigation to understand whether that is normal variance or something else going on.\nIn the current iteration of the format, the data is not being compressed at all, so the in-memory and on-the-wire size are about the same. Compression may be added to the format as an option in the future.\nSummary\nStreaming columnar data can be an efficient way to transmit large datasets to columnar analytics tools like pandas using small chunks. Data services using row-oriented storage can transpose and stream small data chunks that are more friendly to your CPU’s L2 and L3 caches.\nFull benchmarking code\nimport\ntime\nimport\nnumpy\nas\nnp\nimport\npandas\nas\npd\nimport\npyarrow\nas\npa\ndef\ngenerate_data(total_size, ncols):\nnrows\n=\ntotal_size\n/\nncols\n/\nnp.dtype(\n'float64'\n).itemsize\nreturn\npd.DataFrame({\n'c'\n+\nstr\n(i): np.random.randn(nrows)\nfor\ni\nin\nrange\n(ncols)\n})\nKILOBYTE\n=\n1\n<<\n10\nMEGABYTE\n=\nKILOBYTE\n*\nKILOBYTE\nDATA_SIZE\n=\n1024\n*\nMEGABYTE\nNCOLS\n=\n16\ndef\nget_timing(f, niter):\nstart\n=\ntime.clock_gettime(time.CLOCK_REALTIME)\nfor\ni\nin\nrange\n(niter):\nf()\nreturn\n(time.clock_gettime(time.CLOCK_REALTIME)\n-\nstart)\n/\nNITER\ndef\nread_as_dataframe(klass, source):\nreader\n=\nklass(source)\ntable\n=\nreader.read_all()\nreturn\ntable.to_pandas()\nNITER\n=\n5\nresults\n=\n[]\nCHUNKSIZES\n=\n[\n16\n*\nKILOBYTE,\n64\n*\nKILOBYTE,\n256\n*\nKILOBYTE, MEGABYTE,\n16\n*\nMEGABYTE]\nfor\nchunksize\nin\nCHUNKSIZES:\nnchunks\n=\nDATA_SIZE\n//\nchunksize\nbatch\n=\npa.RecordBatch.from_pandas(generate_data(chunksize, NCOLS))\nsink\n=\npa.InMemoryOutputStream()\nstream_writer\n=\npa.StreamWriter(sink, batch.schema)\nfor\ni\nin\nrange\n(nchunks):\nstream_writer.write_batch(batch)\nsource\n=\nsink.get_result()\nelapsed\n=\nget_timing(\nlambda\n: read_as_dataframe(pa.StreamReader, source), NITER)\nresult\n=\n(chunksize, elapsed)\nprint\n(result)\nresults.append(result)",
  "metadata": {
    "title": "Streaming Columnar Data with Apache Arrow – Wes McKinney",
    "description": null,
    "keywords": null,
    "author": "Wes McKinney",
    "og:title": "Streaming Columnar Data with Apache Arrow – Wes McKinney",
    "og:image": "https://wesmckinney.com/images/arrow_file_formats.png",
    "og:site_name": "Wes McKinney",
    "og:locale": "en_US",
    "og:image:alt": "Arrow file formats",
    "og:image:height": "475",
    "og:image:width": "608",
    "twitter:title": "Streaming Columnar Data with Apache Arrow – Wes McKinney",
    "twitter:image": "https://wesmckinney.com/images/arrow_file_formats.png",
    "twitter:creator": "@wesmckinn",
    "twitter:image:alt": "Arrow file formats",
    "twitter:image-height": "475",
    "twitter:image-width": "608",
    "twitter:card": "summary_large_image"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/e461695e683f/code/body-000.txt",
      "chars": 285,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-001.txt",
      "chars": 125,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-002.txt",
      "chars": 38,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-003.txt",
      "chars": 84,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-004.txt",
      "chars": 75,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-005.txt",
      "chars": 141,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-006.txt",
      "chars": 229,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-007.txt",
      "chars": 100,
      "language": "text"
    },
    {
      "file": "attachments/e461695e683f/code/body-008.txt",
      "chars": 1292,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://wesmckinney.com/images/arrow_file_formats.png",
      "bytes": 108108,
      "reused": false,
      "local_path": "attachments/e461695e683f/images/body-e74391d2a478f16126a0.png",
      "src": "https://wesmckinney.com/images/arrow_file_formats.png",
      "alt": "Arrow file formats",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://wesmckinney.com/images/arrow_streaming_benchmarks.png",
      "bytes": 12572,
      "reused": false,
      "local_path": "attachments/e461695e683f/images/body-820fbd01d666bb85173c.png",
      "src": "https://wesmckinney.com/images/arrow_streaming_benchmarks.png",
      "alt": "Arrow streaming performance",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/nongli",
      "label": "Nong Li",
      "type": "a"
    },
    {
      "href": "http://arrow.apache.org/",
      "label": "Apache Arrow",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:10Z",
  "source_url": "https://wesmckinney.com/blog/arrow-streaming-columnar/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5979,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e461695e683f/pages/source.html",
  "local_html": "attachments/e461695e683f/pages/article.html",
  "body_base_path": "attachments/e461695e683f/pages/",
  "body_markdown_path": "attachments/e461695e683f/pages/article.md"
}