{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "How to Build a Pandas DataFrame from Multiple Files",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/building-a-dataframe-from-multiple-files-1590",
  "publish_date": "2021-03-28",
  "corpus_stage": "",
  "notes": "",
  "id": "f15a1bf10e40",
  "collected_at": "2026-10-02T14:01:47",
  "title": "How to Build a Pandas DataFrame from Multiple Files - DEV Community",
  "body_markdown": "In any Data Science project, the steps of\nImporting Data\nfollowed by\nData Cleaning\nand\nExploratory Data Analysis(EDA)\nare extremely important.\nLet us say we have the required dataset in a CSV file, but the dataset is stored across multiple files, instead of a single file. We would ideally like to read in the data from multiple files into a single pandas DataFrame for use in subsequent steps.\nThe most straightforward way to do it is to read in the data from each of those files into separate DataFrames and then concatenate them suitably into a single large DataFrame. This can be memory inefficient and involves writing redundant code. So, is there a better way to do it? Yes, there is. In this blog post, let us try to understand through simple examples.\nLet us import pandas under its usual alias pd.\nimport\npandas\nas\npd\nEnter fullscreen mode\nExit fullscreen mode\nBuilding a DataFrame from multiple files (row-wise)\nWe shall use a sample dataset for our example; let us read the data from\nhttp://bit.ly/smallstocks\ninto a DataFrame stocks using the\nread_csv()\nmethod of pandas.\nstocks\n=\npd\n.\nread_csv\n(\n'\nhttp://bit.ly/smallstocks\n'\n,\nparse_dates\n=\n[\n'\nDate\n'\n])\nprint\n(\nstocks\n)\nEnter fullscreen mode\nExit fullscreen mode\nLet us say, this data was stored in 3 separate CSV files, one for each day, named\nstocks1.csv\n,\nstocks2.csv\nand\nstocks3.csv\nas shown below.\nstocks1\n=\npd\n.\nread_csv\n(\n'\ndata/stocks1.csv\n'\n)\nstocks2\n=\npd\n.\nread_csv\n(\n'\ndata/stocks2.csv\n'\n)\nstocks3\n=\npd\n.\nread_csv\n(\n'\ndata/stocks3.csv\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\nAs said earlier, a naive approach would be to read in each of these CSV files into separate DataFrames, as shown above, and then concatenate them, but this would become cumbersome as the number of such files increases. A better solution is to use the built-in\nglob\nmodule. Let us import\nglob\n.\nimport\nglob\nEnter fullscreen mode\nExit fullscreen mode\nWe can pass in a pattern to\nglob()\n, including wildcard characters, and it will return a list of all files that match that pattern.\nstock_files\n=\nsorted\n(\nglob\n(\n'\ndata/stocks*.csv\n'\n))\nEnter fullscreen mode\nExit fullscreen mode\nIn this case, glob is looking in the “\ndata\n” subdirectory for all CSV files that\nstart with\nthe word “\nstocks\n”.\nglob\nreturns filenames in an arbitrary order, which is why we have sorted the list using Python’s built-in\nsorted()\nmethod.\nNow that we’ve collected all the files over which our dataset is spread across, we can use a generator expression to read in each of the files using\nread_csv()\nand pass the results to the\nconcat()\nfunction, which will concatenate the rows into a single DataFrame.\npd\n.\nconcat\n((\npd\n.\nread_csv\n(\nfile\n)\nfor\nfile\nin\nstock_files\n))\nEnter fullscreen mode\nExit fullscreen mode\nLooks like we’ve successfully accomplished bringing in all data from the three files into a single DataFrame, but, there are duplicate values in the index. To avoid that, we can set the\nignore_index\nargument to\nTrue\nto tell the\nconcat()\nfunction to ignore the index and use the default integer index instead.\npd\n.\nconcat\n((\npd\n.\nread_csv\n(\nfile\n)\nfor\nfile\nin\nstock_files\n),\nignore_index\n=\nTrue\n)\nEnter fullscreen mode\nExit fullscreen mode\nThis method is useful when each file contains rows from our dataset. What if each file instead contains columns from our dataset?\nBuilding a DataFrame from multiple files (column-wise)\nHere’s an example in which the drinks dataset has been split into two CSV files, and each of the files\ndrinks\n=\npd\n.\nread_csv\n(\n'\nhttp://bit.ly/drinksbycountry\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\ndrinks1\n=\npd\n.\nread_csv\n(\n'\ndata/drinks1.csv\n'\n).\nhead\n()\ndrinks2\n=\npd\n.\nread_csv\n(\n'\ndata/drinks2.csv\n'\n).\nhead\n()\nEnter fullscreen mode\nExit fullscreen mode\nSimilar to the procedure we followed earlier, we’ll start by using glob()\ndrink_files\n=\nsorted\n(\nglob\n(\n'\ndata/drinks*.csv\n'\n))\nEnter fullscreen mode\nExit fullscreen mode\nAnd this time, we’ll tell the\nconcat()\nfunction to concatenate along with the columns by specifying the axis argument as ‘columns’.\npd\n.\nconcat\n((\npd\n.\nread_csv\n(\nfile\n)\nfor\nfile\nin\ndrink_files\n),\naxis\n=\n'\ncolumns\n'\n).\nhead\n()\nEnter fullscreen mode\nExit fullscreen mode\nWe see that we’ve obtained a single DataFrame with all six columns. We had considered simple examples to illustrate the use. In practice, where we have datasets chunked across multiple files, this could be a lot more helpful.\nThank you for reading!😊\nReferences\n[1]\nhttps://docs.python.org/3/library/glob.html\n[2]\nhttps://github.com/justmarkham/pandas-videos/blob/master/top_25_pandas_tricks.ipynb\n[3]\nhttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html\nCover image: Photo by\nMarkus Spiske\non\nUnsplash",
  "metadata": {
    "title": "How to Build a Pandas DataFrame from Multiple Files - DEV Community",
    "description": "In any Data Science project, the steps of Importing Data followed by Data Cleaning and Exploratory... Tagged with tutorial, python, beginners, datascience.",
    "keywords": "tutorial, python, beginners, datascience, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/building-a-dataframe-from-multiple-files-1590",
    "og:title": "How to Build a Pandas DataFrame from Multiple Files",
    "og:description": "In any Data Science project, the steps of Importing Data followed by Data Cleaning and Exploratory...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwniyocmkjlydzy53xinj.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "How to Build a Pandas DataFrame from Multiple Files",
    "twitter:description": "In any Data Science project, the steps of Importing Data followed by Data Cleaning and Exploratory...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwniyocmkjlydzy53xinj.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/f15a1bf10e40/code/body-000.txt",
      "chars": 20,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-001.txt",
      "chars": 87,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-002.txt",
      "chars": 126,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-003.txt",
      "chars": 12,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-004.txt",
      "chars": 47,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-005.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-006.txt",
      "chars": 74,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-007.txt",
      "chars": 54,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-008.txt",
      "chars": 98,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-009.txt",
      "chars": 47,
      "language": "text"
    },
    {
      "file": "attachments/f15a1bf10e40/code/body-010.txt",
      "chars": 78,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Feq4yjz7px2i3k45crx4q.jpg",
      "bytes": 74450,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/002.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Feq4yjz7px2i3k45crx4q.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fismapameci2ci9z7rx0x.jpg",
      "bytes": 30400,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/003.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fismapameci2ci9z7rx0x.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2u83laea3rgywl8i7l35.jpg",
      "bytes": 30309,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/004.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2u83laea3rgywl8i7l35.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvwpuc65j5hdsohstlxx1.jpg",
      "bytes": 30101,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/005.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvwpuc65j5hdsohstlxx1.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fqrj0zuaxctfaqx7kls7v.jpg",
      "bytes": 75176,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/006.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fqrj0zuaxctfaqx7kls7v.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fujfbyl5getdseryhs4a8.jpg",
      "bytes": 74450,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/007.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fujfbyl5getdseryhs4a8.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs2bbsc1d0kq3mb57m7bg.jpg",
      "bytes": 30884,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/008.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs2bbsc1d0kq3mb57m7bg.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fozai9u9rqan603e2x4d1.jpg",
      "bytes": 49806,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/009.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fozai9u9rqan603e2x4d1.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fei4jog6d6ugrj30xno0t.jpg",
      "bytes": 45190,
      "reused": true,
      "local_path": "attachments/f15a1bf10e40/images/010.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fei4jog6d6ugrj30xno0t.jpg",
      "alt": "Alt Text",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "http://bit.ly/smallstocks",
      "label": "http://bit.ly/smallstocks",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Feq4yjz7px2i3k45crx4q.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fismapameci2ci9z7rx0x.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2u83laea3rgywl8i7l35.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvwpuc65j5hdsohstlxx1.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fqrj0zuaxctfaqx7kls7v.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fujfbyl5getdseryhs4a8.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs2bbsc1d0kq3mb57m7bg.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fozai9u9rqan603e2x4d1.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fei4jog6d6ugrj30xno0t.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://docs.python.org/3/library/glob.html",
      "label": "https://docs.python.org/3/library/glob.html",
      "type": "a"
    },
    {
      "href": "https://github.com/justmarkham/pandas-videos/blob/master/top_25_pandas_tricks.ipynb",
      "label": "https://github.com/justmarkham/pandas-videos/blob/master/top_25_pandas_tricks.ipynb",
      "type": "a"
    },
    {
      "href": "https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html",
      "label": "https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@markusspiske?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Markus Spiske",
      "type": "a"
    },
    {
      "href": "https://dev.to/s/photos/build?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:13Z",
  "source_url": "https://dev.to/balapriya/building-a-dataframe-from-multiple-files-1590",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4706,
  "expected_body_images": 9,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/f15a1bf10e40/pages/source.html",
  "local_html": "attachments/f15a1bf10e40/pages/article.html",
  "body_base_path": "attachments/f15a1bf10e40/pages/",
  "body_markdown_path": "attachments/f15a1bf10e40/pages/article.md"
}