{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Ibrahim Salami",
  "title_listed": "I Cleaned a Messy CSV File Using Pandas. Here's the Exact Process I Follow Every Time.",
  "keywords": "",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/i-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time/",
  "publish_date": "2025-11-26",
  "corpus_stage": "",
  "notes": "",
  "id": "96296f36702e",
  "collected_at": "2026-10-02T12:07:20",
  "title": "I Cleaned a Messy CSV File Using Pandas .  Here’s the Exact Process I Follow Every Time. | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# I Cleaned a Messy CSV File Using Pandas . Here’s the Exact Process I Follow Every Time.\nStop guessing at data cleaning. Use this repeatable 5-step Python workflow to diagnose and fix the most common data flaws.\n[Ibrahim Salami](https://towardsdatascience.com/author/ibrahim.salami/)\nNovember 26, 202515 min read\n![](../images/000.jpg)\nImage by Author\nOften, data isn't perfect. You’re going to encounter a lot of data inconsistencies. Nulls, negative values, string inconsistencies, etc. If these aren’t handled early in your data analysis workflow, querying and analysing your data would be a pain later on.\nNow, I’ve done data cleaning before using SQL and Excel, not really with Python. So, to learn about Pandas (one of Python’s data analysis libraries), I’ll be dabbling in some data cleaning. \nIn this article, I’ll be sharing with you a repeatable, beginner-friendly data cleaning workflow. By the end of this article, you should be pretty confident in using Python for data cleaning and analysis.\n## The Dataset we’ll be working with\nI’ll be working with a synthetic, messy HR dataset containing typical real-world errors (inconsistent dates, mixed data types, compound columns). This dataset is from Kaggle, and it is designed for practising data cleaning, transformation, exploratory analysis, and preprocessing for data visualisation and machine learning.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) roadmap.\nThe dataset contains over 1,000 rows and 13 columns, including employee information such as names, department-region combinations, contact details, status, salaries, and performance scores. It includes examples of:\n  * Duplicates\n  * Missing values\n  * Inconsistent date formats\n  * Erroneous entries (e.g., non-numeric salary values)\n  * Compound columns (e.g., “Department_Region” like “Cloud Tech-Texas” that can be split)\n\n\nIt contains columns like:\n  * Employee_ID: Unique synthetic ID (e.g., EMP1001)\n  * First_Name, Last_Name: Randomly generated personal names\n  * Name: Full name (may be redundant with first/last)\n  * Age: Includes missing values\n  * Department_Region: Compound column (e.g., “HR-Florida”)\n  * Status: Employee status (Active, Inactive, Pending)\n  * Join_Date: Inconsistent format (YYYY/MM/DD)\n  * Salary: Includes invalid entries (e.g., “N/A”)\n  * Email, Phone: Synthetic contact information\n  * Performance_Score: Categorical performance rating\n  * Remote_Work: Boolean flag (True/False)\n\n\nYou can access the dataset [here](https://www.kaggle.com/datasets/desolution01/messy-employee-dataset) and play around with it\nThe dataset is fully synthetic. It does not contain any real individuals’ data and is safe to use for public, academic, or commercial projects.\nThis dataset is in the public domain under the CC0 1.0 Universal license. You are free to use, modify, and distribute it without restriction.\n## Overview of the Cleaning Workflow\nThe data cleaning workflow I’ll be working with consists of 5 simple stages.\n  1. Load\n  2. Inspect\n  3. Clean\n  4. Review\n  5. Export  \n\n\n\nLet’s dive deeper into each of these stages.\n## Step 1 — Load the CSV (And Handle the First Hidden Issues)\nThere are some things to keep in mind before loading your dataset. However, this is an optional step, and we probably wouldn’t encounter most of these issues in our dataset. But it doesn’t hurt to know these things. Here are some key things to consider while loading.\n### Encoding issues (`utf-8`, `latin-1`)\nEncoding defines how characters are stored as bytes in the file. Python and Pandas usually default to UTF-8, which handles most modern text and special characters globally. However, if the file was created in an older system or a non-English environment, it might use a different encoding, most commonly Latin-1\nSo if you try to read a Latin-1 file with UTF-8, Pandas will encounter bytes it doesn’t recognise as valid UTF-8 sequences. You’ll typically see a UnicodeDecodeError when you try to read a CSV with encoding issues.\nIf perhaps the default load fails, you could try to specify a different encoding:\npython\n\n```\n# First attempt (the default)\ntry:\ndf = pd.read_csv(‘messy_data.csv’)\nexcept UnicodeDecodeError:\n# Second attempt with a common alternative\ndf = pd.read_csv(‘messy_data.csv’, encoding=’latin-1')\n```\n\n### Wrong delimiters\nCSV stands for “Comma Separated Values,” but in reality, many files use other characters as separators, like semicolons (common in Europe), tabs, or even pipes (|). Pandas typically defaults to the comma (,).\nSo, if your file uses a semicolon (;) but you load it with the default comma delimiter, Pandas will treat the entire row as a single column. The result would be a DataFrame with a single column containing entire lines of data, making it impossible to work with.\nThe fix is pretty simple. You can try checking the raw file (opening it in a text editor like VS Code or Notepad++ is best) to see what character separates the values. Then, pass that character to the sep argument like so\npython\n\n```\n# If the file uses semicolons\ndf = pd.read_csv('messy_data.csv', sep=';')\n\n# If the file uses tabs (TSV)\ndf = pd.read_csv('messy_data.csv', sep='\\t')\n```\n\n### Columns that import incorrectly\nSometimes, Pandas guesses the data type for a column based on the first few rows, but later rows contain unexpected data (e.g., text mixed into a column that started with numbers).\nFor instance, Pandas may correctly identify 0.1, 0.2, 0.3 as floats, but if row 100 contains the value N/A, Pandas might force the entire column into an object (string) type to accommodate the mixed values. This sucks because you lose the ability to perform fast, vectorised numeric operations on that column until you clean up the bad values.\nTo fix this, I use the dtype argument to tell Pandas what data type a column should be explicitly. This prevents silent type casting.\npython\n\n```\ndf = pd.read_csv(‘messy_data.csv’, dtype={‘price’: float, ‘quantity’: ‘Int64’})\n```\n\n### Reading the first few rows\nYou could save time by checking the first few rows directly during the loading process using the `nrows `parameter. This is great, especially when you’re working with large datasets, as it allows you to test encoding and delimiters without loading the entire 10 GB file.\npython\n\n```\n# Load only the first 50 rows to confirm encoding and delimiter\ntemp_df = pd.read_csv('large_messy_data.csv', nrows=50)\nprint(temp_df.head())\n```\n\nOnce you’ve confirmed the arguments are correct, you can load the full file.\nLet’s load the Employee dataset. I don’t expect to see any issues here.\npython\n\n```\nimport pandas as pd\ndf = pd.read_csv(‘Messy_Employee_dataset.csv’)\ndf\n```\n\n**Output** :\nmarkup\n\n```\n1020 rows × 12 columns\n```\n\nNow we can move on to **Step 2 : Inspection**\n## Step 2 — Inspect the Dataset\nI treat this phase like a forensic audit. I’m looking for evidence of chaos hidden beneath the surface. If I rush this step, I guarantee myself a world of pain and analytical errors down the line. I always run these four crucial checks before writing any transformation code.\nThe following methods give me the full health report on my 1,020 employee records:\n###  `1. df.head()` and `df.tail()`: Understanding the Boundaries\nI always start with a visual check. I use `df.head()` and `df.tail()` to see the first and last five rows. This is my quick sanity check to see if all columns look aligned and if the data visually makes sense.\n**My Finding:**\nWhen I ran `df.head()`, I noticed my `Employee ID` was sitting in a column, and the DataFrame was using the default numerical index **(0, 1, 2, ...)** instead. \nWhile I know I could set `Employee ID` as the index, for now, I’ll leave it. The bigger immediate visual risk I’m looking for here is data misaligned in the wrong column or obvious leading/trailing spaces on names that will cause trouble later.\n###  `2. df.info()`: Spotting Datatype Problems and Missingness\nThis is the most critical method. It tells me the column names, the data types (`Dtype`), and the exact number of non-null values.\n**My Findings on 1,020 Rows:**\n  * **Missing Age:** My total entry count is **1,020** , but the `Age` column only has **809** non-null values. That’s a significant amount of missing data that I'll have to decide how to handle later—do I impute it, or do I drop the rows?\n  * **Missing Salary:** The `Salary` column has **996** non-null values, which is only a minor gap, but still something I must resolve.\n  * **The ID Type Check:** The `Employee ID` is correctly listed as an `object` (string). This isn't right. IDs are identifiers, not numbers to be averaged, and using the string type prevents Pandas from accidentally stripping leading zeros.\n\n\n### 3. Data Integrity Check: Duplicates and Unique Counts\nAfter checking dtypes, I need to know if I have duplicate records and how consistent my categorical data is.\n  * **Checking for Duplicates:** I ran `df.duplicated().sum()` and got a result of **0**. This is perfect! It means I don't have identical rows cluttering up my dataset.\n  * **Checking Unique Values (**`**df.nunique()**`**):** I use this to understand the diversity within each column. Low counts in categorical columns are fine, but I look for columns that should be unique but aren't, or columns that have _too many_ unique values, suggesting typos.\n  * **Employee_ID** have 1020 unique records. This is perfect. It means all records are unique.\n  * **The First_Name / Last_Name field has eight unique records.** _That's a little odd._ This confirms the dataset's synthetic nature. My analysis won't be skewed by a large variety of names, since I'll treat them as standard strings.\n  * **Department_Region** has 36 unique records. There’s high potential for typos _._ 36 unique values for region/department is too many. I will need to check this column for spelling variations (e.g., “HR” vs. “Human Resources”) in the next step.\n  * **Email (** 64 unique records). With 1,020 employees, having only 64 unique emails suggests many employees share the same placeholder email. I will flag this for exclusion from analysis, as it’s useless for identifying individuals.\n  * **Phone (** 1020 unique records). This is perfect because it confirms phone numbers are unique identifiers.\n  * **Age / Performance Score / Status / Remote Work**(2–4 unique records). These low counts are expected for categorical or ordinal data, meaning they are ready for encoding.\n\n\n###  `4. df.describe()`: Catching Odd and Impossible Values\nI use `df.describe()` to get a statistical summary of all my numerical columns. This is the place where truly impossible values—the \"red flags\"—show up instantly. I mostly focus on the `min` and `max` rows.\n**My Findings:**\nI immediately noticed a problem in what I _expected_ to be the `Phone Number` column, which Pandas mistakenly converted to a numerical type.\nmarkup\n\n```\nMean-4.942253 * 10⁹Min-9.994973 * 10⁹Max-3.896086 * 10⁶25%-7.341992e * 10⁹50%4.943997 * 10⁹75%-2.520391e * 10⁹\n```\n\nIt turns out all the phone number values were massive negative numbers! This confirms two things:\nPandas incorrectly inferred this column as a number, even though phone numbers are strings.\nThere must be characters in the text that Pandas cannot interpret (for example, parentheses, dashes, or country codes). I need to convert this to an object type and clean it up completely.\n### 5. `df.isnull().sum()`: Quantifying Missing Data\nWhile `df.info()` gives me non-null counts, `df.isnull().sum()` gives me the total count of nulls, which is a cleaner way to quantify my next steps.\n**My Findings:**\n  * `Age` has**211 nulls** (1020 - 809 = 211), and\n  * `Salary` has **24** nulls (1020 - 996 = 24). This precise count sets the stage for **Step 3.**\n\n\nThis inspection process is my safety net. If I had missed the **negative phone numbers** , any analytical step that involved numerical data would have failed or, worse, produced skewed results without warning. \nBy identifying the need to treat `Phone Number` as a string and the significant missing values in `Age` _now_ , I have a concrete cleaning list. This prevents runtime errors and, critically, ensures that my final analysis is based on plausible, non-corrupted data.\n## Step 3 — Standardise Column Names, Correct Dtypes, and Handle Missing Values\nWith my list of flaws in hand (missing Age, missing Salary, the terrible negative Phone Numbers, and the messy categorical data), I move into the heavy lifting. I treat this step in three sub-phases: ensuring consistency, fixing corruption, and filling gaps.\n### 1. Standardising Column Names and Setting the Index (The Consistency Rule)\nBefore I do any serious data manipulation, I enforce strict consistency on column names. Why? Because typing `df['Employee ID ']` accidentally instead of `df['employee_id']` is a silent, frustrating error. Once the names are clean, I set the index.\nMy golden rule is **snake_case and lowercase everywhere, and ID columns should be the index.**\nI use a simple command to strip whitespace, replace spaces with underscores, and convert everything to lowercase.\npython\n\n```\n# The Standardization Command\ndf.columns = df.columns.str.lower().str.replace(' ', '_').str.strip()\n# Before: ['Employee_ID', 'First_Name', 'Phone']\n# After: ['employee_id', 'first_name', 'phone']\n```\n\nNow that our columns are standardised. I can move on to set employee_id as an index.\npython\n\n```\n# Set the Employee ID as the DataFrame Index\n# This is crucial for efficient lookups and clean merges later.\ndf.set_index('employee_id', inplace=True)\n\n# Let’s review it real quick\nprint(df.index)\n```\n\n**Output:**\nmarkup\n\n```\nIndex(['EMP1000', 'EMP1001', 'EMP1002', 'EMP1003', 'EMP1004', 'EMP1005','EMP1006', 'EMP1007', 'EMP1008', 'EMP1009',...'EMP2010', 'EMP2011', 'EMP2012', 'EMP2013', 'EMP2014', 'EMP2015','EMP2016', 'EMP2017', 'EMP2018', 'EMP2019'],dtype='object', name='employee_id', length=1020)\n```\n\nPerfect, everything is in place.\n### 2. Fixing Data Types and Corruption (Tackling the Negative Phone Numbers)\nMy `df.describe()` check revealed the most urgent structural flaw: the `Phone` column, which was imported as a garbage numerical type. Since phone numbers are _identifiers_ (not quantities), they must be strings.\n  \nIn this phase, I'll convert the entire column to a string type, which will turn all those negative scientific notation numbers into human-readable text (though still full of non-digit characters). I will leave the actual text cleaning (removing parentheses, dashes, etc.) for a dedicated standardisation step (Step 4).\npython\n\n```\n# Fix the Phone dtype immediately\n# Note: The column name is now 'phone' due to standardization in 3.1\ndf['phone'] = df['phone'].astype(str)\n```\n\n### 3. Handling Missing Values (The Age & Salary Gaps)\nFinally, I address the gaps revealed by `df.info()`: the **211** missing `Age` values and the **24** missing `Salary` values (out of **1,020 total rows**). My strategy depends entirely on the column's role and the magnitude of the missing data:\n  * **Salary (24 missing values)** : In this case, removing or dropping all missing values would be the best strategy. Salary is a critical metric for financial analysis. Imputing it risks skewing conclusions. Since only a small fraction (2.3%) is missing, I choose to drop the incomplete records.\n  * **Age (211 missing values)**. Filling the missing values is the best strategy here. Age is often a feature for predictive modelling (e.g., turnover). Dropping 20% of my data is too costly. I’ll fill the missing values using the **median age** to avoid skewing the distribution with the mean.\n\n\nI execute this strategy with two separate commands:\npython\n\n```\n# 1. Removal: Drop rows missing the critical 'salary' data\ndf.dropna(subset=['salary'], inplace=True)\n\n# 2. Imputation: Fill missing 'age' with the median\nmedian_age = df['age'].median()\ndf['age'].fillna(median_age, inplace=True)\n```\n\nAfter these commands, I would run `df.info()` or `isnull().sum()` again just to confirm that the non-null counts for `salary` and `age` now reflect a clean dataset.\npython\n\n```\n# Rechecking the null counts for salary and age\ndf[‘salary’].isnull().sum())\ndf[‘age’].isnull().sum())\n```\n\n**Output:**\nmarkup\n\n```\nnp.int64(0)\n```\n\nSo far so good!\nBy addressing the structural and missing data issues here, the subsequent steps can focus entirely on value standardisation, such as the messy **36 unique values in**`**department_region**`—which we tackle in the next phase.\n## Step 4 — Value Standardization: Making Data Consistent\nMy DataFrame now has the right structure, but the values inside are still dirty. This step is about consistency. If “IT,” “i.t,” and “Info. Tech” all mean the same department, I need to force them into a single, clean value (“IT”). This prevents errors in grouping, filtering, and any statistical analysis based on categories.\n### 1. Cleaning Corrupted String Data (The Phone Number Fix)\nRemember the corrupted `phone` column from Step 2? It's currently a mess of negative scientific notation numbers that we converted to strings in Step 3. Now, it's time to extract the actual digits.\nSo, I’ll be removing every non-digit character (dashes, parentheses, dots, etc.) and converting the result into a clean, unified format. Regular expressions (`.str.replace()`) are perfect for this. I use `\\D` to match any non-digit character and replace it with an empty string.\npython\n\n```\n# The phone column is currently a string like '-9.994973e+09'\n# We use regex to remove everything that isn't a digit\ndf['phone'] = df['phone'].str.replace(r'\\D', '', regex=True)\n\n# We can also truncate or format the resulting string if needed\n# For example, keeping only the last 10 digits:\ndf['phone'] = df['phone'].str.slice(-10)\nprint(df['phone'])\n```\n\n**Output:**\nmarkup\n\n```\nemployee_idEMP1000 1651623197EMP1001 1898471390EMP1002 5596363211EMP1003 3476490784EMP1004 1586734256...EMP2014 2470739200EMP2016 2508261122EMP2017 1261632487EMP2018 8995729892EMP2019 7629745492Name: phone, Length: 996, dtype: object\n```\n\nLooks much better now. This is always a good practice to clean identifiers that contain noise (like IDs with leading characters or zip codes with extensions).\n### 2. Separating and Standardizing Categorical Data (Fixing the 36 Regions)\nMy `df.nunique()` check revealed **36 unique values** in the `department_region` column. When I reviewed all the unique values in the column, the output revealed that they are all neatly structured as `department-region` (e.g., devops-california, finance-texas, cloud tech-new york).\nI guess one way to solve this is to split this single column into two dedicated columns. I'll split the column at the hyphen (`-`) and assign the parts to new columns: `department` and `region`.\npython\n\n```\n# 1. Split the combined column into two new, clean columns\ndf[['department', 'region']] = df['department_region'].str.split('-', expand=True)\nNext, I’ll drop the department_region column since it’s pretty much useless now\n# 2. Drop the redundant combined column\ndf.drop('department_region', axis=1, inplace=True)\nLet’s review our new columns\nprint(df[[‘department’, ‘region’]])\n```\n\n**Output:**\nmarkup\n\n```\ndepartment regionemployee_idEMP1000 devops californiaEMP1001 finance texasEMP1002 admin nevadaEMP1003 admin nevadaEMP1004 cloud tech florida... ... ...EMP2014 finance nevadaEMP2016 cloud tech texasEMP2017 finance new yorkEMP2018 hr floridaEMP2019 devops illinois[996 rows x 2 columns]\n```\n\nAfter splitting, the new `department` column has only **6 unique values** (e.g., 'devops', 'finance', 'admin', etc.). This is great news. The values are already standardised and ready for analysis! I guess we could always map all similar departments to one single category. But I’m gonna skip that. I don’t want to get too advanced in this article.\n### 3. Converting Date Columns (The Join_Date Fix)\nThe `Join_Date` column is usually read in as a string (`object`) type, which makes time-series analysis impossible. This means we have to convert it to a proper Pandas `datetime` object.\n`pd.to_datetime()` is the core function. I often use `errors='coerce'` as a safety net; if Pandas can't parse a date, it converts that value to `NaT` (Not a Time), which is a clean null value, preventing the whole operation from crashing.\npython\n\n```\n# Convert the join_date column to datetime objects\ndf['join_date'] = pd.to_datetime(df['join_date'], errors='coerce')\n```\n\nThe conversion of dates enables powerful time-series analysis, like calculating average employee tenure or identifying turnover rates by year.\nAfter this step, every value in the dataset is clean, uniform, and correctly formatted. The categorical columns (like `department` and `region`) are ready for grouping and visualisation, and the numerical columns (like `salary` and `age`) are ready for statistical modeling. The dataset is officially ready for analysis.\n## Step 5 — Final Quality Check and Export\nBefore closing the notebook, I always perform one last audit to ensure everything is perfect, and then I export the data so I can perform analysis on it later.\n### The Final Data Quality Check\nThis is quick. I re-run the two most critical inspection methods to confirm that all my cleaning commands actually worked:\n  * `df.info()`**:** I confirm there are **no more missing values** in the critical columns (`age`, `salary`) and that the data types are correct (`phone` is a string, `join_date` is datetime).\n  * `df.describe()`**:** I ensure the statistical summary shows plausible numbers. The `Phone` column should now be absent from this output (since it's a string), and `Age` and `Salary` should have logical minimum and maximum values.\n\n\nIf these checks pass, I know the data is reliable.\n### Exporting the Clean Dataset\nThe final step is to save this cleaned version of the data. I usually save it as a new CSV file to keep the original messy file intact for reference. I use `index=False` here if I don't want the `employee_id` (which is now the index) to be saved as a separate column, or `index=True` if I want to save the index as the first column in the new CSV.\npython\n\n```\n# Exporting the clean DataFrame to a new CSV file\n# We use index=True to keep our primary key (employee_id) in the exported file\ndf.to_csv('cleaned_employee_data.csv', index=True)\n```\n\nBy exporting with a clear, new filename (e.g., `_clean.csv`), you officially mark the end of the cleaning phase and provide a clean slate for the next phase of the project.\n## Conclusion\nHonestly, I used to feel overwhelmed by a messy dataset. The missing values, the weird data types, the cryptic columns — it felt like facing the blank page syndrome.\nBut this structured, repeatable workflow changed everything. By focusing on **Load, Inspect, Clean, Review, and Export** , we established order instantly: standardizing column names, making the `employee_id` the index, and using smart strategies for imputation and splitting messy columns.\nNow, I can jump straight into the fun analysis part without constantly second-guessing my results. If you struggle with the initial data cleaning step, try out this workflow. I’d love to hear how it goes. If you want to play around with the dataset, you can download it [here](https://www.kaggle.com/datasets/desolution01/messy-employee-dataset).\nWanna connect? Feel free to say hi on these platforms\n[LinkedIn](https://www.linkedin.com/in/ibrahim-salami-059863228/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BxTBPgk2fTXSRGVGhY8QDdQ%3D%3D)\n[Twitter](https://x.com/IbbySalam)\n[YouTube](https://www.youtube.com/@ibbysalam)\n[Medium](https://medium.com/@ibbysalam)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-542334df68a32cd56c4a.jpeg)\nWritten By\n## [Ibrahim Salami](https://towardsdatascience.com/author/ibrahim.salami/)\n[See all from Ibrahim Salami](https://towardsdatascience.com/author/ibrahim.salami/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Data Analysis](https://towardsdatascience.com/tag/data-analysis/)[Data Cleaning](https://towardsdatascience.com/tag/data-cleaning/)[Data Visualization](https://towardsdatascience.com/tag/data-visualization/)[Pandas](https://towardsdatascience.com/tag/pandas/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F&title=I%20Cleaned%20a%20Messy%20CSV%20File%20Using%20Pandas%E2%80%8A.%20%E2%80%8AHere%E2%80%99s%20the%20Exact%20Process%20I%20Follow%20Every%C2%A0Time.)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F&title=I%20Cleaned%20a%20Messy%20CSV%20File%20Using%20Pandas%E2%80%8A.%20%E2%80%8AHere%E2%80%99s%20the%20Exact%20Process%20I%20Follow%20Every%C2%A0Time.)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F&text=I%20Cleaned%20a%20Messy%20CSV%20File%20Using%20Pandas%E2%80%8A.%20%E2%80%8AHere%E2%80%99s%20the%20Exact%20Process%20I%20Follow%20Every%C2%A0Time.)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/3-methods-to-reduce-overfitting-of-machine-learning-models-4a7e2c1da9ef/)\n## [3 Methods to Reduce Overfitting of Machine Learning Models](https://towardsdatascience.com/3-methods-to-reduce-overfitting-of-machine-learning-models-4a7e2c1da9ef/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn overfit model is full of surprises but not the good ones. I have come...\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJune 2, 20215 min read\n  * [![](../images/002.jpeg)](https://towardsdatascience.com/why-you-should-know-big-data-3c0c161b9e14/)\n## [Why you should know Big Data](https://towardsdatascience.com/why-you-should-know-big-data-3c0c161b9e14/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nDefining Big Data and its potential threats\n[Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\nFebruary 5, 20229 min read\n  * [![](../images/003.jpg)](https://towardsdatascience.com/explore-pandas-dataframe-with-datatile-59c425b55d01/)\n## [Explore Pandas DataFrame with DataTile](https://towardsdatascience.com/explore-pandas-dataframe-with-datatile-59c425b55d01/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nGet quick insights into your data with just a few lines of code\n[Edwin Tan](https://towardsdatascience.com/author/edwin.tan/)\nJanuary 27, 20222 min read\n  * [![](../images/004.jpg)](https://towardsdatascience.com/insurance-redlining-investigation-16f743cb0ede/)\n## [Insurance 'Redlining' Investigation](https://towardsdatascience.com/insurance-redlining-investigation-16f743cb0ede/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAnalyzing Minority Access to Private Insurance in 1970s Chicago\n[Alex Yelverton](https://towardsdatascience.com/author/akyelverton/)\nJanuary 13, 20216 min read\n  * [![](../images/005.jpeg)](https://towardsdatascience.com/the-best-resource-for-learning-data-science-51c47d4acad6/)\n## [The Best Resource For Learning Data Science](https://towardsdatascience.com/the-best-resource-for-learning-data-science-51c47d4acad6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nMake your learning journey more efficient.\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nAugust 13, 20213 min read\n  * [![](../images/006.jpg)](https://towardsdatascience.com/smartphone-for-activity-recognition-228b9e75c306/)\n## [Smartphone for Activity Recognition](https://towardsdatascience.com/smartphone-for-activity-recognition-228b9e75c306/)\n[Data Visualization](https://towardsdatascience.com/category/data-science/data-visualization/)\nYour phone knows what you're doing, even without a mic or camera\n[Albers Uzila](https://towardsdatascience.com/author/dwiuzila/)\nMay 30, 202119 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/backpropagation-explained-for-beginners-part-1-building-the-intuition/)\n## [Backpropagation Explained for Beginners (Part 1): Building the Intuition](https://towardsdatascience.com/backpropagation-explained-for-beginners-part-1-building-the-intuition/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLet's discover how neural networks learn, step by step\n[Nikhil Dasari](https://towardsdatascience.com/author/nikhil.dasari/)\nJuly 19, 20269 min read\n  * [![](../images/008.png)](https://towardsdatascience.com/data-domains-where-do-i-start-a6d52fef95d1/)\n## [Data Domains - Where do I start?](https://towardsdatascience.com/data-domains-where-do-i-start-a6d52fef95d1/)\n[](https://towardsdatascience.com/i-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time/)\nPractical guidance from the field\n[Piethein Strengholt](https://towardsdatascience.com/author/piethein/)\nDecember 15, 202115 min read\n  * [![](../images/009.jpeg)](https://towardsdatascience.com/how-lucky-is-a-bowl-of-lucky-charms-9040fe2cc560/)\n## [How Lucky is a Bowl of Lucky Charms?](https://towardsdatascience.com/how-lucky-is-a-bowl-of-lucky-charms-9040fe2cc560/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nThe first bowls of the box somehow seem more magical than the latter bowls.\n[G. Jay Kerns](https://towardsdatascience.com/author/gjkernsysu/)\nMarch 16, 202417 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "metadata": {
    "title": "I Cleaned a Messy CSV File Using Pandas .  Here’s the Exact Process I Follow Every Time. | Towards Data Science",
    "description": "Stop guessing at data cleaning. Use this repeatable 5-step Python workflow to diagnose and fix the most common data flaws.",
    "keywords": null,
    "author": "Ibrahim Salami",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "I Cleaned a Messy CSV File Using Pandas .  Here’s the Exact Process I Follow Every Time.",
    "og:description": "Stop guessing at data cleaning. Use this repeatable 5-step Python workflow to diagnose and fix the most common data flaws.",
    "og:url": "https://towardsdatascience.com/i-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/11/data-cleaning-img.jpg",
    "og:image:width": "1024",
    "og:image:height": "824",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Ibrahim Salami",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "15 min read",
    "article:published_time": "2025-11-26T14:13:52.000Z",
    "article:modified_time": "2025-11-26T14:13:52.000Z",
    "article:author": "https://towardsdatascience.com/author/ibrahim.salami/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_000.txt",
      "chars": 195
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_001.txt",
      "chars": 150
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_002.txt",
      "chars": 80
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_003.txt",
      "chars": 142
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_004.txt",
      "chars": 70
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_005.txt",
      "chars": 23
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_006.txt",
      "chars": 111
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_007.txt",
      "chars": 197
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_008.txt",
      "chars": 197
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_009.txt",
      "chars": 276
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_010.txt",
      "chars": 141
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_011.txt",
      "chars": 230
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_012.txt",
      "chars": 103
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_013.txt",
      "chars": 12
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_014.txt",
      "chars": 351
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_015.txt",
      "chars": 234
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_016.txt",
      "chars": 378
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_017.txt",
      "chars": 285
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_018.txt",
      "chars": 118
    },
    {
      "language": "txt",
      "file": "attachments/96296f36702e/code/block_019.txt",
      "chars": 180
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://towardsdatascience.com/images/fallback-image.jpg",
      "bytes": 5092,
      "reused": true,
      "local_path": "attachments/96296f36702e/images/body-63beb50f664a055ed739.jpg",
      "src": "https://towardsdatascience.com/images/fallback-image.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:46:45Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:46:45Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:46:45Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:46:45Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/@ibbysalam",
      "kind": "channel_reference",
      "status": "reference_only"
    },
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/ibrahim.salami/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-analysis/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-cleaning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-visualization/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/pandas/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/3-methods-to-reduce-overfitting-of-machine-learning-models-4a7e2c1da9ef/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/sonery/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/why-you-should-know-big-data-3c0c161b9e14/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/niklas_lang/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/explore-pandas-dataframe-with-datatile-59c425b55d01/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/edwin.tan/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-data-scientist"
    },
    {
      "type": "external",
      "href": "https://www.kaggle.com/datasets/desolution01/messy-employee-dataset"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/in/ibrahim-salami-059863228/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BxTBPgk2fTXSRGVGhY8QDdQ%3D%3D"
    },
    {
      "type": "external",
      "href": "https://x.com/IbbySalam"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/@ibbysalam"
    },
    {
      "type": "external",
      "href": "https://medium.com/@ibbysalam"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=I+Cleaned+a+Messy+CSV+File+Using+Pandas%E2%80%8A.+%E2%80%8AHere%E2%80%99s+the+Exact+Process+I+Follow+Every%C2%A0Time.&u=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=I+Cleaned+a+Messy+CSV+File+Using+Pandas%E2%80%8A.+%E2%80%8AHere%E2%80%99s+the+Exact+Process+I+Follow+Every%C2%A0Time.&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=I+Cleaned+a+Messy+CSV+File+Using+Pandas%E2%80%8A.+%E2%80%8AHere%E2%80%99s+the+Exact+Process+I+Follow+Every%C2%A0Time.&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh"
    },
    {
      "type": "external",
      "href": "https://insightmediagroup.io/creator-network"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:26:14Z",
  "source_url": "https://towardsdatascience.com/i-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 5,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# I Cleaned a Messy CSV File Using Pandas . Here’s the Exact Process I Follow Every Time.\nStop guessing at data cleaning. Use this repeatable 5-step Python workflow to diagnose and fix the most common data flaws.\n[Ibrahim Salami](https://towardsdatascience.com/author/ibrahim.salami/)\nNovember 26, 202515 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/11/data-cleaning-img.jpg)\nImage by Author\nOften, data isn't perfect. You’re going to encounter a lot of data inconsistencies. Nulls, negative values, string inconsistencies, etc. If these aren’t handled early in your data analysis workflow, querying and analysing your data would be a pain later on.\nNow, I’ve done data cleaning before using SQL and Excel, not really with Python. So, to learn about Pandas (one of Python’s data analysis libraries), I’ll be dabbling in some data cleaning. \nIn this article, I’ll be sharing with you a repeatable, beginner-friendly data cleaning workflow. By the end of this article, you should be pretty confident in using Python for data cleaning and analysis.\n## The Dataset we’ll be working with\nI’ll be working with a synthetic, messy HR dataset containing typical real-world errors (inconsistent dates, mixed data types, compound columns). This dataset is from Kaggle, and it is designed for practising data cleaning, transformation, exploratory analysis, and preprocessing for data visualisation and machine learning.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) roadmap.\nThe dataset contains over 1,000 rows and 13 columns, including employee information such as names, department-region combinations, contact details, status, salaries, and performance scores. It includes examples of:\n  * Duplicates\n  * Missing values\n  * Inconsistent date formats\n  * Erroneous entries (e.g., non-numeric salary values)\n  * Compound columns (e.g., “Department_Region” like “Cloud Tech-Texas” that can be split)\n\n\nIt contains columns like:\n  * Employee_ID: Unique synthetic ID (e.g., EMP1001)\n  * First_Name, Last_Name: Randomly generated personal names\n  * Name: Full name (may be redundant with first/last)\n  * Age: Includes missing values\n  * Department_Region: Compound column (e.g., “HR-Florida”)\n  * Status: Employee status (Active, Inactive, Pending)\n  * Join_Date: Inconsistent format (YYYY/MM/DD)\n  * Salary: Includes invalid entries (e.g., “N/A”)\n  * Email, Phone: Synthetic contact information\n  * Performance_Score: Categorical performance rating\n  * Remote_Work: Boolean flag (True/False)\n\n\nYou can access the dataset [here](https://www.kaggle.com/datasets/desolution01/messy-employee-dataset) and play around with it\nThe dataset is fully synthetic. It does not contain any real individuals’ data and is safe to use for public, academic, or commercial projects.\nThis dataset is in the public domain under the CC0 1.0 Universal license. You are free to use, modify, and distribute it without restriction.\n## Overview of the Cleaning Workflow\nThe data cleaning workflow I’ll be working with consists of 5 simple stages.\n  1. Load\n  2. Inspect\n  3. Clean\n  4. Review\n  5. Export  \n\n\n\nLet’s dive deeper into each of these stages.\n## Step 1 — Load the CSV (And Handle the First Hidden Issues)\nThere are some things to keep in mind before loading your dataset. However, this is an optional step, and we probably wouldn’t encounter most of these issues in our dataset. But it doesn’t hurt to know these things. Here are some key things to consider while loading.\n### Encoding issues (`utf-8`, `latin-1`)\nEncoding defines how characters are stored as bytes in the file. Python and Pandas usually default to UTF-8, which handles most modern text and special characters globally. However, if the file was created in an older system or a non-English environment, it might use a different encoding, most commonly Latin-1\nSo if you try to read a Latin-1 file with UTF-8, Pandas will encounter bytes it doesn’t recognise as valid UTF-8 sequences. You’ll typically see a UnicodeDecodeError when you try to read a CSV with encoding issues.\nIf perhaps the default load fails, you could try to specify a different encoding:\npython\n\n```\n# First attempt (the default)\ntry:\ndf = pd.read_csv(‘messy_data.csv’)\nexcept UnicodeDecodeError:\n# Second attempt with a common alternative\ndf = pd.read_csv(‘messy_data.csv’, encoding=’latin-1')\n```\n\n### Wrong delimiters\nCSV stands for “Comma Separated Values,” but in reality, many files use other characters as separators, like semicolons (common in Europe), tabs, or even pipes (|). Pandas typically defaults to the comma (,).\nSo, if your file uses a semicolon (;) but you load it with the default comma delimiter, Pandas will treat the entire row as a single column. The result would be a DataFrame with a single column containing entire lines of data, making it impossible to work with.\nThe fix is pretty simple. You can try checking the raw file (opening it in a text editor like VS Code or Notepad++ is best) to see what character separates the values. Then, pass that character to the sep argument like so\npython\n\n```\n# If the file uses semicolons\ndf = pd.read_csv('messy_data.csv', sep=';')\n\n# If the file uses tabs (TSV)\ndf = pd.read_csv('messy_data.csv', sep='\\t')\n```\n\n### Columns that import incorrectly\nSometimes, Pandas guesses the data type for a column based on the first few rows, but later rows contain unexpected data (e.g., text mixed into a column that started with numbers).\nFor instance, Pandas may correctly identify 0.1, 0.2, 0.3 as floats, but if row 100 contains the value N/A, Pandas might force the entire column into an object (string) type to accommodate the mixed values. This sucks because you lose the ability to perform fast, vectorised numeric operations on that column until you clean up the bad values.\nTo fix this, I use the dtype argument to tell Pandas what data type a column should be explicitly. This prevents silent type casting.\npython\n\n```\ndf = pd.read_csv(‘messy_data.csv’, dtype={‘price’: float, ‘quantity’: ‘Int64’})\n```\n\n### Reading the first few rows\nYou could save time by checking the first few rows directly during the loading process using the `nrows `parameter. This is great, especially when you’re working with large datasets, as it allows you to test encoding and delimiters without loading the entire 10 GB file.\npython\n\n```\n# Load only the first 50 rows to confirm encoding and delimiter\ntemp_df = pd.read_csv('large_messy_data.csv', nrows=50)\nprint(temp_df.head())\n```\n\nOnce you’ve confirmed the arguments are correct, you can load the full file.\nLet’s load the Employee dataset. I don’t expect to see any issues here.\npython\n\n```\nimport pandas as pd\ndf = pd.read_csv(‘Messy_Employee_dataset.csv’)\ndf\n```\n\n**Output** :\nmarkup\n\n```\n1020 rows × 12 columns\n```\n\nNow we can move on to **Step 2 : Inspection**\n## Step 2 — Inspect the Dataset\nI treat this phase like a forensic audit. I’m looking for evidence of chaos hidden beneath the surface. If I rush this step, I guarantee myself a world of pain and analytical errors down the line. I always run these four crucial checks before writing any transformation code.\nThe following methods give me the full health report on my 1,020 employee records:\n###  `1. df.head()` and `df.tail()`: Understanding the Boundaries\nI always start with a visual check. I use `df.head()` and `df.tail()` to see the first and last five rows. This is my quick sanity check to see if all columns look aligned and if the data visually makes sense.\n**My Finding:**\nWhen I ran `df.head()`, I noticed my `Employee ID` was sitting in a column, and the DataFrame was using the default numerical index **(0, 1, 2, ...)** instead. \nWhile I know I could set `Employee ID` as the index, for now, I’ll leave it. The bigger immediate visual risk I’m looking for here is data misaligned in the wrong column or obvious leading/trailing spaces on names that will cause trouble later.\n###  `2. df.info()`: Spotting Datatype Problems and Missingness\nThis is the most critical method. It tells me the column names, the data types (`Dtype`), and the exact number of non-null values.\n**My Findings on 1,020 Rows:**\n  * **Missing Age:** My total entry count is **1,020** , but the `Age` column only has **809** non-null values. That’s a significant amount of missing data that I'll have to decide how to handle later—do I impute it, or do I drop the rows?\n  * **Missing Salary:** The `Salary` column has **996** non-null values, which is only a minor gap, but still something I must resolve.\n  * **The ID Type Check:** The `Employee ID` is correctly listed as an `object` (string). This isn't right. IDs are identifiers, not numbers to be averaged, and using the string type prevents Pandas from accidentally stripping leading zeros.\n\n\n### 3. Data Integrity Check: Duplicates and Unique Counts\nAfter checking dtypes, I need to know if I have duplicate records and how consistent my categorical data is.\n  * **Checking for Duplicates:** I ran `df.duplicated().sum()` and got a result of **0**. This is perfect! It means I don't have identical rows cluttering up my dataset.\n  * **Checking Unique Values (**`**df.nunique()**`**):** I use this to understand the diversity within each column. Low counts in categorical columns are fine, but I look for columns that should be unique but aren't, or columns that have _too many_ unique values, suggesting typos.\n  * **Employee_ID** have 1020 unique records. This is perfect. It means all records are unique.\n  * **The First_Name / Last_Name field has eight unique records.** _That's a little odd._ This confirms the dataset's synthetic nature. My analysis won't be skewed by a large variety of names, since I'll treat them as standard strings.\n  * **Department_Region** has 36 unique records. There’s high potential for typos _._ 36 unique values for region/department is too many. I will need to check this column for spelling variations (e.g., “HR” vs. “Human Resources”) in the next step.\n  * **Email (** 64 unique records). With 1,020 employees, having only 64 unique emails suggests many employees share the same placeholder email. I will flag this for exclusion from analysis, as it’s useless for identifying individuals.\n  * **Phone (** 1020 unique records). This is perfect because it confirms phone numbers are unique identifiers.\n  * **Age / Performance Score / Status / Remote Work**(2–4 unique records). These low counts are expected for categorical or ordinal data, meaning they are ready for encoding.\n\n\n###  `4. df.describe()`: Catching Odd and Impossible Values\nI use `df.describe()` to get a statistical summary of all my numerical columns. This is the place where truly impossible values—the \"red flags\"—show up instantly. I mostly focus on the `min` and `max` rows.\n**My Findings:**\nI immediately noticed a problem in what I _expected_ to be the `Phone Number` column, which Pandas mistakenly converted to a numerical type.\nmarkup\n\n```\nMean-4.942253 * 10⁹Min-9.994973 * 10⁹Max-3.896086 * 10⁶25%-7.341992e * 10⁹50%4.943997 * 10⁹75%-2.520391e * 10⁹\n```\n\nIt turns out all the phone number values were massive negative numbers! This confirms two things:\nPandas incorrectly inferred this column as a number, even though phone numbers are strings.\nThere must be characters in the text that Pandas cannot interpret (for example, parentheses, dashes, or country codes). I need to convert this to an object type and clean it up completely.\n### 5. `df.isnull().sum()`: Quantifying Missing Data\nWhile `df.info()` gives me non-null counts, `df.isnull().sum()` gives me the total count of nulls, which is a cleaner way to quantify my next steps.\n**My Findings:**\n  * `Age` has**211 nulls** (1020 - 809 = 211), and\n  * `Salary` has **24** nulls (1020 - 996 = 24). This precise count sets the stage for **Step 3.**\n\n\nThis inspection process is my safety net. If I had missed the **negative phone numbers** , any analytical step that involved numerical data would have failed or, worse, produced skewed results without warning. \nBy identifying the need to treat `Phone Number` as a string and the significant missing values in `Age` _now_ , I have a concrete cleaning list. This prevents runtime errors and, critically, ensures that my final analysis is based on plausible, non-corrupted data.\n## Step 3 — Standardise Column Names, Correct Dtypes, and Handle Missing Values\nWith my list of flaws in hand (missing Age, missing Salary, the terrible negative Phone Numbers, and the messy categorical data), I move into the heavy lifting. I treat this step in three sub-phases: ensuring consistency, fixing corruption, and filling gaps.\n### 1. Standardising Column Names and Setting the Index (The Consistency Rule)\nBefore I do any serious data manipulation, I enforce strict consistency on column names. Why? Because typing `df['Employee ID ']` accidentally instead of `df['employee_id']` is a silent, frustrating error. Once the names are clean, I set the index.\nMy golden rule is **snake_case and lowercase everywhere, and ID columns should be the index.**\nI use a simple command to strip whitespace, replace spaces with underscores, and convert everything to lowercase.\npython\n\n```\n# The Standardization Command\ndf.columns = df.columns.str.lower().str.replace(' ', '_').str.strip()\n# Before: ['Employee_ID', 'First_Name', 'Phone']\n# After: ['employee_id', 'first_name', 'phone']\n```\n\nNow that our columns are standardised. I can move on to set employee_id as an index.\npython\n\n```\n# Set the Employee ID as the DataFrame Index\n# This is crucial for efficient lookups and clean merges later.\ndf.set_index('employee_id', inplace=True)\n\n# Let’s review it real quick\nprint(df.index)\n```\n\n**Output:**\nmarkup\n\n```\nIndex(['EMP1000', 'EMP1001', 'EMP1002', 'EMP1003', 'EMP1004', 'EMP1005','EMP1006', 'EMP1007', 'EMP1008', 'EMP1009',...'EMP2010', 'EMP2011', 'EMP2012', 'EMP2013', 'EMP2014', 'EMP2015','EMP2016', 'EMP2017', 'EMP2018', 'EMP2019'],dtype='object', name='employee_id', length=1020)\n```\n\nPerfect, everything is in place.\n### 2. Fixing Data Types and Corruption (Tackling the Negative Phone Numbers)\nMy `df.describe()` check revealed the most urgent structural flaw: the `Phone` column, which was imported as a garbage numerical type. Since phone numbers are _identifiers_ (not quantities), they must be strings.\n  \nIn this phase, I'll convert the entire column to a string type, which will turn all those negative scientific notation numbers into human-readable text (though still full of non-digit characters). I will leave the actual text cleaning (removing parentheses, dashes, etc.) for a dedicated standardisation step (Step 4).\npython\n\n```\n# Fix the Phone dtype immediately\n# Note: The column name is now 'phone' due to standardization in 3.1\ndf['phone'] = df['phone'].astype(str)\n```\n\n### 3. Handling Missing Values (The Age & Salary Gaps)\nFinally, I address the gaps revealed by `df.info()`: the **211** missing `Age` values and the **24** missing `Salary` values (out of **1,020 total rows**). My strategy depends entirely on the column's role and the magnitude of the missing data:\n  * **Salary (24 missing values)** : In this case, removing or dropping all missing values would be the best strategy. Salary is a critical metric for financial analysis. Imputing it risks skewing conclusions. Since only a small fraction (2.3%) is missing, I choose to drop the incomplete records.\n  * **Age (211 missing values)**. Filling the missing values is the best strategy here. Age is often a feature for predictive modelling (e.g., turnover). Dropping 20% of my data is too costly. I’ll fill the missing values using the **median age** to avoid skewing the distribution with the mean.\n\n\nI execute this strategy with two separate commands:\npython\n\n```\n# 1. Removal: Drop rows missing the critical 'salary' data\ndf.dropna(subset=['salary'], inplace=True)\n\n# 2. Imputation: Fill missing 'age' with the median\nmedian_age = df['age'].median()\ndf['age'].fillna(median_age, inplace=True)\n```\n\nAfter these commands, I would run `df.info()` or `isnull().sum()` again just to confirm that the non-null counts for `salary` and `age` now reflect a clean dataset.\npython\n\n```\n# Rechecking the null counts for salary and age\ndf[‘salary’].isnull().sum())\ndf[‘age’].isnull().sum())\n```\n\n**Output:**\nmarkup\n\n```\nnp.int64(0)\n```\n\nSo far so good!\nBy addressing the structural and missing data issues here, the subsequent steps can focus entirely on value standardisation, such as the messy **36 unique values in**`**department_region**`—which we tackle in the next phase.\n## Step 4 — Value Standardization: Making Data Consistent\nMy DataFrame now has the right structure, but the values inside are still dirty. This step is about consistency. If “IT,” “i.t,” and “Info. Tech” all mean the same department, I need to force them into a single, clean value (“IT”). This prevents errors in grouping, filtering, and any statistical analysis based on categories.\n### 1. Cleaning Corrupted String Data (The Phone Number Fix)\nRemember the corrupted `phone` column from Step 2? It's currently a mess of negative scientific notation numbers that we converted to strings in Step 3. Now, it's time to extract the actual digits.\nSo, I’ll be removing every non-digit character (dashes, parentheses, dots, etc.) and converting the result into a clean, unified format. Regular expressions (`.str.replace()`) are perfect for this. I use `\\D` to match any non-digit character and replace it with an empty string.\npython\n\n```\n# The phone column is currently a string like '-9.994973e+09'\n# We use regex to remove everything that isn't a digit\ndf['phone'] = df['phone'].str.replace(r'\\D', '', regex=True)\n\n# We can also truncate or format the resulting string if needed\n# For example, keeping only the last 10 digits:\ndf['phone'] = df['phone'].str.slice(-10)\nprint(df['phone'])\n```\n\n**Output:**\nmarkup\n\n```\nemployee_idEMP1000 1651623197EMP1001 1898471390EMP1002 5596363211EMP1003 3476490784EMP1004 1586734256...EMP2014 2470739200EMP2016 2508261122EMP2017 1261632487EMP2018 8995729892EMP2019 7629745492Name: phone, Length: 996, dtype: object\n```\n\nLooks much better now. This is always a good practice to clean identifiers that contain noise (like IDs with leading characters or zip codes with extensions).\n### 2. Separating and Standardizing Categorical Data (Fixing the 36 Regions)\nMy `df.nunique()` check revealed **36 unique values** in the `department_region` column. When I reviewed all the unique values in the column, the output revealed that they are all neatly structured as `department-region` (e.g., devops-california, finance-texas, cloud tech-new york).\nI guess one way to solve this is to split this single column into two dedicated columns. I'll split the column at the hyphen (`-`) and assign the parts to new columns: `department` and `region`.\npython\n\n```\n# 1. Split the combined column into two new, clean columns\ndf[['department', 'region']] = df['department_region'].str.split('-', expand=True)\nNext, I’ll drop the department_region column since it’s pretty much useless now\n# 2. Drop the redundant combined column\ndf.drop('department_region', axis=1, inplace=True)\nLet’s review our new columns\nprint(df[[‘department’, ‘region’]])\n```\n\n**Output:**\nmarkup\n\n```\ndepartment regionemployee_idEMP1000 devops californiaEMP1001 finance texasEMP1002 admin nevadaEMP1003 admin nevadaEMP1004 cloud tech florida... ... ...EMP2014 finance nevadaEMP2016 cloud tech texasEMP2017 finance new yorkEMP2018 hr floridaEMP2019 devops illinois[996 rows x 2 columns]\n```\n\nAfter splitting, the new `department` column has only **6 unique values** (e.g., 'devops', 'finance', 'admin', etc.). This is great news. The values are already standardised and ready for analysis! I guess we could always map all similar departments to one single category. But I’m gonna skip that. I don’t want to get too advanced in this article.\n### 3. Converting Date Columns (The Join_Date Fix)\nThe `Join_Date` column is usually read in as a string (`object`) type, which makes time-series analysis impossible. This means we have to convert it to a proper Pandas `datetime` object.\n`pd.to_datetime()` is the core function. I often use `errors='coerce'` as a safety net; if Pandas can't parse a date, it converts that value to `NaT` (Not a Time), which is a clean null value, preventing the whole operation from crashing.\npython\n\n```\n# Convert the join_date column to datetime objects\ndf['join_date'] = pd.to_datetime(df['join_date'], errors='coerce')\n```\n\nThe conversion of dates enables powerful time-series analysis, like calculating average employee tenure or identifying turnover rates by year.\nAfter this step, every value in the dataset is clean, uniform, and correctly formatted. The categorical columns (like `department` and `region`) are ready for grouping and visualisation, and the numerical columns (like `salary` and `age`) are ready for statistical modeling. The dataset is officially ready for analysis.\n## Step 5 — Final Quality Check and Export\nBefore closing the notebook, I always perform one last audit to ensure everything is perfect, and then I export the data so I can perform analysis on it later.\n### The Final Data Quality Check\nThis is quick. I re-run the two most critical inspection methods to confirm that all my cleaning commands actually worked:\n  * `df.info()`**:** I confirm there are **no more missing values** in the critical columns (`age`, `salary`) and that the data types are correct (`phone` is a string, `join_date` is datetime).\n  * `df.describe()`**:** I ensure the statistical summary shows plausible numbers. The `Phone` column should now be absent from this output (since it's a string), and `Age` and `Salary` should have logical minimum and maximum values.\n\n\nIf these checks pass, I know the data is reliable.\n### Exporting the Clean Dataset\nThe final step is to save this cleaned version of the data. I usually save it as a new CSV file to keep the original messy file intact for reference. I use `index=False` here if I don't want the `employee_id` (which is now the index) to be saved as a separate column, or `index=True` if I want to save the index as the first column in the new CSV.\npython\n\n```\n# Exporting the clean DataFrame to a new CSV file\n# We use index=True to keep our primary key (employee_id) in the exported file\ndf.to_csv('cleaned_employee_data.csv', index=True)\n```\n\nBy exporting with a clear, new filename (e.g., `_clean.csv`), you officially mark the end of the cleaning phase and provide a clean slate for the next phase of the project.\n## Conclusion\nHonestly, I used to feel overwhelmed by a messy dataset. The missing values, the weird data types, the cryptic columns — it felt like facing the blank page syndrome.\nBut this structured, repeatable workflow changed everything. By focusing on **Load, Inspect, Clean, Review, and Export** , we established order instantly: standardizing column names, making the `employee_id` the index, and using smart strategies for imputation and splitting messy columns.\nNow, I can jump straight into the fun analysis part without constantly second-guessing my results. If you struggle with the initial data cleaning step, try out this workflow. I’d love to hear how it goes. If you want to play around with the dataset, you can download it [here](https://www.kaggle.com/datasets/desolution01/messy-employee-dataset).\nWanna connect? Feel free to say hi on these platforms\n[LinkedIn](https://www.linkedin.com/in/ibrahim-salami-059863228/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BxTBPgk2fTXSRGVGhY8QDdQ%3D%3D)\n[Twitter](https://x.com/IbbySalam)\n[YouTube](https://www.youtube.com/@ibbysalam)\n[Medium](https://medium.com/@ibbysalam)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657709013_9ddcph.jpeg)\nWritten By\n## [Ibrahim Salami](https://towardsdatascience.com/author/ibrahim.salami/)\n[See all from Ibrahim Salami](https://towardsdatascience.com/author/ibrahim.salami/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Data Analysis](https://towardsdatascience.com/tag/data-analysis/)[Data Cleaning](https://towardsdatascience.com/tag/data-cleaning/)[Data Visualization](https://towardsdatascience.com/tag/data-visualization/)[Pandas](https://towardsdatascience.com/tag/pandas/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F&title=I%20Cleaned%20a%20Messy%20CSV%20File%20Using%20Pandas%E2%80%8A.%20%E2%80%8AHere%E2%80%99s%20the%20Exact%20Process%20I%20Follow%20Every%C2%A0Time.)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F&title=I%20Cleaned%20a%20Messy%20CSV%20File%20Using%20Pandas%E2%80%8A.%20%E2%80%8AHere%E2%80%99s%20the%20Exact%20Process%20I%20Follow%20Every%C2%A0Time.)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fi-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time%2F&text=I%20Cleaned%20a%20Messy%20CSV%20File%20Using%20Pandas%E2%80%8A.%20%E2%80%8AHere%E2%80%99s%20the%20Exact%20Process%20I%20Follow%20Every%C2%A0Time.)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/3-methods-to-reduce-overfitting-of-machine-learning-models-4a7e2c1da9ef/)\n## [3 Methods to Reduce Overfitting of Machine Learning Models](https://towardsdatascience.com/3-methods-to-reduce-overfitting-of-machine-learning-models-4a7e2c1da9ef/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn overfit model is full of surprises but not the good ones. I have come...\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJune 2, 20215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1oqfRuIleu4CfJ2cEDzHNYg-scaled.jpeg)](https://towardsdatascience.com/why-you-should-know-big-data-3c0c161b9e14/)\n## [Why you should know Big Data](https://towardsdatascience.com/why-you-should-know-big-data-3c0c161b9e14/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nDefining Big Data and its potential threats\n[Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\nFebruary 5, 20229 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/0hRzckgR18fEPSiqG-scaled.jpg)](https://towardsdatascience.com/explore-pandas-dataframe-with-datatile-59c425b55d01/)\n## [Explore Pandas DataFrame with DataTile](https://towardsdatascience.com/explore-pandas-dataframe-with-datatile-59c425b55d01/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nGet quick insights into your data with just a few lines of code\n[Edwin Tan](https://towardsdatascience.com/author/edwin.tan/)\nJanuary 27, 20222 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/0_WlofpnM-cZ3G9Lo-scaled.jpg)](https://towardsdatascience.com/insurance-redlining-investigation-16f743cb0ede/)\n## [Insurance 'Redlining' Investigation](https://towardsdatascience.com/insurance-redlining-investigation-16f743cb0ede/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAnalyzing Minority Access to Private Insurance in 1970s Chicago\n[Alex Yelverton](https://towardsdatascience.com/author/akyelverton/)\nJanuary 13, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/178tnShF7-eQp4dLQc7jFvA-scaled.jpeg)](https://towardsdatascience.com/the-best-resource-for-learning-data-science-51c47d4acad6/)\n## [The Best Resource For Learning Data Science](https://towardsdatascience.com/the-best-resource-for-learning-data-science-51c47d4acad6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nMake your learning journey more efficient.\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nAugust 13, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/0ekmX89TW6N8Bi8bL-scaled.jpg)](https://towardsdatascience.com/smartphone-for-activity-recognition-228b9e75c306/)\n## [Smartphone for Activity Recognition](https://towardsdatascience.com/smartphone-for-activity-recognition-228b9e75c306/)\n[Data Visualization](https://towardsdatascience.com/category/data-science/data-visualization/)\nYour phone knows what you're doing, even without a mic or camera\n[Albers Uzila](https://towardsdatascience.com/author/dwiuzila/)\nMay 30, 202119 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/pexels-vargaphotography-10796342-scaled-1.jpg)](https://towardsdatascience.com/backpropagation-explained-for-beginners-part-1-building-the-intuition/)\n## [Backpropagation Explained for Beginners (Part 1): Building the Intuition](https://towardsdatascience.com/backpropagation-explained-for-beginners-part-1-building-the-intuition/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLet's discover how neural networks learn, step by step\n[Nikhil Dasari](https://towardsdatascience.com/author/nikhil.dasari/)\nJuly 19, 20269 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/1yRkY7XYvjQcX06X6twK70g.png)](https://towardsdatascience.com/data-domains-where-do-i-start-a6d52fef95d1/)\n## [Data Domains - Where do I start?](https://towardsdatascience.com/data-domains-where-do-i-start-a6d52fef95d1/)\n[](https://towardsdatascience.com/i-cleaned-a-messy-csv-file-using-pandas-heres-the-exact-process-i-follow-every-time/)\nPractical guidance from the field\n[Piethein Strengholt](https://towardsdatascience.com/author/piethein/)\nDecember 15, 202115 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1LLwBwCx_S615bcwrJ366ug.jpeg)](https://towardsdatascience.com/how-lucky-is-a-bowl-of-lucky-charms-9040fe2cc560/)\n## [How Lucky is a Bowl of Lucky Charms?](https://towardsdatascience.com/how-lucky-is-a-bowl-of-lucky-charms-9040fe2cc560/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nThe first bowls of the box somehow seem more magical than the latter bowls.\n[G. Jay Kerns](https://towardsdatascience.com/author/gjkernsysu/)\nMarch 16, 202417 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "body_base_path": "attachments/96296f36702e/pages/",
  "body_markdown_path": "attachments/96296f36702e/pages/archived-body.md",
  "local_html": "attachments/96296f36702e/pages/archived-body.html",
  "body_source": "prior_capture"
}