保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Stop Being Data-Driven

Why we are fooled by data and how to stop it Cai Parry-Jones August 29, 202411 min read source: unsplash.com source: unsplash.com Often, when making decisions based on data, we feel as if we have made a more intelligent, accurate choice. The reality is a little different. From real estate purchases to dietary choices to corporate board selections, data can be an amazing catalyst to help make the worst decisions. Imagine data as that brilliant yet unpredictable friend of yours. One moment they're a genius, the next a complete mess. Would you hand them the keys to your beloved Ford Focus? Or would you politely insist they take the passenger seat? This article explores why data belongs firmly in the passenger seat, not behind the wheel. It will examine real-world examples where data-driven decisions could have steered us off course and into the cliff of misinformation. Fortunately, this article also gives actionable, easy-to-use advice on how to avoid such scenarios by following a four-step ladder to assess the quality of data-driven insights:

  1. Evaluating the data source,
  2. Considering the bias of the data presenter,
  3. Recognising the bias of the data reader,
  4. Identifying any logical missteps between the data and the insight.

By the end of this journey, you'll be better prepared to navigate the road of decision-making, while keeping your unpredictable friend safely buckled in the passenger seat. Learn this step by step with the interactive AI and Data Scientist roadmap.

TL;DR

Question One - Where did the data come from?

Here is an excerpt from an article published in The Spectator, titled: 'The unfashionable truth about the riots'. Its subject is the 2024 anti-immigration riots in the UK.

...I decided to do some checking on the employment stats for some of the northern towns that have seen the worst rioting in the past week. I also checked the 2011 statistics and then compared the two. I should warn you in advance that if you're easily depressed, you should look away now. Back in 2011, the proportion on out-of-work benefits (including incapacity benefit) in Sunderland was 18 per cent; today it is 19 per cent. In 2011 the unemployment figure in Rotherham was 16 per cent; today it is 18 per cent. In Hartlepool, it was 21 per cent; today, 23 per cent.

So, where did this data come from? In the example, all we have is "the employment stats"... What the heck does that mean? In the UK there isn't a centralised place for employment statistics, so it's far from obvious what data the author is referring to. You might be content with dismissing the data and thus dismissing the rest of the article, since its premise is based on this data. However, I'm a try-hard, so I put on my best Sherlock Holmes hat and set about finding the un-cited data. Sadly, I am not Sherlock Holmes. Despite considerable online sleuthing, I couldn't find any data that matched the article's. I did, however, find other "employment stats". Specifically the Office of National Statistics (ONS)'s measure of unemployment by region. It paints a very different picture. In fact the polar opposite picture! Of the three areas the article highlights as having reduced in employment since 2011, the ONS in fact finds they all increased!* I'm afraid the only unfashionable truth illuminated by this article is the degradation of news media. (Or was it always like this?) *Data links: Sunderland, Rotherham, and Hartlepool employment stats.

Question Two - What if the opposite result were true?

...Using laboratory and field experiments, we find that signing before–rather than after–the opportunity to cheat makes ethics salient when they are needed most and significantly reduces dishonesty.

This quote comes from the abstract of a highly influential research paper in Behavioural Science. In case you didn't quite understand the quote, since it is written in the classic academic obfuscation style, the quote is saying that signing your name on the top of a document makes you less likely to lie about the document than if you had signed on the bottom. The data came from a collection of researchers based in highly regarded universities, most notably Francesca Gino who was a professor at Harvard Business School at the time of publication. For me, this passes our first question about the data ('Where did the data come from?'). Next we need to ask: 'Would the data be presented if the opposite result were true?' To answer this we first need to understand a little about the people and/or institutes presenting the data. In this case it is a group of Behavioural Science academics. Academia is a cut-throat industry, with an exceptionally high failure rate. Therefore, it is critical for academics to be considered 'successful' as early as possible to avoid being culled. The key factor to determine an academic's success is, unsurprisingly, their work. But how do you measure the quality of their work? A common method is citation count, that is the number of times an academic's work is mentioned in the work of other academics. The issue with this is you are far more likely to to be cited if you have an unexpected/interesting result in your paper. Going back to the original quote, do you think a result where there was no difference in honesty between the group who signed at the top vs the bottom of a document would have been nearly as influential? No, because that would have been the expectation. The conclusion is that there is a strong incentive for academics to generate unexpected/interesting results. Therefore, in our example, we should understand that there is a strong bias for the authors to claim a (significant) difference in honesty between the two groups. This sort of bias is so strong that it has led the majority of scientific research to be unreproducible by fellow academics. Unreproducible means an academic writes a research paper that copies another academic's research paper's method/experiment, but ends up with significantly different results to the original paper. Turning once more to our case study as an example, following the impactful success of the research paper, a number of other studies attempted to run the same or similar experiment. All results of the experiments found no correlation between honesty and location of document signature. It turned out the original paper's finding was very likely the result of data fraud. As were a number of other influential papers co-authored by the prestigious Harvard professor, Francesca Gino. This sort of skin-in-the-game bias isn't only found in academia. From tech vendors posting their latest performances vs competitors, to investment banks presenting to prospective clients that they are no. 1 in the league tables*, bias is rife. And it isn't only other people you should be concerned about. The strongest bias of all is most likely you—yes, YOU. Commonly referred to as confirmation bias, it is the tendency to search for, interpret, favour, and recall information in a way that confirms or supports your prior beliefs. Ever wondered why such a mass of people could be so stupid on so many political topics? Sadly, it is more likely a result of your own confirmation bias than the collective dim-wittedness of millions of people. If you want to avoid the pitfalls of your own biases, as well as the biases of the data presenter, you not only need to ask: 'Would the data be presented if the opposite result were true?' But also: 'Would I consider the data presented if the opposite result were true?' It may be a tough pill to swallow, but overcoming your own bias may enlighten you more than all the research in the world. *May Contain Lies , Alex Edmans. Page 130.

Question Three - Is there a flaw with the leap from data to insight?

For most university graduates, having a degree pays. Over the course of a lifetime, estimates suggest women can expect to earn about £250,000 more if they have a degree, while the figure is roughly £170,000 for men.

This quote comes from the news department of the prestigious British Broadcasting Corporation (BBC). Specifically, their article titled: 'The degrees that make you rich... and the ones that don't'. First things first, is there a reliable data source? The source. BBC News, The degrees that make you rich... and the ones that dont The source. BBC News, The degrees that make you rich... and the ones that don't Great news, the answer is (probably*) yes! The article cites their data from another prestigious organisation, the Institute of Fiscal Studies. You may want to delve deeper into the exact report of the data source, but for a surface-level analysis, I'm happy with where this raw data has come from. Second we want to ask ourselves, 'would this information be presented if the opposite result was true?' Given the BBC has (to my knowledge) no reason to support higher education institutions, it's fair to assume that the BBC would have published an article highlighting no financial benefits to studying at university if the data suggested it. However, unfortunately, we still can't trust the BBC's findings just yet. We have one final important question to ask: 'Does the analysis make sense?' After looking at the article's chart on average earnings by subject, have a go at answering the following question before moving on to the next paragraph: 'Given people with Medicine degrees earn more than the average degree, does it make sense to assume that the Medicine degree was the cause of the higher salary?' My answer: Medicine degrees are necessary to become qualified doctors, and doctors generally earn a high income. This supports the idea that a Medicine degree is the specific reason for average higher future earnings. On the other hand, you don't just need to pick Medicine, you also need to be picked for Medicine. Given the average A-level results for a UK Medicine graduate is an outstanding AAA, you could argue the reason for the high average salaries is because the average Medicine student is exceptionally cognitively capable (in academia at least), which has nothing to do with whether they actually chose Medicine as a degree or not. What about the degree with the second highest average salary, Economics? Unlike Medicine, you don't strictly need an Economics degree to qualify for any specific high-paying jobs. I'd argue an Economics degree is much less likely a direct factor in increasing a graduate's future salary. For example, someone who is genuinely interested in money at school will be more likely to base their career decisions on what pays more. That same person may also be more likely to pick Economics at university.** If that's true, then pushing someone who is not interested in money to study Economics may not benefit their future career or financial prospects as much as the article is implying. Similarly, someone who earns a university degree in general may be little to no better off than if they hadn't gone to university at all. It may simply be a side effect. For example, in the UK children with wealthy parents are significantly more likely to attend university. The wealth of their parents may also have an impact on the child's high-paying job prospects; for example, they might provide invaluable knowledge to their children on how to interview, climb the greasy corporate ladder, and/or network. Given this thought process, the article's finding that "For most university graduates, having a degree pays" is, at the very least, not a proven fact, and should not be treated as such. Deciding what 'makes sense' is subjective; for example, you may not have been convinced by my arguments above. However, it's important to recognise the leaps in assumptions people make from data to insights, and to do your own 'makes sense' assessment before accepting them. *It's important to acknowledge the limitations of time we have on the depth of our investigation. For very serious things you will want to go deeper, but as a pragmatist, it isn't possible for every data analysis we see. **You may be thinking: 'hang on a minute, you didn't provide any data to back up that theory'. And you're right! However, often times we don't have all the data available to us, we simply have to use our own experiences and intuition. As an Economics graduate who has also been fascinated with finance from an early age, the theory makes sense to me.

Wrapping up

In conclusion, data should not be the driver of your decisions. Like an unpredictable friend, data belongs firmly in the passenger seat, preferably with the child-lock on. They can provide suggestions, but ultimately you need to be sceptical before making any decision based off of them. Remember, when looking into data, statistics, and/or research, you need to ask yourself:

  1. Where did this data come from?
  2. Would this data be presented if the opposite result were true?
  3. Would I consider the data if the opposite result were true?
  4. Is there a flaw with the leap from data to insight?

Final note for data scientists/analysts

For data scientists and analysts, these questions are not just a safeguard - they're a responsibility. As gatekeepers of data-driven insights, your role is crucial in ensuring that decisions are informed by sound reasoning, not just raw numbers and algorithms. By applying this mindset, you can avoid being that unpredictable friend in your organisation. And who know's, one day you may even be invited into the driving seat.

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Cai Parry-Jones

See all from Cai Parry-Jones Business StrategyConfirmation BiasData AnalysisData ScienceLogic Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Markov Chain Monte Carlo in Python

A Complete Real-World Implementation The past few months, I encountered one term again and again... Will Koehrsen February 9, 201812 min read

Analyzing Geospatial Data with Python

Data Science A practical data analysis post with Python code. Gustavo Santos August 19, 20237 min read

Confidence Intervals with Python

Machine Learning College Statistics with Python Luís Roque June 22, 202114 min read

Temporary Variables in Python: Readability versus Performance

Data Science Temporary variables can make code clearer. What about the performance of such code? Marcin Kozak June 1, 20238 min read

Attribute Relevance Analysis in Python - IV and WoE

Machine Learning Recently I've written about Recursive Feature Elimination - one of many feature selection techniques I use most often. Today I will speak... Dario Radečić September 9, 20199 min read

The Power and Pitfalls of Vector-Based Image Search

Artificial Intelligence A hands-on guide to setting up image similarity search in Milvus, and why visual replication isn't always enough. Soner Yıldırım June 18, 20267 min read

The Misconception of Retraining: Why Model Refresh Isn’t Always the Fix

Artificial Intelligence Retraining is easy; knowing when not to is the real challenge. In machine learning, performance drops are rarely about stale weights; they’re about misunderstood signals. Shafeeq Ur Rahaman July 30, 20256 min read

An Overview of The PEP 8 Style Guide

Programming Have style with your Python code. Dan Root October 15, 202010 min read

Exploring the Tokyo Neighborhoods: Data-Science in Real Life

Machine Learning Web-Scrapping, Foursquare API, Folium Saptashwa Bhattacharyya April 27, 20199 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close