保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Spearman coefficient: Tool for a generalized correlation analysis

Linear relationships are not all a correlation analysis can reveal. We discuss rank-based correlation that is more generalized and... Tirthajyoti Sarkar November 1, 20214 min read

Linear relationships are not all a correlation analysis can reveal. We discuss rank-based correlation that is more generalized and powerful.

Beyond linear relationship

It's a common data science interview question.

Say, y = _x_² + noise, you plot y vs. x and try to determine the correlation coefficient. What will it be? Most interviewers won't give you credit if you say anything other than zero. The word correlation, in most circumstances, invokes the concept of linearity or linear relationship between variables. Here is a visual illustration of what I meant. Learn this step by step with the interactive AI and Data Scientist, Computer Science, Machine Learning and Data Analyst roadmaps. Image source: Created by the author Image source: Created by the author This notion of linearity is coming from the definition of a particular correlation coefficient that is most widely used in statistical analysis or in data science - Pearson's coefficient. It is named after the famous English statistician Karl Pearson and is given by, where cov(X ,Y) is the covariance of two random variables X and Y and the denominator consists of the product of their standard deviations. Karl Pearson, Image source: Wikimedia, Free media to use Karl Pearson, Image source:Wikimedia, Free media to use Pearson's coefficient suffices for a first-order analysis but suffers from outliers and cannot handle nonlinearity by definition. Therefore, there is a need to look beyond the linear correlation analysis. For such generalized correlation analysis, there are a few more tools. Among them, Spearman's coefficient is the most straightforward to understand and calculate. In this article, we will discuss that.

Spearman's coefficient

Spearman's coefficient is a rank correlation (a measure of statistical dependence between the rankings of two random variables). It is named after Charles Spearman, an English psychologist, famous for factor analysis among other contributions. So, how is this different from the regular correlation coefficient we are familiar with? Spearman coefficient is different in the sense that it is not just about the raw numerical values, but about rankings.

What's a rank?

So, what is a rank? Here is the Wikipedia definition, "A ranking is a relationship between a set of items such that, for any two items, the first is either "ranked higher than", "ranked lower than" or "ranked equal to" the second. " Take two sets of numbers - 1,2,3 and 10,24,102. The exact numerical relationship between these sets may look odd, but they are very similar in a "one-to-one" sense if we consider their ranking only. The second number in each set is greater than the first one and it is lesser than the third number. Two sets are identical from that point of view. The relative order matters and not the actual numerical values. The Pearson (linear) correlation coefficient of these two sets can be quite different than 1. But the Spearman's coefficient will be a perfect 1. Following a similar formula like the Pearson coefficient, it is given by, The only difference is that all the raw numerical values are replaced by their rank values (i.e., order values 1st, 2nd, 3rd, 50th percentile, etc.).

An example with Python

Using the Scipy library, it is quite simple to evaluate the Spearman coefficient of two sets of 1-D or 2-D arrays. Let's show the stark difference between the two types of correlation with a canonical example, text

import numpy as np, matplotlib.pyplot as plt
import scipy.stats as stat

text

x = np.linspace(0,10,20)
y = np.log(1+1/(x+0.1))

text

pearson = stat.pearsonr(x,y)
spearman = stat.spearmanr(x,y)

If we check their values, they will be -0.65 and -1.0 respectively. Here is the visual representation, Image source: Created by the author Image source: Created by the author Why is Spearman's coefficient a perfect -1.0? Because the rank order is exactly identical between the two arrays. As long as the underlying function follows a monotonic (increasing or decreasing) order, this will be true. Obviously, the generating function is a highly nonlinear one (inverse of a logarithm) but that does not impact Spearman's coefficient.

Spearman coefficient is more robust to outliers

If we introduce some random outliers to the same arrays as above, we will see that Spearman's coefficient is more robust to those outliers as the rank order is disturbed much less than the actual numeric values. Here is a sample result with outliers. Image source: Created by the author Image source: Created by the author The Pearson's coefficient is disturbed by the outliers (from -0.65 to -0.79), but the Spearman's coefficient hardly moved (from -1.0 to -0.99).

Summary

In this article, we discussed Spearman's coefficient which is equivalent to the more widely used Pearson's coefficient but for ranked orders of inter-related arrays. It is a special case of more generalized correlation analysis that looks at multiple levels of properties associated with arrays of numbers or sets of values that two random variables can assume. In many cases, going beyond the linear Pearson coefficient can be used such as in the presence of outliers or when rank ordering is more interesting to analyze than raw numerical values. Concepts like Spearman's coefficient will come in handy for data scientists in such scenarios.

Loved the article? Become a Medium member to continue learning without limits . I'll receive a portion of your membership fee if you use the following link,__with no extra cost to you .

Join Medium with my referral link - Tirthajyoti Sarkar

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Tirthajyoti Sarkar

See all from Tirthajyoti Sarkar AnalyticsData ScienceMachine LearningTechnology Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Back-propagation Demystified [Part 1]

Artificial Intelligence Read to learn about the back-propagation algorithm in the context of Deep Learning Manpreet Singh Minhas December 19, 20205 min read

TaatikNet: Sequence-to-Sequence Learning for Hebrew Transliteration

Artificial Intelligence A simple demonstration of character-level seq2seq learning applied to a complex task: converting between Hebrew text and Latin... Morris Alper June 28, 202311 min read

What Do 101 Dalmations and Machine Learning Have in Common?

Data Science There are at least 101 Examples of Machine Learning Data Scientists can use to Create Valuable Insights. Here they are. Ron Ozminkowski, PhD November 30, 202133 min read

How to Find Seasonality Patterns in Time Series

Programming Using Fourier Transform to detect seasonal components Lorenzo Mezzini February 3, 20257 min read

The Ultimate Guide to Cracking Business Case Interviews for Data Scientists (Part 1)

Data Science Metrics frameworks for data scientists to ace product sense interviews Emma Ding March 9, 202111 min read

I Have Been Using Pandas for 3 Years - Here are the 8 Functions I Use the Most

Artificial Intelligence A practical guide Soner Yıldırım February 2, 20227 min read

Survival Analysis When No One Dies: A Value-Based Approach

Statistics A generalized version of Kaplan-Meier allows to model a continuous value (like money) instead of a binary signal (like survival) Samuele Mazzanti May 13, 20259 min read

Steps to Start Training your Custom Tensorflow Model in AWS SageMaker

Machine Learning Create an experiment, train and use checkpoints for a Transformer model Eduardo Muñoz November 19, 20202 min read

Ultimate Function for Classification

Data Science Evaluating Classifiers with a Custom Function Darius Fuller September 25, 20209 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close