{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML Reverse Lecture\n",
    "\n",
    "In this lab we will look at some examples on how to use ML for classification tasks over a music dataset.\n",
    "\n",
    "Specifically, we will use a Kaggle dataset with 218k songs [fetched from Spotify](https://github.com/tgel0/spotify-data). It has a schema that is almost identical to the one we used for lab 2, but with two additional columns: `genre`, and `popularity`. Schema columns are described in the [Spotify API docs](https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/).\n",
    "\n",
    "First, we will do some data exploration over the top 1k most popular songs in the dataset. Our goal in this phase is to understand some of the characteristics that these most popular songs have in common. We'll use visualization techniques and unsupervised ML (k-means clustering, PCA) to better understand the relationship between some of the features.\n",
    "\n",
    "Next, we'll try out a genre prediction task on the entire 218k songs dataset. We'll use supervised ML (classification) for this, and give you a chance to try out the same problem using regression.\n",
    "\n",
    "Finally, we'll see if we can use what we learned during the genre prediction task to try and predict song popularity.\n",
    "\n",
    "\n",
    "## Running this notebook\n",
    "\n",
    "To execute code from a cell, you can either click `Run` at the top, or type `shift+Enter` after clicking a cell.  You can either run the entire notebook (`Restart & Run All` from the `Kernel` drop-down), or run each cell individually.  If you choose the latter, note that it is important that you run cells in order, as later cells depend on earlier ones. And to be able to `Run All` successfully, you'll have to write code for answering some of the questions in the notebook.\n",
    "\n",
    "Once you open your notebook on the browser, and check that the cells are rendering correctly (e.g., try running the \"Python packages\" cell below), we're good to go from there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python packages we'll need\n",
    "\n",
    "First, import the python packages we'll be using for the lab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataframes.\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Plotting.\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib notebook\n",
    "import seaborn as sns\n",
    "sns.set_style('whitegrid')\n",
    "\n",
    "# ML.\n",
    "from sklearn import tree\n",
    "from sklearn.cluster import DBSCAN, KMeans\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import classification_report, make_scorer, accuracy_score, roc_auc_score \n",
    "from sklearn.model_selection import cross_val_score, KFold, train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import SVC, LinearSVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "# Ignore warnings.\n",
    "import warnings\n",
    "warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: data exploration\n",
    "\n",
    "First, we load our dataset into a pandas dataframe. Next, we store the top 1k most popular songs in a separate `top1k` dataframe.\n",
    "\n",
    "**Q1: Filter `df` for the top 1000 songs with largest `popularity` values, and store it on a `top1k` dataframe below.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/spotify_songs.csv')\n",
    "\n",
    "# Q1: YOUR CODE GOES HERE.\n",
    "# top1k = (...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make sure that loading our data worked as we expect. A natural way to do that is to inspect the first few entries with [`head()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.head.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that most of these columns are familiar to us, as they were present in our `top2018.csv` dataset from lab 2. The additional columns here are only `genre` and `popularity`.\n",
    "\n",
    "Let's examine the data types for each of the columns using [`info()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.info.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of `df.info()` tells us that we have:\n",
    "- 7 `object` columns: (`genre`, `artist_name`, `track_name`, `track_id`, `key`, and `time_signature`)\n",
    "- 2 `int64` columns (`popularity` and `duration_ms`)\n",
    "- 9 `float64` (remaining columns)\n",
    "\n",
    "We can also take a look at aggregate stats for each of the numerical columns (both discrete `int64`, and continuous `float64`) the dataset has by running [`describe()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Except for `popularity`, `duration_ms`, `loudness`, and `tempo`, most of the numerical features are already scaled between 0.0 and 1.0.\n",
    "\n",
    "**Q2: Looking at the documentation for [describe()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html), what's a command to describe categorical features? Why is the output different for categorical vs numerical features?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q2: YOUR CODE GOES HERE\n",
    "# df.describe(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's check whether this dataset has missing data in the form of `null`s:\n",
    "\n",
    "**Q3: How would you check whether your data set has null values in any of its columns? Did you find any?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q3: YOUR CODE GOES HERE.\n",
    "# df.(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the dataset's [correlation matrix](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html) to see how column values co-occur with each other. This can help us spot features that may be predictive of other features in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = df.corr()\n",
    "corr.style.background_gradient().set_precision(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that `energy` and `loudness` is the pair of features with the strongest positive correlation, followed by `valence` and `danceability`. We also see strong negative correlation between `acousticness` and a few other features. Finally, some of these have a medium to strong correlation with popularity.\n",
    "\n",
    "**Q4: Why don't we see `genre`, `key`, and `time signature` on the correlation matrix? What could we do to address this?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look at the [documentation for these features](https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/), we see that `energy` is defined as a \"perceptual measure of intensity and activity\".  The example they give for this is death metal (high energy) vs a Bach prelude (low energy). Similarly, `valence` is a score between `0.0` and `1.0` indicating how positive the song is. So it makes sense that the `energy` would be correlated with `loudness`, and `valence` with `danceability`.\n",
    "\n",
    "Now let's examine the distribution values for some of the features we mention above. In particular, let's look at the `top1k` subset, as we may have a better intuition for popular songs. For example, are the `top1k` songs more biased towards high `danceability`?\n",
    "\n",
    "We can use [`seaborn's distplot()`](http://seaborn.pydata.org/generated/seaborn.distplot.html) to plot a histogram for the `danceability` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(top1k['danceability']).set_title('Danceability distribution for top 1k songs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about `energy` and `acousticness`? From the API docs: acousticness is a score between 0.0 and 1.0 that indicates Spotify's confidence that the song is \"acoustic\", e.g., live performances would have a score close to or equal to 1.0. \n",
    "\n",
    "**Q5: Plot the distribution for energy and acousticness below.  What is the mean for each?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q5: YOUR CODE GOES HERE\n",
    "# (...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q6: Discuss with the person next to you: what have we learned so far on the relationship between top1k songs and these 3 features above? How do you think this may affect our `popularity` prediction task?**\n",
    "\n",
    "**In other words, what kind of song would you write if you were trying to make it to the top as a musician, just based on these 3 features alone?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: unsupervised ML with k-means clustering\n",
    "\n",
    "What if instead examining feature by feature we were to try and group songs by how similar they are in terms of these features?\n",
    "\n",
    "If you think of each song as a vector in a multidimensional space, where each dimension is a feature in the dataset, we can use co-sine similarity as a \"similarity\" metric to group together (or \"cluster\") songs that are similar.  And using the same idea, we can also \"profile\" each group, or \"cluster\".\n",
    "\n",
    "An unsupervised ML algorithm that can help us here is [K-means clustering](https://en.wikipedia.org/wiki/K-means_clustering). `K` stands for the number of clusters (or groups) you want to have. The term `mean` refers to the method you use to assign each element in the dataset to a target cluster: by choosing the nearest mean (also known as the \"centroid\" element, representing the cluster) to that element.\n",
    "\n",
    "Specifically, for our case, given a set of songs where each song is a `d-`dimensional real vector, `k-`means clustering aims to partition the set of songs into `k` clusters so as to minimize the within-cluster sum of squares (aka intra-cluster variance).\n",
    "\n",
    "At this point we don't know yet how many clusters we'll need, but let's give it a try using some of the features we've seen so far, plus other characteristics such as `speechiness` (i.e., amount of vocals in the song), and `instrumentalness`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We exclude non-numerical features, and those that are too fine-grained, such as \"key\", or too coarse-grained,\n",
    "# such as \"mode\".\n",
    "df_numerical = top1k.select_dtypes(exclude=[np.object])\n",
    "cluster_features = ['acousticness', 'danceability', 'energy', 'speechiness', 'instrumentalness']\n",
    "df_clusters = df_numerical[cluster_features]\n",
    "\n",
    "# numpy array with features\n",
    "X = np.array(df_clusters)\n",
    "\n",
    "# Q7: Do we need this scaling code?\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X)\n",
    "X = scaler.transform(X)\n",
    "\n",
    "# Hint: does this help you decide?\n",
    "#top1k[cluster_features].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many clusters should we use? If we pick too few, we get too many songs that don't quite \"fit in\" with that cluster (i.e., intra-cluster variance, or error, is too large). If we pick too many, error is minimized, but our clusters are too small.\n",
    "\n",
    "To figure out the \"sweet spot\", we can plot how the error decays as a function of the number of clusters we use.  This will give us a \"elbow\" shaped curve, and we can use the point at which the curve turns \"smooth\" as our number of clusters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dist = []\n",
    "for k in range(1, 10):\n",
    "    km = KMeans(n_clusters=k, init='k-means++', random_state=1337)\n",
    "    km = km.fit(X)\n",
    "    dist.append(km.inertia_)\n",
    "    \n",
    "plt.plot(range(1, 10), dist, 'bx-')\n",
    "plt.xlabel('k')\n",
    "plt.ylabel('Variance')\n",
    "plt.title('Elbow Method for choosing k')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q7: We used co-sine distance between songs (represented as n-dimensional vectors) to cluster them a 5-D space (n=5 features).  A [`StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) standardizes features by removing the mean and scaling to unit variance.  Do we need to use that scaler in our features (see code 2 cells above)? Plot the elbow method above both with and without scaler to see the difference.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the \"Elbow Method\" curve above that at `k=4` our curve starts turning smooth. We can use either `k=4` or `k=5` as our number of clusters for k-means below. We'll go with `k=5`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# k-means clustering\n",
    "kmeans = KMeans(n_clusters=5, random_state=1337)\n",
    "kmeans.fit(X)\n",
    "y = kmeans.predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now try and visualize the clusters in a 2-D space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize the clusters\n",
    "centers = kmeans.cluster_centers_\n",
    "print('Features:\\n', cluster_features)\n",
    "print('Centroids:\\n', centers)\n",
    "\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='viridis')\n",
    "plt.scatter(centers[:, 0], centers[:, 1], c='red', s=200, alpha=0.5);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The centroid values for each dimension tells us some information on what the cluster profiles are, e.g., the last cluster is high on acousticness and low on both danceability and energy, so it's likely capturing live performances.\n",
    "\n",
    "But otherwise, as it turns out, directly visualizing the clusters in a 2-D space isn't so helpful for us at this point.\n",
    "\n",
    "First, we're using a 5-D dimensional space for clustering, so without doing something smart about how we reduce the number of dimensions when polotting, we're losing a lot of information on the visualization above.  Second, most songs in this dataset are highly biased towards one end of each of some features (e.g., we have a lot more danceable songs than we have non-danceable ones), so we'll have a lot of clusters fairly close to each other.\n",
    "\n",
    "We have a few options for dimensionality reduction that can help us here. The first one is Principal Component Analysis ([PCA](https://en.wikipedia.org/wiki/Principal_component_analysis)). Using PCA, we can tell what is the smallest set of features that contain the most information about our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=len(cluster_features))\n",
    "pca_components = pca.fit_transform(X)\n",
    "\n",
    "# plot the PCA variances\n",
    "pca_features = range(pca.n_components_)\n",
    "plt.bar(pca_features, pca.explained_variance_ratio_)\n",
    "plt.xlabel('Principal components')\n",
    "plt.ylabel('Explained variance (%)')\n",
    "plt.xticks(pca_features)\n",
    "\n",
    "# save pca components in a df\n",
    "pca_components = pd.DataFrame(pca_components, columns = ['PC0', 'PC1', 'PC2', 'PC3', 'PC4'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the plot above that after the second PC, the gain in information is reduced. That is, the first component contributes almost 30% of the information, and the second one contributes over 25%. So if we use just the first 2 PCs, we already have close to 60% of the data being preserved. If we kept the first 3 PCs, we get closer to 80%.\n",
    "\n",
    "Let's try and plot the k-means clusters using PCA instead, and see if that makes it easier to visualize the clusters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# k-means clustering on first 2 PCA components\n",
    "X_pca = pca_components\n",
    "kmeans_pca = KMeans(n_clusters=5, random_state=31337)\n",
    "kmeans_pca.fit(X_pca)\n",
    "y_pca = kmeans_pca.predict(X_pca)\n",
    "\n",
    "# visualize the clusters\n",
    "centers_pca = kmeans_pca.cluster_centers_\n",
    "plt.scatter(X_pca['PC0'], X_pca['PC1'], c=y_pca, s=50, cmap='viridis')\n",
    "plt.scatter(centers_pca[:, 0], centers_pca[:, 1], c='red', s=200, alpha=0.5);\n",
    "plt.xlabel('Principal Component 1')\n",
    "plt.ylabel('Principal Component 2')\n",
    "\n",
    "# if you're curious, inspect the min and max values for the PC dimensions\n",
    "# X_pca[['PC0', 'PC1']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see above, using k-means clustering after PCA, and plotting our clusters with the first 2 PCs as dimensions, helps a bit. Specifically, intra-cluster variance is smaller, and we can see that the centroids are not that far apart from each other (likely indicating that songs in the top1k are not that different from each other). But we can still improve this visualization.\n",
    "\n",
    "A more powerful technique for dimensionality reduction is t-Distributed Stochastic Neighbor Embedding ([t-SNE](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding)). We'll leave for you to **try on your own** by following this [t-SNE vs PCA tutorial](https://www.datacamp.com/community/tutorials/introduction-t-sne), which has several example comparisons between the results you get with PCA vs t-SNE for visualization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: supervised ML for prediction tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this last part, we'll focus on two prediction tasks.\n",
    "\n",
    "In the first task, we'll try and predict genre for each song. We'll first try and see if we can do that on the `top1k` songs, and then with the entire dataset.\n",
    "\n",
    "In the second task, we'll try to predict song popularity, but using a larger dataset with features similar to those of the dataset above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) genre prediction\n",
    "\n",
    "As mentioned above, the dataset we've been using so far has been augmented with genre information (stored in the `genre` column). We'll use values in that column as labels for training a supervised ML algorithm to predict song genre.\n",
    "\n",
    "And unlike in k-means clustering, here we'll use all features as input. We start by scaling our dataset again, follow that by splitting into our data into training and test datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use our entire set of numerical features, instead of just the 5 we used for k-means clustering:\n",
    "df_numerical = top1k.select_dtypes(exclude=[np.object])\n",
    "X = np.array(df_numerical)\n",
    "\n",
    "# scaling won't make a difference for classifiers, but we leave it here in case you want to try other models\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X)\n",
    "X = scaler.transform(X)\n",
    "\n",
    "# the features and the label we'll use for training our supervised algo:\n",
    "features = X\n",
    "labels = top1k['genre']\n",
    "\n",
    "# split our data into training and test\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(features, labels, random_state=31337)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then train our [decision tree classifier](https://scikit-learn.org/stable/modules/tree.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train a decision tree on it\n",
    "genre_tree = DecisionTreeClassifier(random_state=31337)\n",
    "genre_tree.fit(train_features, train_labels)\n",
    "\n",
    "# predict labels on test data\n",
    "pred_labels_tree = genre_tree.predict(test_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how well the tree did on our genre prediction task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_results = classification_report(test_labels, pred_labels_tree)\n",
    "print('Decision Tree Results:\\n',  tree_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's pretty bad: our weighted average precision is as bad as random.  This is probably because this dataset doesn't have enough samples of each genre (there are only 1000 songs in total). As expected, we have a bias only towards genres in the top 1000.\n",
    "\n",
    "**Q8: Did we pick the wrong model? Try a [RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) instead below, and see if we can get better results:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q8: YOUR CODE GOES HERE.\n",
    "# (...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another problem here is that not only do we have a small dataset, but we also have a large number of genres relative to the size of the dataset.\n",
    "\n",
    "Let's try and see if we can do better with more samples by using the larger dataset, and also reducing it down to prediction of only 3 mainstream genres:\n",
    "\n",
    "**Q9: How many samples do we have per genre on the larger dataset (`df` pd dataframe)?** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q9: YOUR CODE GOES HERE.\n",
    "# df.(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This may be enough, but as we can see the number of samples is not exactly balanced for each genre, so we may still get some bias.  Also, it's a fair amount of genres, so let's pick 3 common ones and see how we can do at that task. We'll go with \"Hip-Hop\", \"Rock\", and \"Jazz\", which should be different enough from each other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hip_hop = df.loc[df['genre'] == 'Hip-Hop']\n",
    "rock = df.loc[df['genre'] == 'Rock']\n",
    "jazz = df.loc[df['genre'] == 'Jazz']\n",
    "\n",
    "# make it equal number of songs for each\n",
    "hip_hop = hip_hop.sample(n=len(rock), random_state=31337)\n",
    "jazz = jazz.sample(n=len(rock), random_state=31337)\n",
    "df = pd.concat([rock, hip_hop, jazz])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And train our forest on this larger dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select only numerical features\n",
    "df_numerical = df.select_dtypes(exclude=[np.object])\n",
    "X = np.array(df_numerical)\n",
    "\n",
    "# scaling won't make a difference for classifiers, but we leave it here in case you want to try other models\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X)\n",
    "X = scaler.transform(X)\n",
    "\n",
    "# features and label\n",
    "features = X\n",
    "labels = df['genre']\n",
    "\n",
    "# split our data into training and test\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(features, labels, random_state=31337)\n",
    "\n",
    "# train a RFC on it\n",
    "genre_forest = RandomForestClassifier(random_state=31337)\n",
    "genre_forest.fit(train_features, train_labels)\n",
    "\n",
    "# predict labels on test data\n",
    "pred_labels_forest = genre_forest.predict(test_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how we did this time around:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_forest_results = classification_report(test_labels, pred_labels_forest)\n",
    "print('RFC Results:\\n',  genre_forest_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time around we did a lot better: 92% precision for Jazz, and an average precision of 88% for all 3 genres.\n",
    "\n",
    "**Q10: To try on your own: train a [`DecisionTreeClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) with different values of `max_depth` and use `tree.plot()` to [`plot it`](https://scikit-learn.org/stable/modules/tree.html).**\n",
    "\n",
    "**What are some pros/cons of using a decision tree vs a random forest classifier?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q10: YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) popularity prediction\n",
    "\n",
    "Let's use what we learned above to try out another prediction task: song popularity. First, we'll do some feature engineering:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make sure we start with a clean slate\n",
    "df = pd.read_csv('data/spotify_songs.csv')\n",
    "\n",
    "# hot-encode \"mode\" categorical column\n",
    "df.loc[df['mode'] == 'Major', 'mode'] = 1\n",
    "df.loc[df['mode'] == 'Minor', 'mode'] = 0\n",
    "\n",
    "# hot-encode \"key\" categorical columns, i.e.,\n",
    "# A -> 0, A# -> 1, etc.\n",
    "keys = df['key'].unique()\n",
    "for i in range(len(keys)):\n",
    "    df.loc[df['key'] == keys[i], 'key'] = i\n",
    "\n",
    "# ditto for time signatures\n",
    "signatures = df['time_signature'].unique()\n",
    "for i in range(len(signatures)):\n",
    "    df.loc[df['time_signature'] == signatures[i], 'time_signature'] = i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll pose our popularity prediction task as a binary decision between \"popular\" and \"unpopular\". Our data is exponentially distributed in terms of popularity (**how can we check that?**), so we'll choose a cutoff at the 75th percentile to split our songs at:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert popularity from [1, 100] to \"popular\" or \"not popular\" by splitting below and above the 75th %\n",
    "pop_75th = np.percentile(df['popularity'], 75)\n",
    "df.loc[df['popularity'] < pop_75th, 'popularity'] = 0 \n",
    "df.loc[df['popularity'] >= pop_75th, 'popularity'] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And as we've done before, split into train and test datasets. We'll use these to train a [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select only numerical features\n",
    "df_numerical = df.select_dtypes(exclude=[np.object])\n",
    "\n",
    "# remove our label from features\n",
    "df_features = df_numerical.drop(['popularity'], axis=1)\n",
    "\n",
    "# standard scale\n",
    "X = np.array(df_features)\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X)\n",
    "X = scaler.transform(X)\n",
    "\n",
    "# features and label\n",
    "features = X\n",
    "labels = df['popularity']\n",
    "\n",
    "# split our data into training and test, reserving 20% of rows for test set\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(features,\n",
    "                                                                            labels, test_size=0.2, random_state=31337)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's train a few other ML algos (some more powerful) on these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train RFC\n",
    "rfc = RandomForestClassifier(random_state=31337)\n",
    "rfc.fit(train_features, train_labels)\n",
    "\n",
    "# predict labels on test data\n",
    "pred_labels_rfc = rfc.predict(test_features)\n",
    "accuracy_rfc = accuracy_score(test_labels, pred_labels_rfc)\n",
    "print('RFC Accuracy: ' + str(accuracy_rfc))\n",
    "\n",
    "auc_rfc = roc_auc_score(test_labels, pred_labels_rfc)\n",
    "print('RFC AUC: ' + str(auc_rfc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad!\n",
    "\n",
    "Now go forth and **try out for yourself with other models, and compare their performance**. In particular, we recommend you try out `xgboost`'s [`XGBClassifier`](https://xgboost.readthedocs.io/en/latest/python/index.html), [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html), and [LinearSVC](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html) and see how those do on the same task. Do they beat the RFC above?\n",
    "\n",
    "You can also try out regression on the `popularity` ranks, instead of the binary decision classification problem we formulated above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q11: YOUR CODE GOES HERE\n",
    "# Try out xgboost's XGBClassifier as well as try and formulate the problem as a regression on popularity ranks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}