{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 5: Visualization\n", "\n", "Visualizations help you to explore a dataset, discover anomalies or errors, find areas of interest, explain ideas convincingly to others, and check your assumptions. In this lab we will introduce two popular visualization tools commonly used in python. \n", "\n", "In the in-class portion of this lab we will use some of the FEC data from lab 1. Since the point of this lab is visualization and not running complex queries over data, we will provide code for the transformations in this lab.\n", "\n", "One of the most popular plotting libraries today is [Matplotlib](https://matplotlib.org/). According to the documentation, \"Matplotlib tries to make easy things easy and hard things possible.\" For the most basic plots, line graphs, single bar plot's etc., matplotlib has functions for just these things.\n", "\n", "As plots get more complex, many data scientists use [Seaborn](https://seaborn.pydata.org/), a wrapper around matplotlib that improves on its theme, and has convenience functions for more complex plots that tend to be common. You can see a ton of examples in Seaborn's [example gallery](https://seaborn.pydata.org/examples/index.html).\n", "\n", "With Matplotlib and Seaborn you often have to manually move labels, adjust axes, add data labels, etc. An alternative is declarative visualization. Akin to the difference between imperative approaches like Pandas vs declarative approaches like SQL for manipulating tabular data, declarative visualizations allow users to specify what visualization they want over their data by describing the connection between data and visual encodings.\n", "\n", "In this lab we will use a python declarative visualization library, [Altair](https://altair-viz.github.io/), itself built atop the popular declarative visualization tool [Vega](https://vega.github.io/vega/), to demonstrate how to get some effective visualizations in relatively few lines of code. \n", "\n", "While it is difficult to be comprehensive in a short lecture, the idea behind this excercize is to give an introduction to the flavor of some of these tools so that you can try out things on your own later. As usual, useful resource include the documentation and StackOverflow." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: Python Plotting Libraries (Matplotlib, Seaborn, & Pandas)\n", "\n", "In the first part of this lab, we will load our data and start creating some simple visualizations, much like the questions we were answering in Lab 1. While we can't explore all of the things you can do with these libraries, we will try to give you a flavor of what is possible. \n", "\n", "We start by importing packages and loading our datasets." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "cands = pd.read_csv(\"data/cand_summary.txt\", delimiter=\"|\")\n", "#Removing US territories\n", "cands = cands[~cands.CAND_OFFICE_ST.isin([\"AS\", \"GU\", \"MP\", \"US\", \"DC\", \"MH\", \"PR\", \"VI\"])]\n", "#Add a column for CAND_OFFICE ('P', 'H' or 'S', for President, House, and Senate respectively)\n", "cands[\"CAND_OFFICE\"] = cands.CAND_ID.str[:1]\n", "pacs = pd.read_csv(\"data/pac_summary.txt\", delimiter=\"|\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ``cands`` table is the same as the cand_summary table from lab 1 but includes an additional column for the election cycle year. E.g 2015-2016 are represented in the dataset by ELECTION_CYCLE_YEAR of 2016. The data goes back to 1980. Similarly, the ``pacs`` table contains summary information of politiacal action committees dating to 2000." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cands.dtypes" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pacs.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can get the 2018 house candidates as follows, sorted by total fundraising receipts." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "house_2018 = cands[(cands.ELECTION_CYCLE_YR == 2018) &\n", " (cands.CAND_OFFICE == \"H\")] \\\n", " .sort_values(\"TTL_RECEIPTS\", ascending=False)\n", "house_2018.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's start making some plots! We'll start by looking at the ratio of individual contributions to total receipts for all 2018 House candidates using a scatterplot." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.scatter(house_2018.TTL_RECEIPTS, house_2018.TTL_INDIV_CONTRIB)\n", "\n", "#Increase the size of the figure for visibility\n", "plt.gcf().set_size_inches(10,6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's worth thinking about what is going on here. Matplotlib has a default **figure** that starts with one **axes** object. When using the ``plt.`` functions we edit the default axis or the default figure (depending on what property we are manipulating). Alternatively we can fetch the axis or figure object and operate on that directly. This is useful if you want to create multiple plots in the same figure. An example of multiple plots on the same figure is [here](https://matplotlib.org/devdocs/gallery/subplots_axes_and_figures/subplots_demo.html).\n", "\n", "There are a couple of issues with the figure above: It has no title, no axis labels, and our area of interest is pretty hard to see. Let's fix that." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Get the current axes object\n", "ax = plt.gca()\n", "\n", "#Create Scatterplot\n", "ax.scatter(house_2018.TTL_RECEIPTS, house_2018.TTL_INDIV_CONTRIB)\n", "\n", "#Set the title\n", "ax.set_title(\"2016 House Individual Contributions vs Total Fundraising\")\n", "\n", "#Set the axis labels\n", "ax.set_xlabel(\"TTL_RECEIPTS ($)\")\n", "ax.set_ylabel(\"TTL_INDIV_CONTRIB ($)\")\n", "\n", "#Adjust the axis limits \n", "ax.set_xlim((0, 20000000))\n", "ax.set_ylim((0, 20000000))\n", "\n", "#Increase figure size on the current figure\n", "plt.gcf().set_size_inches(10,8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is looking better, but who are these outliers who receive tons of money but almost none by individuals? \n", "\n", "The first thing we can do is add multiple series on the same axis to see which party these house candidates belong to. We'll split our data into two sets, one for Republicans and one for Democrats. We will then add both scatterplots to the same axes object and plot them over each other. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Get the current figure and axes\n", "ax = plt.gca()\n", "\n", "#split rep, dem and third party\n", "rep_2018 = house_2018[house_2018.PTY_AFFILIATION ==\"REP\"]\n", "dem_2018 = house_2018[house_2018.PTY_AFFILIATION ==\"DEM\"]\n", "\n", "# Note that we are adding a label here, we are not getting this directly from the data\n", "# We also manually set the color of each.\n", "ax.scatter(rep_2018.TTL_RECEIPTS, rep_2018.TTL_INDIV_CONTRIB, label=\"REP\", color=\"RED\")\n", "ax.scatter(dem_2018.TTL_RECEIPTS, dem_2018.TTL_INDIV_CONTRIB, label=\"DEM\", color=\"BLUE\")\n", "\n", "# We can also add a line y=x to show when all contributions come from individuals. \n", "# So we can see how far some candidates deviate from this ideal, \n", "# the \"--\" designates a dotted line\n", "ax.plot([0, 20000000], [0, 20000000], \"--\")\n", "\n", "\n", "#Set the title\n", "ax.set_title(\"2016 House Individual Contributions vs Total Fundraising\")\n", "\n", "#Set the axis labels\n", "ax.set_xlabel(\"TTL_RECEIPTS ($)\")\n", "ax.set_ylabel(\"TTL_INDIV_CONTRIB ($)\")\n", "\n", "#Add a legend\n", "plt.gcf().legend()\n", "\n", "#Adjust the axis limits \n", "ax.set_xlim((0, 20000000))\n", "ax.set_ylim((0, 20000000))\n", "\n", "#Increase figure size\n", "plt.gcf().set_size_inches(10,8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can see that there are a number of politicians raising a ton of money with almost none of it coming from indivudal contributers. But who exactly are those outliers? One thing we could do is to add data labels for some of these points (See the example [here](https://matplotlib.org/3.1.1/tutorials/text/annotations.html)). We'll come back to this later.\n", "\n", "Even with these simple plots this is starting to get pretty verbose. Matplotlib tends to be a manual process in this way. However, One advantage is the ability to do nearly anything you want. \n", "\n", "Seaborn is a wrapper around matplotlib that tends to be a bit easier to use and more easily produces cleaner plots without much manipulation. In the next few steps we'll look at how to show how the total amount of money raised changes each election cycle. Fortunately this data goes back to 1980!\n", "\n", "But first we'll revisit the same plot, to see how we would do the same thing in Seaborn.\n", "\n", "Note that we can just pass the data to Seaborn and select which columns we want, rather than selecting out columns or datasets ahead of time. The axis labels are taken care of automatically.\n", "\n", "One nice thing about Seaborn is that it's a wrapper around Matplotlib so once you produce a plot, you can still manipulate it with the same standard set of Matplotlib functions. Below we adjust the axis limits." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Set the Seaborn default theme on Matmplotlib\n", "sns.set()\n", "\n", "#Note that we use by using hue, Seaborn automatically splits the data into separate series and colors them accordingly.\n", "sns.relplot(x=\"TTL_RECEIPTS\", y=\"TTL_INDIV_CONTRIB\", hue=\"PTY_AFFILIATION\", \\\n", " data=house_2018[house_2018.PTY_AFFILIATION.isin([\"REP\",\"DEM\"])])\n", "\n", "#Adjust the axis limits using Matplotlib\n", "plt.xlim((0, 20000000))\n", "plt.ylim((0, 20000000))\n", "\n", "#Increase figure size\n", "plt.gcf().set_size_inches(10,8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Instead of plotting Democrats and Republicans atop one another, we can also use a \"Factor Grid\" to split out different categories into different plots. Let's look at senators this time.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "major_party_senate_2018 = cands[(cands.ELECTION_CYCLE_YR == 2018) & \n", " (cands.CAND_OFFICE == \"S\") & \n", " (cands.PTY_AFFILIATION.isin([\"REP\", \"DEM\"]))]\n", "fgrid = sns.FacetGrid(major_party_senate_2018, col=\"PTY_AFFILIATION\")\n", "\n", "# Since we have already declared our facet to be \"PTY_AFFILIATION\"\n", "# Seaborn knows to create two plots side by side\n", "fgrid = fgrid.map(plt.scatter, \"TTL_RECEIPTS\", \"TTL_INDIV_CONTRIB\")\n", "\n", "#Adjust the axis limits \n", "plt.xlim((0, 20000000))\n", "plt.ylim((0, 20000000))\n", "\n", "#Increase figure size\n", "plt.gcf().set_size_inches(10,8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok. Now let's see how campaign contributions change over time using Seaborn. The first thing we have to do is manipulate our data to get total campaign contributions for the last 40 years." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "house_cands = cands\n", "contrib_sum = house_cands.groupby(\"ELECTION_CYCLE_YR\").sum()\n", "contrib_sum" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# This reset index removes the index column from the grouped data for ease of use in Seaborn\n", "contrib_sum = cands.groupby(\"ELECTION_CYCLE_YR\").sum().reset_index()\n", "\n", "sns.barplot(x=\"ELECTION_CYCLE_YR\", y=\"TTL_RECEIPTS\", data=contrib_sum)\n", "plt.gcf().set_size_inches(10,8)\n", "\n", "#Rotate the labels on the x axis for readability\n", "plt.xticks(rotation=45)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see the rapid increase in spending on elections over the past 40 years. Although the US population has increased by 50% and the inflation rate is about 3x since 1980, the spending on elections far outpaces this.\n", "\n", "Next, create a grouped bar plot using Seaborn that compares Democrat and Republican spending on **House** Campaigns since 1980. How many years did Republicans spend more than Democrats? (You may start by copying the code for the above plot)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create your plot here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we are tracking data over time, what might be a different plot that could be a useful way to view this data?\n", "\n", "**Clicker Question**\n", "\n", "Now change the above plot to instead be of the type we just decided." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create your plot here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Political action committees (PACs) are organizations that raise money to support political campaigns or political ideas. These include party committees (like the Democratic National Committee), committees organized around ideas or particular issues, or special interest groups (like the American Medical Association).\n", "\n", "Say we are now interested not just in the total amount of money raised during an election cycle, but on where that money comes from over time? What about distributed across both candidates campaigns as well as different types of political action committees (PACs).\n", "\n", "\n", "What might be a good way of visualizing this?\n", "\n", "**Clicker Question**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at how we would accomplish this in Matplotlib/Seaborn. The first thing we have to do is manipulate our data so that it contains what we want. Remember that we want to separate out spending on house, presidential, and Senate elections, as well as different spending on political action committees (PACs) per year. Unfortunately the PAC data only goes back to the 2000 election cycle, so we'll settle for looking at the last 20 years instead of the last 40.\n", "\n", "In this case we will show how to use the plotting convenience functions of pandas, that internally use Matplotlib.\n", "\n", "Below we combine the ``cands`` and ``pacs`` dataset. For visual clarity, we'll look at only the top 3 types of PACS. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Sum up and get only the fields we are interested in\n", "election_spending = cands[cands.ELECTION_CYCLE_YR >= 2000]. \\\n", " groupby([\"ELECTION_CYCLE_YR\", \"CAND_OFFICE\"])\\\n", " .sum().reset_index()\n", "\n", "#We will change the\n", "election_spending = election_spending[[\"ELECTION_CYCLE_YR\", \"CAND_OFFICE\", \"TTL_RECEIPTS\"]].rename({\"CAND_OFFICE\":\"SPENDING_TYPE\"}, axis=\"columns\")\n", "\n", "\n", "pac_spending = pacs.groupby([\"ELECTION_CYCLE_YR\", \"CMTE_TP\"]).sum().reset_index()\n", "\n", "#Get the top 3 Pacs\n", "top_pacs = pac_spending.groupby(\"CMTE_TP\").sum().sort_values(\"TTL_RECEIPTS\", ascending=False).index[:3]\n", "pac_spending = pac_spending[pac_spending.CMTE_TP.isin(top_pacs)]\n", "pac_spending = pac_spending[[\"ELECTION_CYCLE_YR\", \"CMTE_TP\", \"TTL_RECEIPTS\"]].rename({\"CMTE_TP\":\"SPENDING_TYPE\"}, axis=\"columns\")\n", "\n", "total_spending = pd.concat([election_spending, pac_spending])\n", "\n", "\n", "#Rename to meaningful spending types\n", "spending_type_map = {\"H\":\"House\", \"S\":\"Senate\", \"P\":\"President\", \"Q\":\"Standard PAC\", \"O\":\"Super PAC\", \"Y\":\"Party PAC\"}\n", "total_spending.SPENDING_TYPE = total_spending.SPENDING_TYPE.apply(lambda x: spending_type_map[x])\n", "total_spending" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have our data in the proper format, we will create our plot by first creating a pivot table.\n", "\n", "A pivot table contains aggregate data for a dataset. Each row and column represent a category. Each cell contains an aggregate for data points in each row. For instance, beelow we create rows for each ``ELECTION_CYCLE_YEAR`` and columns for each ``SPENDING_TYPE``. The cell corresponding to ``2016`` and ``House`` sums up all values where ``ELECTION_CYCLE_YEAR`` is 2016 and ``SPENDING_TYPE`` is ``House``\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#This will create rows for election cycle years, and columns for each distinct spending type.\n", "total_spending_pivot = total_spending.pivot(index=\"ELECTION_CYCLE_YR\", columns=\"SPENDING_TYPE\")\n", "\n", "#Reorder columns for plotting prettyness\n", "column_order = ['Party PAC', 'Standard PAC', \"Super PAC\", \"House\", \"Senate\", \"President\"]\n", "total_spending_pivot = total_spending_pivot.reindex([(\"TTL_RECEIPTS\", x) for x in column_order], axis = 1)\n", "total_spending_pivot" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "total_spending_pivot.plot.bar(y = \"TTL_RECEIPTS\", stacked=True)\n", "plt.gcf().set_size_inches(15,10)\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Declarative Visualizations (Altair & Vega-Lite)\n", "\n", "In the first part of the reversed lecture we looked at some of the most popular plotting libraries in python. You will often find plots made with these libraries in academic papers and online.\n", "\n", "In the preceding section, you may have felt that a lot of this is hardcoded. In each case we had to massage our data into a form that was close to what we wanted to plot in pandas, before passing it to the plotting library. We had to manually adjust axes, and aggregate ahead of time.\n", "\n", "An alternative to this approach is the idea of **Declarative visualization**. The difference in style is expressed in the Altair [documentation](https://altair-viz.github.io/getting_started/overview.html). \n", "\n", " The key idea is that you are declaring links between data columns and visual encoding channels, such as the x-axis, y-axis, color, etc. The rest of the plot details are handled automatically. Building on this declarative plotting idea, a surprising range of simple to sophisticated plots and visualizations can be created using a relatively concise grammar.\"\n", " \n", "We do this by giving input data, then declaring a set of transformations to get to visual encoding channels.\n", "\n", "Altair is a python wrapper around Vega-lite. A declarative visualization grammar. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import altair as alt\n", "from vega_datasets import data\n", "from altair import datum\n", "alt.renderers.enable('notebook')\n", "alt.data_transformers.disable_max_rows()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll start by revisiting the scatterplot example from above, instead looking at Senate candidates. Because of the way that Altair works, it is faster to render the fewer points we have. So we'll give it Senate candidates only to reduce the size of our dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "senate_cands = cands[cands.CAND_OFFICE==\"S\"]\n", "alt.Chart(senate_cands).mark_point(). \\\n", " encode(x=\"TTL_RECEIPTS:Q\",\n", " y=\"TTL_INDIV_CONTRIB:Q\",\n", " color=\"PTY_AFFILIATION:N\"). \\\n", "transform_filter(\n", " ((datum.PTY_AFFILIATION == \"REP\") | (datum.PTY_AFFILIATION == \"DEM\")))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is a bunch going on here so let's break it down a step at a time.\n", "\n", "```\n", "alt.Chart(cands).mark_point()\n", "```\n", "This line declares that we are creating a chart over some set of data, in our case the candidate dataset. We also specify the kind of mark we are going to make on our plot. In this case a \"point\", we might also want a bar, a circle, some area, etc. We will look at some other marks later.\n", "\n", "We then encode our data using visual channels. Altair can infer what data type to use (Recall Monday's lecture on visual data types: Quantitative, Ordinal, Nominal, and Temporal). As a reminder of the diference these see [this Altair Documentation](https://altair-viz.github.io/user_guide/encoding.html#encoding-data-types) part of the Altair documentation. Below we'll add one mapping from column to visual encoding at a time, and observe the impact." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "alt.Chart(senate_cands).mark_point(). \\\n", " encode()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we have not specified any encoding for our data, all our points are plotted atop each other, this is obviously not very useful." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "alt.Chart(senate_cands).mark_point(). \\\n", " encode(x=\"TTL_RECEIPTS:Q\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Q above tells Altair that this is a Quantitative value.\n", "\n", "Here we plot the total receipts along the x axis, note that labeling is taken care of automatically. We start to see that most Senate candidates are mostly clustered near the low end of total receipts with only a few reaching beyond the $20 million mark. One thing we might be interested in at this point is creating a histogram. Let's do that. We will create bins on total receipts. Then we will add a count (with log axis)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "alt.Chart(senate_cands).mark_point(). \\\n", " encode(alt.X(\"TTL_RECEIPTS:Q\", bin=True))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This by itself is not very useful without a y-axis. Let's add one for the count of these values on the y axis, with log scale, to create a histogram)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "alt.Chart(senate_cands).mark_point(). \\\n", " encode(alt.X(\"TTL_RECEIPTS:Q\", bin=True),\n", " alt.Y(\"count()\", scale=alt.Scale(type=\"log\")))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can see that the mark type is probably not appropriate for what we are trying to show, we'll switch from \"points\" to \"bars\" to get a histogram of our data. We can see that Most candidates raise between 0 and $10 million." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "alt.Chart(senate_cands).mark_bar(). \\\n", " encode(alt.X(\"TTL_RECEIPTS:Q\", bin=True),\n", " alt.Y(\"count()\", scale=alt.Scale(type=\"log\")))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll get back to the remaining visual encodings for our dataset now. We'll encode the total individual contributions along the y-axis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "alt.Chart(senate_cands).mark_point(). \\\n", " encode(x=\"TTL_RECEIPTS:Q\",\n", " y=\"TTL_INDIV_CONTRIB:Q\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now have all our Senate candidates plotted, comparing total receipts and total individual contributions. We will add back color to indicate party affiliation." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "alt.Chart(senate_cands).mark_point(). \\\n", " encode(x=\"TTL_RECEIPTS:Q\",\n", " y=\"TTL_INDIV_CONTRIB:Q\",\n", " color=\"PTY_AFFILIATION:N\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Woah, while that gave split the parties up by color, there are a bunch of small parties that are cluttering up our dataset.\n", "\n", "We can filter it down to get just Republicans and Democrats. While we could perform this manipulation in Pandas, here we will show how to add transformations to data in Altair." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "senate_cands = cands[cands.CAND_OFFICE==\"S\"]\n", "alt.Chart(senate_cands).mark_point(). \\\n", " encode(x=\"TTL_RECEIPTS:Q\",\n", " y=\"TTL_INDIV_CONTRIB:Q\",\n", " color=\"PTY_AFFILIATION:N\"). \\\n", "transform_filter(\n", " ((datum.PTY_AFFILIATION == \"REP\") | (datum.PTY_AFFILIATION == \"DEM\")))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also split it into two plots by party, by using a facet chart." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "senate_cands = cands[cands.CAND_OFFICE==\"S\"]\n", "alt.Chart(senate_cands).mark_point(). \\\n", " encode(x=\"TTL_RECEIPTS:Q\",\n", " y=\"TTL_INDIV_CONTRIB:Q\",\n", " color=\"PTY_AFFILIATION:N\"). \\\n", " facet(column=\"PTY_AFFILIATION:N\").\\\n", "transform_filter(\n", " ((datum.PTY_AFFILIATION == \"REP\") | (datum.PTY_AFFILIATION == \"DEM\")))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remember how we were trying to find just who some of these outliers were for house candidates? Altair makes this trivial with tooltips. Conceptually, we are adding another visual encoding channel to the output with the candidate name. Let's add a tooltip for the candidate and election cycle to see which candidates are outliers." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "senate_cands = cands[cands.CAND_OFFICE==\"S\"]\n", "alt.Chart(senate_cands).mark_point(). \\\n", " encode(x=\"TTL_RECEIPTS:Q\",\n", " y=\"TTL_INDIV_CONTRIB:Q\",\n", " color=\"PTY_AFFILIATION:N\",\n", " tooltip=[\"CAND_NAME\", \"ELECTION_CYCLE_YR\"]). \\\n", " facet(column=\"PTY_AFFILIATION:N\").\\\n", "transform_filter(\n", " ((datum.PTY_AFFILIATION == \"REP\") | (datum.PTY_AFFILIATION == \"DEM\")))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We've now explored a small subset of the things you can do with Altair. So here's a chance for you to play around.\n", "\n", "**Clicker Question**\n", "\n", "Let's say we want to create a plot that shows how much Republicans and Democrats each spent on house elections going back to 1980. What kind of plot might you create?\n", "\n", "Try other marks in the following cell to find one you think is appropriate for the data. See the set of marks in the [Altair documentation](https://altair-viz.github.io/user_guide/marks.html)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Play around with different marks for this dataset. Which ones make sense to you?\n", "\n", "house_cands = cands[(cands.CAND_OFFICE.isin([\"H\", \"S\"])) & (cands.PTY_AFFILIATION.isin([\"REP\", \"DEM\"]))]\n", "alt.Chart(house_cands).mark_point().encode(\n", " x=\"ELECTION_CYCLE_YR:O\",\n", " y=\"sum(TTL_RECEIPTS)\",\n", " color=\"PTY_AFFILIATION:N\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are a couple more cool things you can do with Altair/Vega. In particular, visualizations with maps and interactive visualizations.\n", "\n", "Let's say we want to see which states in the last three election cycles had Senate races that raised a disproportionate amount of money per capita. Senate elections are every 6 years so looking at the previous 6 years of data ensures that all Senate seats have had an election.\n", "\n", "We'll start by creating a Choropleth map. That is, a map that is shaded relative to some statistic. Here let's look at the senate candidates that raised the most money per capita." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# First we'll add our state population data to our table.\n", "# we can do this in Altair too, but for convenience we'll do it outside in pandas.\n", "dist_pop = pd.read_csv(\"data/dist_pop.txt\", delimiter=\"|\")\n", "\n", "# Add the state population to the senate table with a merge\n", "state_pop = dist_pop[[\"state\", \"population\"]].groupby(\"state\").sum().reset_index()\n", "senate_pop = pd.merge(left=cands[(cands.ELECTION_CYCLE_YR >= 2014) &\n", " (cands.ELECTION_CYCLE_YR <= 2018) &\n", " (cands.CAND_OFFICE==\"S\")],\n", " right=state_pop, left_on=\"CAND_OFFICE_ST\", right_on=\"state\")\n", "senate_pop" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll create our map using a built in dataset containing mappings of states to their geographies. Much of this code is adopted from Altair's [example gallery](https://altair-viz.github.io/gallery/index.html)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Fetch the state geograpy data\n", "state_geo = alt.topo_feature(data.us_10m.url, 'states')\n", "\n", "#state_geo only contains an opaque numeric state id. \n", "#We need to map our postal code to this state id, the following 3 lines do this\n", "\n", "#Mapping of postal code to the state id used in the geographic data\n", "states = {'AK': 'Alaska','AL': 'Alabama','AR': 'Arkansas','AS': 'American Samoa','AZ': 'Arizona','CA': 'California','CO': 'Colorado','CT': 'Connecticut','DC': 'District of Columbia','DE': 'Delaware','FL': 'Florida','GA': 'Georgia','GU': 'Guam','HI': 'Hawaii','IA': 'Iowa','ID': 'Idaho','IL': 'Illinois','IN': 'Indiana','KS': 'Kansas','KY': 'Kentucky','LA': 'Louisiana','MA': 'Massachusetts','MD': 'Maryland','ME': 'Maine','MI': 'Michigan','MN': 'Minnesota','MO': 'Missouri','MP': 'Northern Mariana Islands','MS': 'Mississippi','MT': 'Montana','NA': 'National','NC': 'North Carolina','ND': 'North Dakota','NE': 'Nebraska','NH': 'New Hampshire','NJ': 'New Jersey','NM': 'New Mexico','NV': 'Nevada','NY': 'New York','OH': 'Ohio','OK': 'Oklahoma','OR': 'Oregon','PA': 'Pennsylvania','PR': 'Puerto Rico','RI': 'Rhode Island','SC': 'South Carolina','SD': 'South Dakota','TN': 'Tennessee','TX': 'Texas','UT': 'Utah','VA': 'Virginia','VI': 'Virgin Islands','VT': 'Vermont','WA': 'Washington','WI': 'Wisconsin','WV': 'West Virginia','WY': 'Wyoming'}\n", "state_to_id = {\"Alabama\":\"1\",\"Alaska\":\"2\",\"Arizona\":\"4\",\"Arkansas\":\"5\",\"California\":\"6\",\"Colorado\":\"8\",\"Connecticut\":\"9\",\"Delaware\":\"10\",\"District of Columbia\":\"11\",\"Florida\":\"12\",\"Georgia\":\"13\",\"Hawaii\":\"15\",\"Idaho\":\"16\",\"Illinois\":\"17\",\"Indiana\":\"18\",\"Iowa\":\"19\",\"Kansas\":\"20\",\"Kentucky\":\"21\",\"Louisiana\":\"22\",\"Maine\":\"23\",\"Maryland\":\"24\",\"Massachusetts\":\"25\",\"Michigan\":\"26\",\"Minnesota\":\"27\",\"Mississippi\":\"28\",\"Missouri\":\"29\",\"Montana\":\"30\",\"Nebraska\":\"31\",\"Nevada\":\"32\",\"New Hampshire\":\"33\",\"New Jersey\":\"34\",\"New Mexico\":\"35\",\"New York\":\"36\",\"North Carolina\":\"37\",\"North Dakota\":\"38\",\"Ohio\":\"39\",\"Oklahoma\":\"40\",\"Oregon\":\"41\",\"Pennsylvania\":\"42\",\"Rhode Island\":\"44\",\"South Carolina\":\"45\",\"South Dakota\":\"46\",\"Tennessee\":\"47\",\"Texas\":\"48\",\"Utah\":\"49\",\"Vermont\":\"50\",\"Virginia\":\"51\",\"Washington\":\"53\",\"West Virginia\":\"54\",\"Wisconsin\":\"55\",\"Wyoming\":\"56\",\"Puerto Rico\":\"72\"}\n", "\n", "#Set the state name in our data instead of the postal code\n", "senate_pop[\"STATE_ID\"]=senate_pop.CAND_OFFICE_ST.apply(lambda x: state_to_id[states[x]])\n", "\n", "#Get the per capita receipts for each state\n", "senate_pop[\"PER_CAPITA_RECEIPTS\"]=senate_pop.TTL_RECEIPTS/senate_pop.population\n", "\n", "# Get the sum of per capita and total receipts in our senate data\n", "agg_senate_pop = senate_pop[[\"STATE_ID\", \"CAND_OFFICE_ST\", \"PER_CAPITA_RECEIPTS\", \"TTL_RECEIPTS\"]].\\\n", " groupby(\"STATE_ID\").agg({\"CAND_OFFICE_ST\":\"first\", \"PER_CAPITA_RECEIPTS\":\"sum\", \"TTL_RECEIPTS\":\"sum\"}) .reset_index()\n", "\n", "#Plots the map from the given dataset, and uses a map projection\n", "per_capita_spending = alt.Chart(state_geo).mark_geoshape().project(\n", " type='albersUsa'\n", "#looks up the state in our dataset\n", ").transform_lookup(\n", " lookup='id',\n", " # Fetches only the columns of interest\n", " from_=alt.LookupData(agg_senate_pop, 'STATE_ID', [\"PER_CAPITA_RECEIPTS\", \"CAND_OFFICE_ST\", \"TTL_RECEIPTS\"])\n", ").encode(color=\"PER_CAPITA_RECEIPTS:Q\", tooltip=[\"CAND_OFFICE_ST:N\",\"TTL_RECEIPTS:Q\"])\n", "per_capita_spending" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, Altair makes using interactions straightforward. We have already seen a bit of this with tooltips when hovering over states above. But you can also perform interactive filtering on datasets.\n", "\n", "Let's create a scatterplot of our Senate candidates, and then we will allow users to filter over different regions to see which parties have low ratios of individual contributions to total receipts. We create a selector for our data that will allow us to filter data points. We'll also zoom into our area of interest by setting the scale on the x and y axis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Create our interaction\n", "brush = alt.selection(type=\"interval\", resolve=\"global\")\n", "\n", "# Get only Republicans and Democrats\n", "base = alt.Chart(senate_pop[senate_pop.PTY_AFFILIATION.isin([\"REP\", \"DEM\"])])\n", "\n", "#Create the scatterplot with our selection\n", "points = base.mark_point(clip=True).encode(\n", " alt.X(\"TTL_RECEIPTS:Q\", scale=alt.Scale(domain=(1,20_000_000))),\n", " alt.Y(\"TTL_INDIV_CONTRIB:Q\", scale=alt.Scale(domain=(1,20_000_000))),\n", " color=alt.condition(brush, \"PTY_AFFILIATION:N\", alt.value(\"lightgray\"))).add_selection(brush)\n", "\n", "#Filter our bar chart by the selection\n", "count = base.mark_bar().encode(\n", " x=\"PTY_AFFILIATION:N\",\n", " color=\"PTY_AFFILIATION:N\",\n", " y=\"sum(TTL_RECEIPTS)\").transform_filter(brush)\n", "\n", "#This concatonates plots side by side, You could place them atop one another using &\n", "points | count" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can filter the data in the right bar plot by dragging our mouse over points on the scatter plot. You can try it above.\n", "\n", "Of course there is a ton of functionality of these visualization libraries that we did not have the time to get into. For the take home portion of the lab, you will create your own visualization over a dataset of your choice. This will give you the chance to explore these libraries on your own and try new things.\n", "\n", "We're looking forward to seeing the visualizations you create!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }