{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "upy_H2JTdwyf"
},
"source": [
"# CS 124 Programming Assignment 2: Logistic Regression and Text Classification (`Winter 2026`)\n",
"\n",
"Now that you are familiar with `Jupyter Notebooks` and have had some experience working with text data, it's time to begin investigating some real `Natural Language Processing` (`NLP`) tasks, specifically those requiring text classification.\n",
"\n",
"Text classification tasks come up in a huge range of contexts; quite often we may be provided with text data in some form and be interested in labeling or categorizing it in some way.\n",
"\n",
"In this assignment, you will be working on classifying messages for disaster aid using a `Logistic Regression` (`LR`) classifier.\n",
"Victims of natural disasters have urgent needs for food, water, shelter, medicine, and other forms of aid.\n",
"These needs are often communicated through text messages, social media posts, and local newspapers. Because of their ability to automatically process large amounts of text, `NLP` techniques can play an important role in ensuring that people receive potentially life-saving aid.\n",
"Our goal will be to perform text classification on messages sent in the aftermath of natural disasters.\n",
"\n",
"We will be utilizing a `Python` module called `NumPy` in this and the next assignment. We recommend that you review the NumPy tutorial on the code website to get more comfortable with it.\n",
"\n",
"**You are encouraged to work with a partner!** We want the assignments in `CS 124` to bring you joy.\n",
"One way to ensure this is to work with a partner!\n",
"You are free to work with one other partner in our assignments.\n",
"If you choose to work with a partner, we ask that each partner work on each part of the assignment in jointly instead of splitting parts.\n",
"The partnership decision is independent for each assignment, so you can choose to work alone, work with the same partner or work with a different partner in the future assignments, which is a good way to meet your fellow classmates!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i0T5kYg_3Y7S"
},
"source": [
"\n",
"## Contents"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-DXv8PA2-hU5"
},
"source": [
"Listed below are the contents of the assignment. In the `Data Exploration` \n",
"section, you will look into the Disaster Aid Classification (`Triage`) dataset \n",
"we will use in the assignment.\n",
"In the `Logistic Regression` section, you will implement a `Logistic Regression`\n",
"classifier to determine whether a message sent in the aftermath of a natural \n",
"disaster is about aid.\n",
"In the `Tips` section, we share some useful tips for your implementation.\n",
"In the `Evaluation on the Triage Dataset` section, you will evaluate your \n",
"`Logistic Regression` classifier on the `Triage` dataset.\n",
"Please read through all of this notebook before you start working through the assignment.\n",
"In the `Ethics, Fairness, and Model Fragility` section, you will examine how translation gaps and semantic ambiguities can affect model performance, and reflect on fairness considerations in real-world deployment.\n",
"In the last section, we will briefly touch on the `hyperparameters` you can adjust for your `Logistic Regression` classifier. \n",
"Note that the links may not work on `Google Colab`.\n",
"\n",
"* [`Part 1. Data Exploration`](#data_exploration)\n",
"* [`Part 2. Logistic Regression`](#logistic_regression)\n",
"* [`Part 3. Evaluation on the Triage Dataset`](#evaluation_triage)\n",
"* [`Part 4. Ethics, Fairness, and Model Fragility`](#ethics)\n",
"* [`Part 5. Note on Hyperparameters`](#hyperparameters)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_pBXnsqTqHu6"
},
"source": [
"\n",
"## Roadmap\n",
"\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rwzw4SXt-tJy"
},
"source": [
"As an overview, there are `5` methods you need to implement in this assignment:\n",
"* In `Part 2.1 Sigmoid`: **`sigmoid()`** function\n",
"* In `Part 2.2 Logistic Loss`: **`logistic_loss()`** function\n",
"* In `Part 2.3 Gradient Descent`: **`gradient_descent()`** function\n",
"* In `Part 2.4 Logistic Regression Classifier`: **`__init__()`**, **`train()`**, **`classify()`**, and **`get_weights()`** methods of the **[`LogisticRegressionClassifier(Classifier)`](#logistic_regression)** class\n",
"* In `Part 3.2 Sanity Check`: **`reflection_response_3_2()`** function, which is a short answer question\n",
" \n",
"Here is how your implementation will be evaluated:\n",
"* In `Part 3. Evaluation on the Triage Dataset`, your implementation will be evaluated with respect to the `triage` dataset.\n",
" * In `Part 3.1 Accuracy`, you will check the accuracy of your `Logistic Regression` model both on the `train` and `dev` sets, each with and without stop words filtered via the vectorizer's parameter. Hence, there will be a total of `4` accuracy tests in this section. We recommend going back to your implementation if the accuracies you get in this section are far from what we have provided.\n",
" In addition to what you will see in this section, our autograder will run `2` more tests on the same dataset, this time using a `test` set, with or without stop words. **Note: You will not need to write any code to filter stop words.** \n",
" To see the autograder tests, you can submit your notebook on `Gradescope`.\n",
" * In `Part 3.2 Sanity Check`, we will check the answer you include in the `reflection_response_3_2()` function. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e4gZzwRnPpWw"
},
"source": [
"\n",
"## Submitting"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HuKzBe8IYqBT"
},
"source": [
"**Submit your empty assignment to Gradescope now to see the autograder output!**\n",
"You will submit your assignment via [`Gradescope`](www.gradescope.com), where we have an autograder set up.\n",
"You can submit your assignment any number of times before the deadline.\n",
"As a general rule of thumb, we recommend submitting early and often in any `Computer Science` class if you have the option, to prevent any last minute errors with autograders.\n",
"Submitting early also helps gauge how you are doing on the visible test cases of the autograder and gives you a chance to fix your submission accordingly.\n",
"In fact, start with submitting your assignment now (even if you haven't coded anything), so that you are familiar with the submission process and know what kind of autograder feedback is available to you.\n",
"You can re-submit as you make progress.\n",
"Don't forget to update your submission with your final version once you are done!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Y15qB5bxwR3N"
},
"source": [
"**Partners.**\n",
"You are welcome (and encouraged) to work with one partner.\n",
"If you do work with a partner, only one of you needs to submit the assignment on `Gradescope` and tag the other as a group member."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XCMk0AEJYupX"
},
"source": [
"**Environment.**\n",
"Before you submit, make sure your code works in the environment described in the [`Environment Check`](#environment_check) section, as this is the environment our autograder will be run on.\n",
"If you have completed the setup steps in `PA0` and run this notebook in the `cs124` environment you created according to the instructions, you are good!\n",
"Note that you must not use any other dependencies (such as other `Python` modules), as doing so may cause the autograder to fail!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9DjXAbXQYxcD"
},
"source": [
"**Saving Your Notebook**.\n",
"Make sure to save the recent changes in your notebook before you submit.\n",
"This is especially important if you are running your notebook on `Google Colab` as connection quality sometimes cause your notebook to be in an unsaved state.\n",
"The following error is also common on `Google Colab`, if the file you are working on is open in more than one tabs, so we are recommending keeping copies of your work if you are collaborating with your partner on `Colab`.\n",
"```\n",
"This file was updated remotely or in another tab. To force a save, overwriting the last update, select Save from the File menu\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lMSPTRReQFXo"
},
"source": [
"\n",
"\n",
"\n",
"\n",
"**Files.**\n",
"Once you are done, you only need to submit the file listed below.\n",
"**DO NOT** alter the file name.\n",
"```\n",
"pa2.ipynb\n",
"```\n",
"\n",
"**Custom Dependencies.**\n",
"Sometimes you may want to put parts of your code into `.py` files and call them from your notebook instead of having all your functions in the notebook, or utilize extra datasets.\n",
"If this is the case, please put your extra files in a folder\n",
"named `deps/` (this folder should be on the same level as `pa2.ipynb`)\n",
"and upload a `zip` file (any name is fine) containing this folder and\n",
"`pa2.ipynb` to submit on `Gradescope`.\n",
"Note that these should be at the top directory of the `.zip` file (e.g. they should not be in a directory in the `.zip` file, as this will lead our autograder to fail at finding them).\n",
"To prevent this, ensure that you are only zipping the items mentioned, and not the folder containing them.\n",
"`Gradescope` will then automatically `unzip` the folder so that your\n",
"submission contains the following.\n",
"```\n",
"deps/\n",
"pa2.ipynb\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iRZdYG4raTMj"
},
"source": [
"**Submission Script.**\n",
"For your convenience, we are providing the following submission script that lets you automatically create a `zip` file to submit.\n",
"Simply run it and submit `submission.zip` to `Gradescope`.\n",
"Note that the script assumes that you have the `zip` utility installed.\n",
"You would need to install it if you don't already have it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3UmC4xynbYbc",
"outputId": "0d94f413-1c61-41ea-84b8-938461060afe",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"if [[ ! -f \"./pa2.ipynb\" ]]\n",
"then\n",
" echo \"WARNING: Did not find notebook in Jupyter working directory. This probably means you're running on Google Colab. You'll need to go to File->Download .ipynb to download your notebok and other files, then zip them locally. See the README for more information.\"\n",
"else\n",
" echo \"Found notebook file, creating submission zip...\"\n",
" zip -r submission.zip pa2.ipynb deps/\n",
"fi"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rC7Ji3nxbo2Z"
},
"source": [
"If you are running your notebook on `Google Colab`, see the `README` for instructions on how to submit.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mHHWQZvAZPZ4"
},
"source": [
"**Autograder.**\n",
"Once you submit, double check the autograder output to ensure that your submission didn't cause any error."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PnnkT9eMhBlJ"
},
"source": [
"\n",
"## Environment Check"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2cyv1CAE-o-m"
},
"source": [
"This assignment assumes that you have correctly set up the `cs124` conda environment and installed the required `Python` modules.\n",
"The cell below checks that you are running the correct version of `Python` and activated the `cs124` conda environment.\n",
"If you are running the notebook on `Google Colab`, you need to download the `Python` extra modules we use in the assignment separately.\n",
"If you get an error running this cell, it means that you are either using the wrong `Conda` environment\n",
"or Python version!\n",
"If the latter, please exit this notebook, kill the notebook server with `CTRL-C`, and\n",
"try running:\n",
"\n",
"`$ conda activate cs124`\n",
"\n",
"Then restarting your notebook server with\n",
"\n",
"`$ jupyter notebook`\n",
"\n",
"If this doesn't work, you should go back and follow the installation instructions in `PA0`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rUVLxfnHiPbK",
"tags": [
"essential"
]
},
"outputs": [],
"source": [
"import os\n",
"assert os.environ['CONDA_DEFAULT_ENV'] == \"cs124\"\n",
"\n",
"import sys\n",
"assert sys.version_info.major == 3 and sys.version_info.minor == 10"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OdAYv1X28NIT"
},
"source": [
"\n",
"## Part 0. Setup\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hQ8pMJZJDIJu"
},
"source": [
"**Getting the Necessary Files.** The cell below downloads the necessary files we will use in this assignment, if you don't already have them.\n",
"This may be the case, for example, if you are running the assignment on `Google Colab`. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LkwF0Sb28d7-",
"outputId": "46d0e929-b287-405c-acbb-5ead463a9077",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"if [[ ! -d \"./data\" ]]\n",
"then\n",
" echo \"Missing extra files. Downloading...\"\n",
" git clone https://github.com/cs124/pa2-logistic-regression.git\n",
" cp -r ./pa2-logistic-regression/{data,deps,util.py} .\n",
"fi"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0dSKvfgi-Mna"
},
"source": [
"**Importing Modules.** Run the next cell to import the necessary modules we will use in this assignment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OhuxbSxv-99D",
"tags": [
"essential"
]
},
"outputs": [],
"source": [
"\"\"\" Modules included in the Python Standard Library \"\"\"\n",
"\n",
"# collections module contain useful Python data structures, such as dictionaries\n",
"# with special properties\n",
"from collections import defaultdict\n",
"\n",
"# operator module allows us to use functions such as add() instead of operators\n",
"# such as +\n",
"import operator\n",
"\n",
"# random modules allows us to insert randomization to our code\n",
"import random\n",
"\n",
"# typing module contains type objects. We will use these types to ensure that \n",
"# the inputs and outputs passed to the functions you will be implementing are \n",
"# of the correct type\n",
"from typing import List, Dict, Union"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SXB-ukJxVXMj",
"tags": [
"essential"
]
},
"outputs": [],
"source": [
"\"\"\" Third party modules \"\"\"\n",
"\n",
"# numpy is a widely used scientific computing package, allowing us to do large\n",
"# matrix operations efficiently\n",
"import numpy as np\n",
"\n",
"# matplotlib is a popular library used by researchers to plot graphs\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# sklearn is a popular machine learning library, providing useful tools for \n",
"# machine learning tasks\n",
"from sklearn.feature_extraction.text import CountVectorizer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "t5qPbMwQVC_a",
"tags": [
"essential"
]
},
"outputs": [],
"source": [
"\"\"\" Our custom functions and classes \"\"\"\n",
"\n",
"# Helper functions and classes we will use later\n",
"from util import load_data, Classifier, Example, evaluate, remove_stop_words\n",
"from util import check_logistic_loss, check_gradient_descent"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ma5Fcxt9_X1d"
},
"source": [
"**WARNING:** **DO NOT** import or use any other packages except the ones imported above and other packages in the Python standard library.\n",
"This means you should not use `spaCy`, `NLTK`, `gensim`, or other functionality in `scikit-learn` besides `CountVectorizer`, even though those are provided in the `conda` environment we set up for you.\n",
"If your solution uses any such extra dependencies it will fail the autograder."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L98XWjQyjUw8"
},
"source": [
"\n",
"## Part 1. Data Exploration for the Triage Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F8srZPWSq5IQ"
},
"source": [
"As usual, the first thing to do is to understand and characterize the data!\n",
"The data for this assignment contains about `26K` documents from several major natural disasters, as listed below.\n",
"\n",
"* [Earthquake in Haiti (2010)](https://en.wikipedia.org/wiki/2010_Haiti_earthquake)\n",
"* [Floods in Pakistan (2010)](https://en.wikipedia.org/wiki/2010_Pakistan_floods)\n",
"* [Earthquake in Chile (2010)](https://en.wikipedia.org/wiki/2010_Chile_earthquake)\n",
"* [Hurricane Sandy in North America (2012)](https://en.wikipedia.org/wiki/Hurricane_Sandy)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_u-uRvlRPhDP"
},
"source": [
"**Dataset.** The documents in our dataset are either text messages, social media (`Twitter`) posts, or snippets from news articles.\n",
"In addition to the specific events listed above the dataset contains a number of news articles spanning dozens of different disasters.\n",
"All messages have been translated and annotated by humans on the crowdsourcing platform `CrowdFlower` (now branded under [`Appen`](https://appen.com/)).\n",
"However, some of the translations are not perfect, and you may encounter some words in other\n",
"languages.\n",
"Unfortunately, `NLP` researchers often have to work with `messy` data.\n",
"If you are curious about the crowdsourcing translation effort for messages\n",
"from Haiti in particular, feel free to check out [this paper](https://nlp.stanford.edu/pubs/munro2010translation.pdf).\n",
"\n",
"Your task is to classify each document as being aid-related, class `aid`, or not aid-related, class `not`.\n",
"Messages that are aid-related include individuals' requests for food, water, or shelter etc.\n",
"The `aid` class also includes news reports about dire situations and disaster relief efforts.\n",
"Below are several examples of aid-related documents, belonging to class `aid`.\n",
"```\n",
"Hello Good Morning We live on 31 Delmas we are without water without food and what we had have finished Please do something for us!\n",
"```\n",
"```\n",
"I am sending this SMS from Layah district for my sister whose house has got destroyed in a flood\n",
"So, the problem she faces now is that she hasn't got any 'Watan Card'or any financial aid from the government.\n",
"She has 5 children too.\n",
"```\n",
"```\n",
"Redcross came to my house and gave my family food ...\n",
"Guess were not getting power anytime soon . #sandy #RedCross\n",
"```\n",
"```\n",
"Relief officials have stressed the vital importance of bringing in clean drinking water and sanitation equipment to avoid deadly epidemics that in a\n",
"worst case scenario could claim as many or more lives than the tsunami itself.\n",
"```\n",
"Below are several examples of non-aid-related documents, belonging to class `not`:\n",
"```\n",
"A cold front is found over Cuba this morning.\n",
"It could cross Haiti tomorrow.\n",
"Isolated rain showers are expected over our region tonight.\n",
"```\n",
"```\n",
"Hurricane : A storm which New Yorkers use as an excuse to drink and eat junk\n",
"food in their pajamas for 48 hours . #sandy\n",
"```\n",
"```\n",
"By secret ballot, the Council elected Pakistan, Bahrain and the Republic of\n",
"Korea from the Asian States, while Iran and Saudi Arabia did not receive enough\n",
"votes to qualify.\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lCm95WnCcw-p"
},
"source": [
"**Training, Validation, and Test Sets.**\n",
"The data is divided into a `training` set, `development` (`validation`) set, and `test` set.\n",
"The `training` set is used to learn, compute the statistics\n",
"for, your model.\n",
"These statistics are then used to classify the documents in the\n",
"`development` and `test` sets.\n",
"For this assignment, you have access to the\n",
"`training` set and the `dev` set.\n",
"The `test` set is hidden, but your submission will be evaluated on it as well.\n",
"Although we do not share the specific test examples we use, the autograder we provide will output target accuracies your classifier should achieve for each of these sets for full credit. \n",
"All you need to do to see these is to submit your assignment.\n",
"Remember that you can always re-submit!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "V2YYfAJYb5Bm"
},
"source": [
"**Exploration.**\n",
"Let's take a look at some of the data.\n",
"We have defined a `Dataset` class for you to store the loaded data in a way we can easily access later, and a function `load_data()` to load it in the format we want.\n",
"You do not need to check the specifics of our `Dataset` class, we will explain exactly how you will use it in this assignment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Z4SSGY25eIoS",
"outputId": "7019827b-9a67-4c91-938b-e0b7afb5030b",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Load our dataset\n",
"dataset = load_data(\"./data/triage\")\n",
"\n",
"# Check that the type of our dataset is the Dataset class we defined for you\n",
"# in util.py.\n",
"print(type(dataset))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tJDNkOtsgV66"
},
"source": [
"We are interested in the following two fields of the `Dataset` class: `train` and `dev`.\n",
"Given that `dataset` is an instance of the `Dataset` class, we can access these fields with `dataset.train` and `dataset.dev`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KUgCeJsUgkRK",
"outputId": "cb926386-3470-4e3d-da9a-67ca39794552",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"print(f\"dataset.train contains {len(dataset.train)} examples\")\n",
"print(type(dataset.train[0]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ut61mksEmR1r"
},
"source": [
"Each of `dataset.train` and `dataset.dev` is a list of `Example`'s.\n",
"Similar to `Dataset`, `Example` is a class we have defined to represent each data point we have.\n",
"The `Example` class has two fields we will be concerned with: `words` and `label`.\n",
"`words` field corresponds to the list of words making up the example.\n",
"`label` field corresponds to the label of the data point, which is an integer that can only take one of two values: `1` for `aid` and `0` for `not` aid."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LwRvqrY5h6lz",
"outputId": "72c843a9-7e61-47ec-9266-6b41c42f0c8e",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"print(\"First training example:\")\n",
"print(\"Words: {}\".format(dataset.train[0].words))\n",
"print(\"Label: {}\".format(dataset.train[0].label))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3lRaJN9dhorh"
},
"source": [
"In summary, we use the custom defined classes `Example` and `Dataset` to represent our dataset and data points in a nice format so that we can work with them easily.\n",
"This is achieved by the `load_data()` function we called earlier.\n",
"At a high level, when we pass it the path `./data/triage`, `load_data()` finds the `CSV` files located there. \n",
"Within each of these `CSV` files, each line is a single example, consisting\n",
"of a document (`string`) and a corresponding label.\n",
"The function `load_data()` reads each line in the `CSV` files as a new `Example`.\n",
"It tokenizes each document it reads and sets the `words` field of the `Example` class to be a list of words.\n",
"You can check the `CSV` files located in `./data/triage` to see the original format of these files.\n",
"\n",
"Note that the data you are given is already preprocessed; all punctuation has been removed, except hashtags and apostrophes, and all text has been converted to lowercase.\n",
"If you were working on a new task, you would likely need to complete the preprocessing step yourself.\n",
"Depending on the specific `NLP` task, preprocessing can significantly improve performance.\n",
"You do not need to do any additional preprocessing for our task.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "94Qmp95Mc2d9"
},
"source": [
"\n",
"## Part 2. Logistic Regression"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1nfmLc4hpvCs"
},
"source": [
"Now that we have our data set up, we can get started on implementing our \n",
"`Logistic Regression` classifier!\n",
"First, let's start off with some preliminaries to double-check our `NumPy` skills\n",
"and our understanding of the `Logistic Regression` algorithm."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Part 2.1 Sigmoid"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One of the building blocks of `Logistic Regression` is the `Sigmoid` function, \n",
"which we described in lecture.\n",
"It is the method we use to convert the outputs of our computation \n",
"$(z = w * x + b)$ from a real number between negative infinity and infinity to \n",
"a probability between `0` and `1`.\n",
"Your first task is to implement the `Sigmoid` function below!\n",
"\n",
"Remember that the `Sigmoid` function is defined as:\n",
"\n",
"$$\n",
"\\sigma(z) = \\frac{1}{1 + e^{-z}}\n",
"$$\n",
"\n",
"where $z$ is the input to the function.\n",
"\n",
"Now, let's implement the `Sigmoid` function!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"todo"
]
},
"outputs": [],
"source": [
"def sigmoid(x: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" TODO: Implement the sigmoid function.\n",
"\n",
" Args:\n",
" x: A numpy array.\n",
" Returns:\n",
" s: The numpy array with sigmoid applied element-wise\n",
"\n",
" HINTS:\n",
" * Use np.exp() because your input can be a numpy array\n",
" \"\"\"\n",
" # CODE START\n",
" pass\n",
" # CODE END"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once you are done with your implementation, try visualizing it to\n",
"double-check that we have the right idea.\n",
"Hopefully the graph looks reasonable!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Create a numpy array containing evenly separated numbers, from -10 to 10 \n",
"# (inclusive)\n",
"x = np.arange(-10, 10, .01)\n",
"\n",
"# Call our sigmoid function on the newly created array\n",
"y = sigmoid(x)\n",
"\n",
"# Plot\n",
"plt.plot(x, y, color='blue', lw=2)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Part 2.2 Logistic Loss"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now there's just one more building block we need: the `logistic loss` function, \n",
"which also known as the `cross-entropy loss`. \n",
"Intuitively, the loss function is simply a way to measure how far our prediction\n",
"$y_{\\text{pred}} = \\sigma(WX + b)$ is from the true label $y_{\\text{true}}$.\n",
"\n",
"Remember that the logistic loss function is defined as:\n",
"\n",
"$$\n",
"L(y_{\\text{pred}}, y_{\\text{true}}) = - [y_{\\text{true}} \\log(y_{\\text{pred}}) + (1 - y_{\\text{true}}) \\log(1 - y_{\\text{pred}})]\n",
"$$\n",
"\n",
"where $y_{\\text{pred}}$ is the predicted probability of the true label $y_{\\text{true}}$.\n",
"\n",
"Now implement the `logistic loss` function below!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"todo"
]
},
"outputs": [],
"source": [
"def logistic_loss(y_pred: np.ndarray, y_true: np.ndarray) -> float:\n",
" \"\"\"\n",
" TODO: Implement the computation of the logistic loss function.\n",
"\n",
" Your function should take in not just a single predicted label\n",
" and true label, but a vector of predictions and a vector of true labels.\n",
" This shouldn't affect your implementation much, as numpy allows you to \n",
" operate naturally on vectors.\n",
"\n",
" Your function should return the average logistic loss over all the\n",
" examples as a single float.\n",
"\n",
" Args:\n",
" y_pred: A 1D vector of predicted labels for each example, of shape\n",
" (num_examples,).\n",
" y_true: A 1D vector of true labels for each example, of shape\n",
" (num_examples,).\n",
"\n",
" Returns:\n",
" float: The average logistic loss over all examples as a float.\n",
"\n",
" HINTS:\n",
" * np.mean() and np.log() could be helpful!\n",
" * If you run into \"RuntimeWarning: divide by zero encountered in log\" \n",
" issues, try adding a very small number epsilon (like 1e-8) to the \n",
" input any time you call np.log(). This will ensure that the input to \n",
" the log is never exactly 0, which gives an undefined result.\n",
" \"\"\"\n",
" # CODE START\n",
" pass\n",
" # CODE END"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To check our implementation, let's compare it to our intuition about how the\n",
"loss should behave.\n",
"The loss is supposed to represent how far our prediction is from the true label. \n",
"In other words, if our prediction is way off, the loss should be very high.\n",
"If our prediction gets closer to the true value, the loss should drop towards \n",
"`0`.\n",
"Let's consider a few cases to understand how the loss should change on specific\n",
"input!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"def print_loss(y_pred, y_true):\n",
" print(\"Predicted = {}, True = {} : Loss = {}\".format(\n",
" y_pred, y_true, logistic_loss(np.array([y_pred]),\n",
" np.array([y_true]))))\n",
"\n",
"print_loss(0.0, 1)\n",
"print_loss(0.1, 1)\n",
"print_loss(0.3, 1)\n",
"print_loss(0.5, 1)\n",
"print_loss(0.7, 1)\n",
"print_loss(0.9, 1)\n",
"print_loss(0.99, 1)\n",
"print_loss(0.999999, 1)\n",
"print_loss(1, 1)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Take a look at the results.\n",
"What happens when $y_{pred} = y_{true}$? What happens when $y_{pred} = 0$ and \n",
"$y_{true} = 1$?\n",
"Do they agree with what you expect from the formula for the logistic loss?\n",
"Why might this behavior be okay in practice, given how we are going to use the\n",
"`logistic loss` (i.e. `gradient descent`)?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Sanity check for logistic_loss()\n",
"check_logistic_loss(logistic_loss)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Part 2.3 Gradient Descent"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that you have implemented the `sigmoid()` and `logistic_loss()`\n",
"functions, our next step is to introduce our optimization function.\n",
"We will use the `gradient descent` algorithm to learn the parameters for our \n",
"`logistic regression` classifier from the data.\n",
"We have implemented the majority of this algorithm for you, but you will need to compute:\n",
"\n",
"- The gradient of the loss with respect to `W` (`dW`).\n",
"- Compute the gradient of the loss with respect to `b` (`db`).\n",
"- Update `W` \n",
"- Update `b` \n",
"\n",
"The lecture slides give the derivative of the loss for individual elements of the vector in terms of $w_j$ and $b$. However, in your python implementation, you'll want to compute the gradient updates using the matrix form. \n",
"\n",
"Let A denote the vector of predictions, and Y denote the vector of true labels. To compute `dW`, `db` (note that $m$ is the batch size):\n",
"\n",
"- $dW = \\frac{1}{m} X_{\\text{batch}}^T \\left(A_{\\text{batch}} - Y_{\\text{batch}}\\right)$\n",
"- $db = \\frac{1}{m} \\sum_{i=1}^{m} \\Bigl(A^{(i)} - Y_{\\text{batch}}^{(i)}\\Bigr)$\n",
"\n",
"To update $W$ and $b$, you want to utilize the gradients of the loss function to move $W$ and $b$ in the opposite direction, while taking into account the learning rate. The updates are given by:\n",
"\n",
"- $W = W - \\alpha dW$\n",
"- $b = b - \\alpha db$\n",
"\n",
"where $\\alpha$ is the learning rate.\n",
"\n",
"**Hints:**\n",
" - `A` and `Y_batch` are 1D arrays of shape `(batch_size,)`, while `X_batch` has shape `(batch_size, num_features)`. If you want to apply matrix multiplication between `X_batch` and `(A - Y_batch)`, you must reshape `(A - Y_batch)` to a column vector of shape `(batch_size, 1)`. For example, you can use `X_batch.T @ (A - Y_batch).reshape(-1, 1)` to compute the product with shape `(num_features, 1)`.\n",
" - If you want to convert the shape of the product from `(num_features, 1)` to `(num_features,)`, you can use `.flatten()`.\n",
" - The formula for `dW` includes a `1/m` term. After computing the matrix product and flattening, divide the result by `batch_size` to average over the batch.\n",
" - For `db`, the formula is the average of `(A - Y_batch)`. You can use `np.mean(A - Y_batch)`.\n",
" - Once you have computed dW and db, update `W` and `b` using the formulas: `W = W - alpha * dW` and `b = b - alpha * db`.\n",
"\n",
"Now implement the `gradient_descent` function below!\n",
"\n",
"\n",
"\n",
"We will have a deeper discussion on `gradient descent` in the context of neural\n",
"networks.\n",
"While you only have to write a few lines of code for this section, we encourage you to take a look and try to\n",
"understand what the function is doing as a whole, as well as what each of the\n",
"parameters (`alpha`, `epsilon`, `num_iterations`) does.\n",
"You can also see how it makes use of the `sigmoid()` and `logistic_loss()`\n",
"functions we just implemented!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MfWXy7PkqK0S",
"tags": [
"essential"
]
},
"outputs": [],
"source": [
"# DO NOT CHANGE\n",
"def gradient_descent(X: np.ndarray,\n",
" Y: np.ndarray,\n",
" batch_size: int = 2000,\n",
" alpha: float = 0.5,\n",
" num_iterations: int = 1000,\n",
" print_every: int = 100,\n",
" epsilon: float = 1e-8) -> (np.ndarray, float):\n",
" \"\"\"\n",
" Runs batch gradient descent on the provided data and returns the resulting\n",
" trained weight vector and bias.\n",
"\n",
" Args:\n",
" X: A numpy array of shape (num_examples, num_features) containing\n",
" the training data.\n",
" Y: A numpy array of shape (num_examples,) containing the training\n",
" labels.\n",
" batch_size: The number of examples in each batch.\n",
" alpha: The learning rate for gradient descent.\n",
" num_iterations: The number of iterations to run gradient descent\n",
" for.\n",
" print_every: How often (after how many iterations) to print the\n",
" loss and iteration number.\n",
" epsilon: The early stopping condition. When the absolute change\n",
" in the loss is less than epsilon, gradient descent will\n",
" stop early.\n",
"\n",
" TODO:\n",
" Fill in the code to compute dW, db, W, and b using the gradient descent equations \n",
" for logistic regression explained above. \n",
" \n",
" Once you have the gradients of the loss function, use them to update W and b to move\n",
" in the opposite direction, taking into account the learning rate.\n",
" \n",
" - dW: the gradient of the loss with respect to W\n",
" - db: the gradient of the loss with respect to b\n",
" - W: the weight vector \n",
" - b: the bias vector \n",
" \n",
" Returns:\n",
" (np.ndarray, float): The learned weight vector W and bias b\n",
" \"\"\"\n",
" # DO NOT CHANGE\n",
" # Initialize the weight vector W to all zeros of shape (num_features, )\n",
" W = np.zeros((X.shape[1],))\n",
" b = 0 \n",
" Y = np.array(Y)\n",
" \n",
" # Initialize loss, which is updated every iteration\n",
" loss = 0\n",
" for i in range(num_iterations):\n",
" # If the batch size is greater than or equal to the entire dataset,\n",
" # we just use all the data in one batch\n",
" if batch_size >= X.shape[0]:\n",
" X_batch = X\n",
" Y_batch = Y\n",
" else:\n",
" # Otherwise, we randomly sample examples of size 'batch_size' to form the current batch\n",
" batch_indices = np.random.randint(X.shape[0], size=batch_size)\n",
" batch_indices = np.random.randint(X.shape[0], size=batch_size)\n",
" X_batch = X[batch_indices, :]\n",
" Y_batch = Y[batch_indices]\n",
" \n",
" # 1) Compute predicted probabilities for our batch using the sigmoid function\n",
" # A has shape (batch_size, ) \n",
" A = sigmoid(np.dot(X_batch, W) + b)\n",
" \n",
" # TODO: 2) Compute the gradients dW and db,\n",
" # and update the params W and b using the gradients and learning rate alpha. \n",
" # dW has shape (num_features, )\n",
" # db is a scalar\n",
" \n",
" # CODE START\n",
" pass\n",
" # CODE END\n",
" \n",
" # 4) Store the old loss so we can stop early if needed \n",
" prev_loss = loss\n",
" \n",
" # 5) Compute the new loss using logistic_loss()\n",
" loss = logistic_loss(A, Y_batch)\n",
" \n",
" # If the absolute change in loss is below our threshold, \n",
" # we break out of the loop and stop early \n",
" if abs(prev_loss - loss) < epsilon:\n",
" break\n",
" \n",
" # Print progress for every 'print_every' iterations\n",
" if (i+1) % print_every == 0:\n",
" \n",
" # Convert probabilities into binaries (0 or 1)\n",
" predictions = A\n",
" predictions[predictions >= 0.5] = 1\n",
" predictions[predictions < 0.5] = 0\n",
" \n",
" # Compute accuracy for batch\n",
" accuracy = np.mean(predictions == Y_batch)\n",
" print(\"Iteration {}/{}: Batch Accuracy: {}, Batch Loss = {}\".format(\n",
" i + 1,\n",
" num_iterations,\n",
" accuracy,\n",
" loss\n",
" ))\n",
" # 6) Return the final learned parameters: weight vector W and bias b\n",
" return W, b\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Sanity check for gradient_descent()\n",
"check_gradient_descent(gradient_descent)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Part 2.4 Logistic Regression Classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Without any further ado, here is the skeleton code for the full \n",
"`Logistic Regression` classifier.\n",
"Your task is to finish it up.\n",
"You will use the functions you have implemented above."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"todo"
]
},
"outputs": [],
"source": [
"class LogisticRegressionClassifier(Classifier):\n",
" \"\"\"\n",
" TODO: Implement the Logistic Regression classifier.\n",
" \"\"\"\n",
" def __init__(self,\n",
" filter_stop_words: bool = None,\n",
" batch_size: int = 2000,\n",
" alpha: float = 0.5,\n",
" num_iterations: int = 1000,\n",
" print_every: int = 100,\n",
" epsilon: float = 1e-8):\n",
" super().__init__(filter_stop_words)\n",
" ngram = 1\n",
" tokenizer = str.split if self.filter_stop_words else None\n",
"\n",
" \"\"\"\n",
" self.vectorizer is a countVectorizer we have created for you. Use\n",
" it to obtain a feature vector of word counts for each example\n",
" in your training data.\n",
" Documentation: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html\n",
" \n",
" Note: You do not need to write code to filter stop words as this is handled automatically for you \n",
" via the filter_stop_words param\n",
" \"\"\"\n",
" # Convert stop_words set to sorted list (required by newer sklearn versions)\n",
" mod_stop_words = None if self.stop_words is None else sorted(list(self.stop_words))\n",
" self.vectorizer = CountVectorizer(min_df=20,\n",
" ngram_range=(ngram, ngram),\n",
" stop_words=mod_stop_words,\n",
" tokenizer=tokenizer)\n",
"\n",
" # Parameters to use for gradient_descent()\n",
" self.batch_size = batch_size\n",
" self.alpha = alpha\n",
" self.num_iterations = num_iterations\n",
" self.print_every = print_every\n",
" self.epsilon = epsilon\n",
"\n",
" # TODO: Add other data structures needed in classify() or train()\n",
" # CODE START\n",
" pass\n",
" # CODE END\n",
"\n",
" def train(self, examples: List[Example]) -> None:\n",
" \"\"\"\n",
" TODO: Implement a function to train a logistic regression model.\n",
"\n",
" Args:\n",
" examples: A Python list containing instances of the Example class,\n",
" which correspond to the datapoints in the dataset.\n",
"\n",
" HINTS:\n",
" * Use the vectorizer we have defined (self.vectorizer) to convert each\n",
" example into a feature vector of word counts. Read the documentation\n",
" for the CountVectorizer. Is there a method you can use to do this?\n",
" * Call gradient_descent() we have defined above to return the learned \n",
" weight vector and bias. You can save these for later use in \n",
" classify().\n",
" * You should use the parameters (batch_size, alpha, num_iterations,\n",
" print_every, epsilon) provided as arguments to the\n",
" LogisticRegressionClassifier when calling gradient_descent().\n",
" * Call self.X.toarray() after you have populated self.X with counts.\n",
" This converts it from a sparse matrix to a dense matrix so\n",
" we can use it to perform gradient descent.\n",
" * Depending on your implementation, your model may take some time to \n",
" train!\n",
" \"\"\"\n",
" # CODE START\n",
" pass\n",
" # CODE END\n",
"\n",
" def classify(self, examples: List[Example],\n",
" return_scores: bool = False) -> Union[List[int], List[float]]:\n",
" \"\"\"\n",
" TODO: Implement a function to classify the given examples!\n",
"\n",
" Args:\n",
" examples: A Python list containing instances of the Example class,\n",
" which correspond to the datapoints in the dataset.\n",
" return_scores: A boolean flag indicating whether the raw scores should\n",
" be returned\n",
"\n",
" Returns:\n",
" Union[List[int], List[float]]: The classification result for each \n",
" example in a list. If the return_scores flag is False, the \n",
" function should return an object of type List[int], a list of 0 \n",
" or 1, corresponding to each class.\n",
" If the return_scores flag is True, the function should instead\n",
" return an object of type List[float], the raw scores from the \n",
" sigmoid function.\n",
"\n",
" HINTS:\n",
" * We use the `Union` class in the return type to indicate that it could\n",
" be a List[int] or List[float], which indicate a list containing \n",
" integers or a list containings floats, respectively. Whether you \n",
" will return a list of int or float depends on the value of the \n",
" return_scores flag.\n",
" * If sigmoid(X * W + b) is greater or equal to 0.5, the example\n",
" belongs to class 1, which is the positive class. Otherwise, it \n",
" belongs to class 0, which is the negative class. You can use the \n",
" sigmoid function you implemented above.\n",
" * You should use the weight vector and bias you computed earlier in \n",
" train().\n",
" * You can use np.dot or np.matmul to do matrix multiplication.\n",
" \"\"\"\n",
" # CODE START\n",
" pass\n",
" # CODE END\n",
"\n",
" def get_weights(self) -> np.ndarray:\n",
" \"\"\"\n",
" TODO: Implement a function to return the trained weights.\n",
"\n",
" Returns:\n",
" np.ndarray: Trained weights, a NumPy array in the sahpe \n",
" (num_features,).\n",
" \"\"\"\n",
" # CODE START\n",
" pass\n",
" # CODE END"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v-ZY04F8zuVb"
},
"source": [
"\n",
"## Part 3. Evaluation on the Triage Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2uizs3Cy0AN0"
},
"source": [
"### Part 3.1. Accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xYgmZfrMzXzS"
},
"source": [
"Once your implementation is ready, you can try evaluating it on the disaster aid classification dataset as shown below.\n",
"Our implementation achieves the following statistics when run for the default number of iterations, so if you are getting similar results that probably means that your implementation is working well!\n",
"\n",
"```\n",
"Performance on Unigrams, no stopword removal:\n",
"Accuracy (train): 0.7903164496816497\n",
"Accuracy (dev): 0.7726389428682472\n",
"Performance on Unigrams with stopword removal:\n",
"Accuracy (train): 0.7758243846811745\n",
"Accuracy (dev): 0.7602020987174505\n",
"```\n",
"Our `autograder` will test the accuracy achieved by your implementation of the `LogisticRegressionClassifier` on `train`, `dev`, and `test` datasets, both with and without stop words. \n",
"If you aren't getting the performance you are expecting in the below cell, go back to your `train` and `classify`, and `gradient_descent` methods. For example, if your loss is not going down, double check your implementation of the gradient descent function. **Remember, the classifier automatically filters stop words via the `filter_stop_words` param and the given vectorizer, so no additional code is required for stop word handling.** \n",
"If you are curious about the exact cutoffs we use to grade your work, you can submit your notebook to the autograder set up on `Gradescope`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_yrkKIi2EKVy",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Load our dataset\n",
"dataset = load_data(\"./data/triage\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "r4rzyTrkzVO9",
"outputId": "c7dc7cdb-adb3-43d3-a570-e91da4f375f3",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"print(\"Performance on Unigrams, no stopword removal:\")\n",
"lr_classifier = LogisticRegressionClassifier(filter_stop_words=False)\n",
"evaluate(lr_classifier, dataset)\n",
"\n",
"print(\"Performance on Unigrams w/ stopword removal:\")\n",
"lr_classifier_swr = LogisticRegressionClassifier(filter_stop_words=True)\n",
"evaluate(lr_classifier_swr, dataset)\n",
"\n",
"print(\"Performance on Unigrams w/ stopword removal:\")\n",
"lr_classifier_swr = LogisticRegressionClassifier(filter_stop_words=True, num_iterations=3000)\n",
"evaluate(lr_classifier_swr, dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mNcYa1XGKIIG"
},
"source": [
"### Part 3.2 Sanity Check"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cM2qYDkYKMtx"
},
"source": [
"Once we've implemented and trained our model, it's often helpful to do some investigating to confirm that it's behaving the way we expect. For a `Logistic Regression` model, there are couple of different ways we can do this.\n",
"\n",
"For our `Logistic Regression` model, after training we have access to a weight\n",
"vector that contains a weight for each feature.\n",
"We can connect these weights back to the list of features (in our case, these will be unigrams). \n",
"\n",
"The next two cells should print your weight vectors. If the weights are zero, or if the weights are unusually large, double check your `gradient_descent` function.\n",
"\n",
"Features with larger weights are those that the model associates with the \n",
"`positive` label, and those with smaller weights are those the model associates\n",
"with the `negative` label.\n",
"If it is not clear why this is true, try thinking back to the equation used\n",
"to compute the `Logistic Regression` output.\n",
"If a weight for a feature is a large positive number, and that \n",
"feature appears in an example, what will happen to the output for \n",
"that example?\n",
"What about a feature with a large negative weight?\n",
"Let's examine the features with the largest and smallest weights in our\n",
"trained classifier."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"features = lr_classifier.vectorizer.get_feature_names_out()\n",
"weights = lr_classifier.get_weights()\n",
"\n",
"features_to_weights = [(features[i], weights[i])\n",
" for i in range(len(features))]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"top_10_features = sorted(features_to_weights,\n",
" key=operator.itemgetter(1), reverse=True)[:10]\n",
"for feature, weight in top_10_features:\n",
" print(\"{}: weight = {}\".format(feature, weight))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"bottom_10_features = sorted(features_to_weights,\n",
" key=operator.itemgetter(1))[:10]\n",
"for feature, weight in bottom_10_features:\n",
" print(\"{}: weight = {}\".format(feature, weight))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Reflection Question.**\n",
"Do these features agree with your intuition?\n",
"Return your answer in the `reflection_response` function below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO: Include your reflection below\n",
"def reflection_response_3_2():\n",
" reflection = \"\"\n",
" return reflection"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ve3QXPqc5TGr",
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"**False Positives and Negatives.** Another good thing to check is where your model made errors. In this case, our\n",
"task was binary classification, so there are two possible types of errors\n",
"that could have occurred:\n",
"\n",
"* `False Positives`: Our model predicted a high probability of label = 1 for a negative example.\n",
"* `False Negatives`: Our model predicted a low probability of label = 1 for a positive example.\n",
"\n",
"We can look for exactly these two types of errors using the `return_scores`\n",
"flag we asked you to implement for the `classify()` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "j95TGjLK5TGs",
"pycharm": {
"name": "#%%\n"
},
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"def get_false_negatives_and_false_positives(classifier, examples):\n",
" predicted_scores = classifier.classify(examples, return_scores=True)\n",
"\n",
" false_negatives = []\n",
" false_positives = []\n",
"\n",
" for pred_score, example in zip(\n",
" predicted_scores, examples):\n",
" if example.label == 1 and pred_score < 0.5:\n",
" false_negatives.append((example.words, pred_score))\n",
" elif example.label == 0 and pred_score >= 0.5:\n",
" false_positives.append((example.words, pred_score))\n",
"\n",
" return false_negatives, false_positives\n",
"\n",
"fn, fp = get_false_negatives_and_false_positives(lr_classifier, dataset.dev)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7HQabfgj1Iez"
},
"source": [
"Now that we have the false negatives and false positives, we can find the\n",
"\"worst\" ones and examine them to try to figure out where our model went wrong.\n",
"\n",
"The **worst** ones would be:\n",
"* The `false negatives` with the lowest probabilities of label = 1\n",
"* The `false positives` with the highest probabilities of label = 1\n",
"\n",
"Run the cells below, and think about the following questions:\n",
"* Do these errors seem reasonable?\n",
" Can you think of why the model may have made them?\n",
"* How do the different settings (stop word removal vs. no stop word removal) affect the \n",
" performance relative to each other? \n",
" Do these results seem reasonable?\n",
" If not, what might explain what you're seeing?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ogpOEF4c5TGs",
"outputId": "c27a08fb-5eec-4165-9017-528e4da40800",
"pycharm": {
"name": "#%%\n"
},
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"top_10_fn = sorted(fn,\n",
" key=operator.itemgetter(1))[:10]\n",
"for words, prob in top_10_fn:\n",
" print(\"prob = {}: {}...\".format(prob, words[:min(len(words), 10)]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4weD966k5TGt",
"outputId": "75725477-815e-486b-8a0f-5a28387c16b7",
"pycharm": {
"name": "#%%\n"
},
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"if len(fp) == 0:\n",
" print(\"No false positives found!\")\n",
"\n",
"top_10_fp = sorted(fp, key=operator.itemgetter(1), reverse=True)[:10]\n",
"for words, prob in top_10_fp:\n",
" print(\"prob = {}: {}\".format(prob, words))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UKRIDRHH5TGt",
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Hopefully these questions have gotten you thinking about what your model\n",
"is doing and what its weaknesses and problems might be."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Part 4. Ethics, Fairness, and Model Fragility"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that you have built and tuned a classifier, it is important to consider how it might behave when deployed in a real-world humanitarian crisis. Machine learning models are not neutral; the choices made during preprocessing and the inherent nature of \"bag-of-words\" algorithms can lead to systemic biases. In this section, you will explore how these mathematical models interact with the complexities of human language."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Helper function to test custom strings against the trained model.\n",
"# Run this cell before proceeding to the questions below.\n",
"\n",
"def run_ethics_test(classifier, custom_messages):\n",
" \"\"\"\n",
" Tests custom strings against the trained Logistic Regression model.\n",
" \"\"\"\n",
" # Create Example objects and match the preprocessing used in the assignment\n",
" test_examples = [Example(words=msg.lower().split(), label=None) for msg in custom_messages]\n",
" \n",
" # Get raw sigmoid probabilities using the classify method you implemented\n",
" scores = classifier.classify(test_examples, return_scores=True)\n",
" \n",
" print(f\"{'Probability':<12} | {'Message Snippet'}\")\n",
" print(\"-\" * 75)\n",
" for msg, score in zip(custom_messages, scores):\n",
" print(f\"{score:<12.4f} | {msg[:85]}...\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Part 4.1 The Translation Gap"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In real-world disasters, messages are often written in local dialects or translated by automated systems before being classified. These translations can lose crucial context or contain spelling variations that aren't in the training set.\n",
"\n",
"**Task:** Compare the probability outputs for the \"Clean\" and \"Messy\" versions of the aid request below.\n",
"\n",
"* **Original Japanese:** \"避難所に食べ物と水が足りません。助けてください。\" \n",
"* **Message A (Clean):** \"We need food and water at the school shelter immediately.\"\n",
"* **Message B (Literal Translation):** \"Food and water are urgently needed at the school's evacuation center.\"\n",
"\n",
"**Reflection:**\n",
"1. Run the code cell below. \n",
"In the `reflection_response_4_1` function, write your response to the following questions in the `reflection` string:\n",
"3. Select a probability threshold to decide if a message is `aid` or `not`(e.g., 0.5). How could your cutoff lead to unfairness in aid allocation?\n",
"4. Can you think of applications where using logistic regression to make a prediction would be inappropriate?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"translation_messages = [\n",
" \"We need food and water at the school shelter immediately.\",\n",
" \"Food and water are urgently needed at the school's evacuation center.\"\n",
"]\n",
"\n",
"print(\"--- Translation Gap Test Results ---\")\n",
"run_ethics_test(lr_classifier, translation_messages)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO: Include your response to the 4.1 reflection below\n",
"def reflection_response_4_1():\n",
" reflection = \"\"\n",
" return reflection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Part 4.2 Semantic Misinterpretation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because Logistic Regression relies on specific word weights, it can be skewed by messages that include high-weight words regardless of their actual meaning. For example, \"food\" may be requested as a life-saving necessity, but an article could be about food as a culinary delicacy. Because the model lacks semantic context, it only sees the mathematical importance of the token.\n",
"\n",
"Use a large language model (LLM) to generate a synthetic news article that is clearly **not** about a disaster (e.g., a review of a fancy restaurant or a sports report) but is designed to \"fool\" your model into predicting a high probability for the `aid` class. Use your `top_10_features` from Part 3.2 to understand which high-weight words can be used to mislead the model.\n",
"\n",
"**Hint:** Intentionally include as many high-weight word as possible in a non-disaster context.\n",
"\n",
"**Reflection:**\n",
"1. Paste your LLM-generated article in the code cell below and run it.\n",
"Respond to the following questions in the reflection string of the `reflection_response_4_2` function:\n",
"3. What probability score did your model assign to it? \n",
"4. If an organization used this model to prioritize resources, how could \"noise\" (like luxury news or festivals) hinder the delivery of actual life-saving resources to victims?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# TODO: Paste the text generated by your LLM inside the quotes below.\n",
"llm_generated_article = [\n",
" \"\"\n",
"]\n",
"\n",
"print(\"--- Test Results ---\")\n",
"run_ethics_test(lr_classifier, llm_generated_article)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO: Include your response to the 4.2 reflection below\n",
"def reflection_response_4_2():\n",
" reflection = \"\"\n",
" return reflection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Part 5. Note on Hyperparameters"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can try adjusting the values of some of the arguments to `LogisticRegressionClassifier`, like `alpha` or `num_iterations`, re-running the evaluation, and seeing if the performance has changed.\n",
"We haven't really discussed what these values represent, but you should be able to notice that by adjusting them you can significantly change your model's performance!\n",
"\n",
"These are examples of `hyperparameters` of our model.\n",
"You can think of them as adjustable settings that control how our model works and how it learns.\n",
"Oftentimes you will want to experiment with different choices for these parameters to find ones that work best and give the best possible performance.\n",
"This optimal choice of the hyperparameters will depend on the particular dataset/task as well."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2O2n-GyiIIVT"
},
"source": [
"## Ending Remarks"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8pI3K3ewfb2g"
},
"source": [
"Congratulations, you are done with the assignment!\n",
"Refer to the [`Submitting`](#submitting) for submission instructions."
]
}
],
"metadata": {
"celltoolbar": "Tags",
"colab": {
"collapsed_sections": [],
"name": "pa2-solution.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.19"
}
},
"nbformat": 4,
"nbformat_minor": 4
}