{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*This notebook is from https://github.com/neubig/anlp-code* by Graham Neubig\n",
    "\n",
    "We added additional printing of feature weights in the Error Analysis section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a Sentiment Classifier\n",
    "\n",
    "This is a notebook for [CMU CS11-711 Advanced NLP](https://cmu-l3.github.io/anlp-spring2026/) that trains a sentiment classifier based on data. Specifically, it uses a bag-of-words to extract features, and the structured perceptron algorithm to train the classifier.\n",
    "\n",
    "It will take in a text `X` and return a `label` of \"1\" if the sentiment of the text is positive, \"-1\" if the sentiment of the text is negative, and \"0\" if the sentiment of the text is neutral. You can test the accuracy of your classifier on the [Stanford Sentiment Treebank](http://nlp.stanford.edu/sentiment/index.html) by running the notebook all the way to end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Setup code, do imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction\n",
    "\n",
    "Feature extraction code, how do we get the features we use in training? By default we just use every word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_features(x: str) -> dict[str, float]:\n",
    "    features = {}\n",
    "    x_split = x.split(' ')\n",
    "    for x in x_split:\n",
    "        features[x] = features.get(x, 0) + 1.0\n",
    "    return features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, initialize the feature weights to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "feature_weights = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Reading\n",
    "\n",
    "Read in the data from the training and dev (or finally test) sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_xy_data(filename: str) -> tuple[list[str], list[int]]:\n",
    "    x_data = []\n",
    "    y_data = []\n",
    "    with open(filename, 'r') as f:\n",
    "        for line in f:\n",
    "            label, text = line.strip().split(' ||| ')\n",
    "            x_data.append(text)\n",
    "            y_data.append(int(label))\n",
    "    return x_data, y_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, y_train = read_xy_data('./data/train.txt')\n",
    "x_dev, y_dev = read_xy_data('./data/dev.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Rock is destined to be the 21st Century 's new `` Conan '' and that he 's going to make a splash even greater than Arnold Schwarzenegger , Jean-Claud Van Damme or Steven Segal .\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(x_train[0])\n",
    "print(y_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference Code\n",
    "\n",
    "How we run the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_classifier(features: dict[str, float]) -> int:\n",
    "    score = 0\n",
    "    for feat_name, feat_value in features.items():\n",
    "        score = score + feat_value * feature_weights.get(feat_name, 0)\n",
    "    if score > 0:\n",
    "        return 1\n",
    "    elif score < 0:\n",
    "        return -1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Code\n",
    "\n",
    "Learn the weights of the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch 1: 100%|██████████| 8544/8544 [00:00<00:00, 196120.58it/s]\n",
      "Epoch 2: 100%|██████████| 8544/8544 [00:00<00:00, 251737.79it/s]\n",
      "Epoch 3: 100%|██████████| 8544/8544 [00:00<00:00, 273312.08it/s]\n",
      "Epoch 4: 100%|██████████| 8544/8544 [00:00<00:00, 298467.80it/s]\n",
      "Epoch 5: 100%|██████████| 8544/8544 [00:00<00:00, 294862.66it/s]\n"
     ]
    }
   ],
   "source": [
    "NUM_EPOCHS = 5\n",
    "for epoch in range(1, NUM_EPOCHS+1):\n",
    "    # Shuffle the order of the data\n",
    "    data_ids = list(range(len(x_train)))\n",
    "    random.shuffle(data_ids)\n",
    "    # Run over all data points\n",
    "    for data_id in tqdm.tqdm(data_ids, desc=f'Epoch {epoch}'):\n",
    "        x = x_train[data_id]\n",
    "        y = y_train[data_id]\n",
    "        # We will skip neutral examples\n",
    "        if y == 0:    \n",
    "            continue\n",
    "        # Make a prediction\n",
    "        features = extract_features(x)\n",
    "        predicted_y = run_classifier(features)\n",
    "        # Update the weights if the prediction is wrong\n",
    "        if predicted_y != y:\n",
    "            for feature in features:\n",
    "                feature_weights[feature] = feature_weights.get(feature, 0) + y * features[feature]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Code\n",
    "\n",
    "How we evaluate the classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_accuracy(x_data: list[str], y_data: list[int]) -> float:\n",
    "    total_number = 0\n",
    "    correct_number = 0\n",
    "    for x, y in zip(x_data, y_data):\n",
    "        y_pred = run_classifier(extract_features(x))\n",
    "        total_number += 1\n",
    "        if y == y_pred:\n",
    "            correct_number += 1\n",
    "    return correct_number / float(total_number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 444, 0: 229, -1: 428}\n"
     ]
    }
   ],
   "source": [
    "label_count = {}\n",
    "for y in y_dev:\n",
    "    if y not in label_count:\n",
    "        label_count[y] = 0\n",
    "    label_count[y] += 1\n",
    "print(label_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.7437968164794008\n",
      "Dev/test accuracy: 0.5776566757493188\n"
     ]
    }
   ],
   "source": [
    "train_accuracy = calculate_accuracy(x_train, y_train)\n",
    "test_accuracy = calculate_accuracy(x_dev, y_dev)\n",
    "print(f'Train accuracy: {train_accuracy}')\n",
    "print(f'Dev/test accuracy: {test_accuracy}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error Analysis\n",
    "\n",
    "An important part of improving any system is figuring out where it goes wrong. The following two functions allow you to randomly observe some mistaken examples, which may help you improve the classifier. Feel free to write more sophisticated methods for error analysis as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_feature_contributions(features):\n",
    "    output = {}\n",
    "    for feat_name, feat_value in features.items():\n",
    "        output[feat_name] = feat_value * feature_weights.get(feat_name, 0)\n",
    "    return output\n",
    "\n",
    "def find_errors(x_data, y_data):\n",
    "    error_ids = []\n",
    "    y_preds = []\n",
    "    id2contributions = {}\n",
    "    for i, (x, y) in enumerate(zip(x_data, y_data)):\n",
    "        features = extract_features(x)\n",
    "        y_preds.append(run_classifier(features))\n",
    "        if y != y_preds[-1]:\n",
    "            error_ids.append(i)\n",
    "            id2contributions[i] = get_feature_contributions(features)\n",
    "    for _ in range(5):\n",
    "        my_id = random.choice(error_ids)\n",
    "        x, y, y_pred = x_data[my_id], y_data[my_id], y_preds[my_id]\n",
    "\n",
    "        print(f'{x}\\ntrue label: {y}\\npredicted label: {y_pred}')\n",
    "        contributions = sorted(id2contributions[my_id].items(), key=lambda x: -x[1])\n",
    "        for feat_name, contribution in contributions:\n",
    "            print(f'Feature: {feat_name} ({contribution})')\n",
    "        \n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prurient playthings aside , there 's little to love about this English trifle .\n",
      "true label: -1\n",
      "predicted label: 0\n",
      "Feature: love (6.0)\n",
      "Feature: English (6.0)\n",
      "Feature: there (4.0)\n",
      "Feature: aside (1.0)\n",
      "Feature: Prurient (0.0)\n",
      "Feature: playthings (0.0)\n",
      "Feature: this (0.0)\n",
      "Feature: 's (-1.0)\n",
      "Feature: to (-1.0)\n",
      "Feature: . (-1.0)\n",
      "Feature: , (-2.0)\n",
      "Feature: trifle (-3.0)\n",
      "Feature: about (-4.0)\n",
      "Feature: little (-5.0)\n",
      "\n",
      "Sam Mendes has become valedictorian at the School for Soft Landings and Easy Ways Out .\n",
      "true label: -1\n",
      "predicted label: 1\n",
      "Feature: Out (7.0)\n",
      "Feature: Mendes (4.0)\n",
      "Feature: for (4.0)\n",
      "Feature: and (4.0)\n",
      "Feature: has (3.0)\n",
      "Feature: Sam (1.0)\n",
      "Feature: valedictorian (0.0)\n",
      "Feature: the (0.0)\n",
      "Feature: School (0.0)\n",
      "Feature: Soft (0.0)\n",
      "Feature: Landings (0.0)\n",
      "Feature: Easy (0.0)\n",
      "Feature: Ways (0.0)\n",
      "Feature: . (-1.0)\n",
      "Feature: at (-3.0)\n",
      "Feature: become (-4.0)\n",
      "\n",
      "Not since Tom Cruise in Risky Business has an actor made such a strong impression in his underwear .\n",
      "true label: 1\n",
      "predicted label: -1\n",
      "Feature: an (4.0)\n",
      "Feature: a (4.0)\n",
      "Feature: since (3.0)\n",
      "Feature: has (3.0)\n",
      "Feature: strong (3.0)\n",
      "Feature: his (1.0)\n",
      "Feature: Not (0.0)\n",
      "Feature: Cruise (0.0)\n",
      "Feature: Risky (0.0)\n",
      "Feature: Business (0.0)\n",
      "Feature: such (0.0)\n",
      "Feature: Tom (-1.0)\n",
      "Feature: actor (-1.0)\n",
      "Feature: underwear (-1.0)\n",
      "Feature: . (-1.0)\n",
      "Feature: made (-2.0)\n",
      "Feature: impression (-3.0)\n",
      "Feature: in (-10.0)\n",
      "\n",
      "This film seems thirsty for reflection , itself taking on adolescent qualities .\n",
      "true label: -1\n",
      "predicted label: 1\n",
      "Feature: for (4.0)\n",
      "Feature: film (3.0)\n",
      "Feature: taking (3.0)\n",
      "Feature: on (2.0)\n",
      "Feature: This (1.0)\n",
      "Feature: thirsty (0.0)\n",
      "Feature: adolescent (0.0)\n",
      "Feature: qualities (0.0)\n",
      "Feature: reflection (-1.0)\n",
      "Feature: . (-1.0)\n",
      "Feature: , (-2.0)\n",
      "Feature: seems (-3.0)\n",
      "Feature: itself (-3.0)\n",
      "\n",
      "A morose little soap opera about three vapid , insensitive people who take turns hurting each other .\n",
      "true label: 0\n",
      "predicted label: -1\n",
      "Feature: take (4.0)\n",
      "Feature: who (3.0)\n",
      "Feature: each (2.0)\n",
      "Feature: opera (0.0)\n",
      "Feature: insensitive (0.0)\n",
      "Feature: hurting (0.0)\n",
      "Feature: morose (-1.0)\n",
      "Feature: turns (-1.0)\n",
      "Feature: . (-1.0)\n",
      "Feature: A (-2.0)\n",
      "Feature: three (-2.0)\n",
      "Feature: , (-2.0)\n",
      "Feature: other (-3.0)\n",
      "Feature: soap (-4.0)\n",
      "Feature: about (-4.0)\n",
      "Feature: people (-4.0)\n",
      "Feature: little (-5.0)\n",
      "Feature: vapid (-6.0)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "find_errors(x_dev, y_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize feature weights\n",
    "\n",
    "We can inspect the weights that were learned for various features. Below we show the largest, smallest, and randomly selected feature weights. Inspecting them may give insight into the learned classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top-k\n",
      "('refreshing', 13.0)\n",
      "('powerful', 13.0)\n",
      "('remarkable', 12.0)\n",
      "('solid', 12.0)\n",
      "('rare', 12.0)\n",
      "('appealing', 12.0)\n",
      "('fun', 11.0)\n",
      "('charming', 11.0)\n",
      "('ability', 10.0)\n",
      "('eyes', 10.0)\n",
      "('works', 10.0)\n",
      "('manages', 10.0)\n",
      "('heart', 10.0)\n",
      "('wonderfully', 10.0)\n",
      "('always', 10.0)\n",
      "('wild', 10.0)\n",
      "('assured', 10.0)\n",
      "('summer', 10.0)\n",
      "('enjoyable', 9.0)\n",
      "('beautifully', 9.0)\n",
      "('sharp', 9.0)\n",
      "('amused', 9.0)\n",
      "('human', 9.0)\n",
      "('sweet', 9.0)\n",
      "('brilliant', 9.0)\n",
      "\n",
      "Bottom-k\n",
      "('stupid', -17.0)\n",
      "('worst', -16.0)\n",
      "('depressing', -13.0)\n",
      "('pretentious', -13.0)\n",
      "('devoid', -13.0)\n",
      "('flat', -12.0)\n",
      "('contrived', -12.0)\n",
      "('TV', -12.0)\n",
      "('none', -12.0)\n",
      "('tired', -11.0)\n",
      "('failure', -11.0)\n",
      "('dull', -10.0)\n",
      "('scene', -10.0)\n",
      "('instead', -10.0)\n",
      "('cold', -10.0)\n",
      "('mess', -10.0)\n",
      "('loses', -10.0)\n",
      "('uneven', -10.0)\n",
      "('Lawrence', -10.0)\n",
      "('harder', -10.0)\n",
      "('left', -9.0)\n",
      "('rises', -9.0)\n",
      "('suffers', -9.0)\n",
      "('too', -9.0)\n",
      "('advice', -9.0)\n",
      "\n",
      "Random k\n",
      "('sends', 5.0)\n",
      "('kills', -1.0)\n",
      "('beer', -2.0)\n",
      "('coherence', -3.0)\n",
      "('Native', 1.0)\n",
      "('tattoo', 1.0)\n",
      "('Elfriede', -1.0)\n",
      "('wonderful', 6.0)\n",
      "('predicament', 2.0)\n",
      "('Schwarzenegger', -1.0)\n",
      "('Del', -2.0)\n",
      "('self-esteem', 1.0)\n",
      "('diabolical', 0.0)\n",
      "('protestors', -2.0)\n",
      "('sinuously', -1.0)\n",
      "('Invigorating', 1.0)\n",
      "('favors', -2.0)\n",
      "('cleavage', 2.0)\n",
      "('scrape', -2.0)\n",
      "('naught', -2.0)\n",
      "('shame', -2.0)\n",
      "('Children', 3.0)\n",
      "('roll', 6.0)\n",
      "('dumbness', -1.0)\n",
      "('snaps', -1.0)\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "k = 25\n",
    "topk_features = sorted(feature_weights.items(), key=lambda x: -x[1])[:k]\n",
    "bottomk_features = sorted(feature_weights.items(), key=lambda x: x[1])[:k]\n",
    "randomk_features = random.sample(list(feature_weights.items()), k)\n",
    "\n",
    "print(\"Top-k\")\n",
    "for feature in topk_features:\n",
    "    print(feature)\n",
    "\n",
    "print(\"\\nBottom-k\")\n",
    "for feature in bottomk_features:\n",
    "    print(feature)\n",
    "\n",
    "print(\"\\nRandom k\")\n",
    "for feature in randomk_features:\n",
    "    print(feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "prototype",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}