{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "header",
   "metadata": {},
   "source": [
    "# Inter-Annotator Agreement (IAA)\n",
    "\n",
    "Lecture 14 | CMU ANLP Fall 2025 | Instructor: Sean Welleck\n",
    "\n",
    "This notebook shows a simple example of computing the Cohen's Kappa inter-annotator agreement (IAA) metric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "background",
   "metadata": {},
   "source": [
    "## Background\n",
    "\n",
    "When multiple annotators label the same data, we need to measure their agreement to assess annotation quality. A common metric is **Cohen's Kappa (κ)**, defined as:\n",
    "  \n",
    "  $$\\kappa = \\frac{P_o - P_e}{1 - P_e}$$\n",
    "  \n",
    "  where $P_e$ is the expected agreement by chance, and $P_o$ is the proportion of items where annotators agree."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "setup-header",
   "metadata": {},
   "source": [
    "### Setup and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "imports",
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import annotations\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "confmat-header",
   "metadata": {},
   "source": [
    "### Building a confusion matrix\n",
    "\n",
    "The first step in computing agreement is to build a confusion matrix showing how often each label from Rater 1 co-occurs with each label from Rater 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "confmat-func",
   "metadata": {},
   "outputs": [],
   "source": [
    "def confusion_matrix_from_labels(y1, y2, label_order=None):\n",
    "    y1 = np.asarray(y1)\n",
    "    y2 = np.asarray(y2)\n",
    "    if y1.shape[0] != y2.shape[0]:\n",
    "        raise ValueError(\"y1 and y2 must have the same length.\")\n",
    "    if label_order is None:\n",
    "        labels = sorted(list(set(list(y1) + list(y2))))\n",
    "    else:\n",
    "        labels = list(label_order)\n",
    "    \n",
    "    idx = {lab: i for i, lab in enumerate(labels)}\n",
    "    C = np.zeros((len(labels), len(labels)), dtype=int)\n",
    "    for a, b in zip(y1, y2):\n",
    "        C[idx[a], idx[b]] += 1\n",
    "    \n",
    "    dfC = pd.DataFrame(\n",
    "        C, \n",
    "        index=pd.Index(labels, name=\"Rater 1\"), \n",
    "        columns=pd.Index(labels, name=\"Rater 2\")\n",
    "    )\n",
    "    return dfC, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "kappa-header",
   "metadata": {},
   "source": [
    "### Computing Cohen's Kappa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "kappa-func",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cohen_kappa_from_confmat(C: np.ndarray | pd.DataFrame):\n",
    "    if isinstance(C, pd.DataFrame):\n",
    "        C = C.values\n",
    "    \n",
    "    n = C.sum()\n",
    "    if n == 0:\n",
    "        raise ValueError(\"Empty confusion matrix.\")\n",
    "    \n",
    "    # Observed agreement: diagonal entries\n",
    "    Po = (np.trace(C)) / n\n",
    "    \n",
    "    # Chance agreement: product of marginal proportions\n",
    "    r_marg = C.sum(axis=1) / n  # rater1 marginal proportions\n",
    "    c_marg = C.sum(axis=0) / n  # rater2 marginal proportions\n",
    "    Pe = float(np.dot(r_marg, c_marg))  # sum_k p1_k * p2_k\n",
    "    \n",
    "    denom = 1 - Pe\n",
    "    if denom == 0:\n",
    "        kappa = np.nan  # undefined when marginals are degenerate\n",
    "    else:\n",
    "        kappa = (Po - Pe) / denom\n",
    "    \n",
    "    return kappa, Po, Pe\n",
    "\n",
    "def cohen_kappa(y1, y2, label_order=None, return_conf_mat=False):\n",
    "    C, labels = confusion_matrix_from_labels(y1, y2, label_order=label_order)\n",
    "    kappa, Po, Pe = cohen_kappa_from_confmat(C)\n",
    "    if return_conf_mat:\n",
    "        return kappa, Po, Pe, C\n",
    "    return kappa, Po, Pe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bootstrap-header",
   "metadata": {},
   "source": [
    "### Bootstrap confidence intervals\n",
    "\n",
    "To quantify uncertainty in our kappa estimate, we can use the bootstrap to compute confidence intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bootstrap-func",
   "metadata": {},
   "outputs": [],
   "source": [
    "def bootstrap_ci_kappa(y1, y2, B=5000, alpha=0.05, label_order=None, random_state=0):\n",
    "    rng = np.random.default_rng(random_state)\n",
    "    y1 = np.asarray(y1)\n",
    "    y2 = np.asarray(y2)\n",
    "    n = len(y1)\n",
    "    if n != len(y2):\n",
    "        raise ValueError(\"y1 and y2 must have same length.\")\n",
    "    \n",
    "    kappas = np.empty(B)\n",
    "    for b in range(B):\n",
    "        idx = rng.integers(0, n, size=n)\n",
    "        kb, _, _ = cohen_kappa(y1[idx], y2[idx], label_order=label_order, return_conf_mat=False)\n",
    "        kappas[b] = kb\n",
    "    \n",
    "    lo = np.quantile(kappas, alpha/2)\n",
    "    hi = np.quantile(kappas, 1 - alpha/2)\n",
    "    return float(lo), float(hi)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "example-header",
   "metadata": {},
   "source": [
    "## Example: Sentiment annotation task\n",
    "\n",
    "Suppose two annotators labeled 674 tweets as either positive (1) or not-positive (0). Let's compute their agreement.\n",
    "\n",
    "For demonstration purposes, we'll simulate plausible annotation data with moderate agreement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "simulate",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Simulated 674 annotations\n",
      "Rater 1 positive rate: 0.497\n",
      "Rater 2 positive rate: 0.519\n"
     ]
    }
   ],
   "source": [
    "# Simulate a plausible 674-item annotation scenario with moderate agreement\n",
    "n = 674\n",
    "rng = np.random.default_rng(2025)\n",
    "\n",
    "# Balanced underlying sentiment for demo\n",
    "true_label = rng.binomial(1, 0.5, size=n)\n",
    "\n",
    "# Rater error rates (demo): rater1 10% error, rater2 12% error\n",
    "# Build errors via a latent factor for mild dependence\n",
    "latent = rng.normal(size=n)\n",
    "err1 = (latent + rng.normal(scale=1.0, size=n) > 2.0)  # low error prob\n",
    "err2 = (latent + rng.normal(scale=1.0, size=n) > 1.8)\n",
    "\n",
    "rater1 = np.where(err1, 1-true_label, true_label)\n",
    "rater2 = np.where(err2, 1-true_label, true_label)\n",
    "\n",
    "print(f\"Simulated {n} annotations\")\n",
    "print(f\"Rater 1 positive rate: {rater1.mean():.3f}\")\n",
    "print(f\"Rater 2 positive rate: {rater2.mean():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compute-header",
   "metadata": {},
   "source": [
    "### Computing agreement metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "compute",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Confusion Matrix:\n",
      "Rater 2    0    1\n",
      "Rater 1          \n",
      "0        293   46\n",
      "1         31  304\n",
      "\n",
      "Agreement Metrics:\n",
      "  Observed agreement (P₀): 0.8858\n",
      "  Chance agreement (Pₑ):   0.4999\n",
      "  Cohen's kappa (κ):       0.7716\n",
      "  95% CI:                  [0.7239, 0.8161]\n"
     ]
    }
   ],
   "source": [
    "# Compute kappa and confusion matrix\n",
    "kappa, Po, Pe, C = cohen_kappa(rater1, rater2, label_order=[0,1], return_conf_mat=True)\n",
    "\n",
    "# Compute bootstrap CI\n",
    "ci_lo, ci_hi = bootstrap_ci_kappa(rater1, rater2, B=3000, alpha=0.05, label_order=[0,1], random_state=7)\n",
    "\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(C)\n",
    "print(f\"\\nAgreement Metrics:\")\n",
    "print(f\"  Observed agreement (P₀): {Po:.4f}\")\n",
    "print(f\"  Chance agreement (Pₑ):   {Pe:.4f}\")\n",
    "print(f\"  Cohen's kappa (κ):       {kappa:.4f}\")\n",
    "print(f\"  95% CI:                  [{ci_lo:.4f}, {ci_hi:.4f}]\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}