{ "cells": [ { "cell_type": "markdown", "id": "header", "metadata": {}, "source": [ "# Basic Power Analysis \n", "\n", "Lecture 14 | CMU ANLP Fall 2025 | Instructor: Sean Welleck\n", "\n", "Power analysis helps determine the required sample size to detect a meaningful difference between two systems (e.g., model A vs. model B) with sufficient statistical confidence.\n", "\n", "*Based on the methodology in \"Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations\" ([arXiv:2411.00640](https://arxiv.org/pdf/2411.00640))*" ] }, { "cell_type": "markdown", "id": "background", "metadata": {}, "source": [ "## Background\n", "\n", "- **Minimum detectable effect (δ)**: The smallest difference we care about detecting\n", "- **Power (1-β)**: The probability of detecting a true difference when it exists (typically 0.80)\n", "- **Significance level (α)**: The probability of falsely detecting a difference (typically 0.05)\n", "- **Required sample size (n)**: How many test examples we need\n" ] }, { "cell_type": "markdown", "id": "setup-header", "metadata": {}, "source": [ "### Setup and utility functions" ] }, { "cell_type": "code", "execution_count": 22, "id": "imports", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from scipy.stats import norm" ] }, { "cell_type": "markdown", "id": "sim-header", "metadata": {}, "source": [ "### Simulating paired binary scores\n", "\n", "For binary outcomes (correct/incorrect), we simulate paired scores with a specified correlation structure." ] }, { "cell_type": "code", "execution_count": 23, "id": "sim-func", "metadata": {}, "outputs": [], "source": [ "def simulate_paired_scores_with_corr(n, mu_A, mu_B, rho, rng=None):\n", " if rng is None:\n", " rng = np.random.default_rng()\n", " cov = np.array([[1.0, rho],[rho, 1.0]])\n", " z = rng.multivariate_normal(mean=[0,0], cov=cov, size=n)\n", " tA = norm.ppf(mu_A)\n", " tB = norm.ppf(mu_B)\n", " scores_A = (z[:,0] <= tA).astype(int)\n", " scores_B = (z[:,1] <= tB).astype(int)\n", " return scores_A, scores_B" ] }, { "cell_type": "markdown", "id": "omega-header", "metadata": {}, "source": [ "### Estimating variance from pilot data\n", "\n", "The variance of the paired differences (ω²) is a key parameter for power analysis. We estimate it from a pilot study." ] }, { "cell_type": "code", "execution_count": 24, "id": "omega-func", "metadata": {}, "outputs": [], "source": [ "def estimate_omega2(sA, sB):\n", " d = sA.astype(float) - sB.astype(float)\n", " return float(np.var(d, ddof=1))" ] }, { "cell_type": "markdown", "id": "reqn-header", "metadata": {}, "source": [ "### Computing required sample size\n", "\n", "Given a desired minimum detectable effect size δ, significance level α, power, and variance estimate ω², we can compute the required number of test examples." ] }, { "cell_type": "code", "execution_count": 26, "id": "reqn-func", "metadata": {}, "outputs": [], "source": [ "def required_n(delta: float, alpha: float, power: float, omega2: float) -> int:\n", " beta = 1 - power\n", " z_alpha2 = norm.ppf(1 - alpha/2.0)\n", " z_beta = norm.ppf(1 - beta)\n", " n = (z_alpha2 + z_beta)**2 * (omega2) / (delta**2)\n", " return int(np.ceil(n))" ] }, { "cell_type": "markdown", "id": "example-header", "metadata": {}, "source": [ "## Example: Determining sample size for a model comparison\n", "\n", "Suppose we have two models with similar accuracy (~86%), and we want to detect differences as small as 1-3%. Let's use a pilot study to estimate the variance and determine the required sample size." ] }, { "cell_type": "code", "execution_count": 27, "id": "pilot", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pilot study results:\n", " Sample size: 156\n", " System A accuracy: 0.8718\n", " System B accuracy: 0.8718\n", " Estimated ω²: 0.1161\n" ] } ], "source": [ "# Pilot study parameters\n", "mu_A = 0.86 # System A accuracy\n", "mu_B = 0.86 # System B accuracy \n", "n_pilot = 156 # Pilot study size\n", "\n", "# Simulate pilot data with correlation rho=0.8\n", "scores_A, scores_B = simulate_paired_scores_with_corr(n_pilot, mu_A, mu_B, rho=0.8)\n", "\n", "# Estimate variance from pilot\n", "omega2_hat = estimate_omega2(scores_A, scores_B)\n", "\n", "print(\"Pilot study results:\")\n", "print(f\" Sample size: {n_pilot}\")\n", "print(f\" System A accuracy: {scores_A.mean():.4f}\")\n", "print(f\" System B accuracy: {scores_B.mean():.4f}\")\n", "print(f\" Estimated ω²: {omega2_hat:.4f}\")" ] }, { "cell_type": "markdown", "id": "table-header", "metadata": {}, "source": [ "### Computing required sample sizes for different effect sizes" ] }, { "cell_type": "code", "execution_count": 28, "id": "table", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
delta (min detectable difference)required_n (paired items)
00.0109115
10.0154052
20.0202279
30.0301013
\n", "
" ], "text/plain": [ " delta (min detectable difference) required_n (paired items)\n", "0 0.010 9115\n", "1 0.015 4052\n", "2 0.020 2279\n", "3 0.030 1013" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Standard parameters\n", "alpha = 0.05\n", "power = 0.80\n", "\n", "# Different minimum detectable differences to consider\n", "deltas = [0.01, 0.015, 0.02, 0.03]\n", "\n", "# Compute required n for each delta\n", "rows = []\n", "for delta in deltas:\n", " rows.append({\n", " \"delta (min detectable difference)\": delta,\n", " \"required_n (paired items)\": required_n(delta, alpha, power, omega2_hat)\n", " })\n", "\n", "summary = pd.DataFrame(rows)\n", "summary" ] }, { "cell_type": "code", "execution_count": null, "id": "7280f6b7", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.9.0" } }, "nbformat": 4, "nbformat_minor": 5 }