{ "cells": [ { "cell_type": "markdown", "id": "a6b93b38-493c-4b9b-a6df-82ef055cfe00", "metadata": {}, "source": [ "# Dynascoring" ] }, { "cell_type": "code", "execution_count": 1, "id": "46c53791-fe0c-4831-ad16-0678cf171adc", "metadata": {}, "outputs": [], "source": [ "__author__ = \"Christopher Potts\"\n", "__version__ = \"CS224u, Stanford, Spring 2023\"" ] }, { "cell_type": "markdown", "id": "75adfeb6-f95c-4da5-afe5-2fd1c860273f", "metadata": {}, "source": [ "## Contents\n", "\n", "1. [Overview](#Overview)\n", "1. [Set-up](#Set-up)\n", "1. [Scoring function](#Scoring-function)\n", "1. [Example](#Example)" ] }, { "cell_type": "markdown", "id": "d442dc27-ab38-473e-836f-e07019116208", "metadata": {}, "source": [ "## Overview" ] }, { "cell_type": "markdown", "id": "21cf99da-b444-4f00-8e8f-3ac9d6164b1c", "metadata": {}, "source": [ "This notebook provides an implementation of the dynascoring method of [Ma et al 2021](https://papers.nips.cc/paper/2021/hash/55b1927fdafef39c48e5b73b5d61ea60-Abstract.html). Dynascores allow you to synthesize multiple metrics into a single score, with weights on the metrics expressing your assessment of the relative importance of the metrics. The notebook implements the function and then illustrates with an example from the paper." ] }, { "cell_type": "markdown", "id": "48d08744-5272-48ef-ab68-22eb4e9fc005", "metadata": {}, "source": [ "## Set-up" ] }, { "cell_type": "code", "execution_count": 2, "id": "fcfcac4e-d9fb-4283-ac48-22b7e658c167", "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "markdown", "id": "ce62ca2a-78f6-4ef3-b65b-70fee4f0d20b", "metadata": {}, "source": [ "## Scoring function" ] }, { "cell_type": "code", "execution_count": 3, "id": "a57ec549-26e0-4eed-8b0a-0434a4afd987", "metadata": {}, "outputs": [], "source": [ "def dynascore(\n", " data,\n", " weights,\n", " perf_metric_field_name,\n", " direction_multipliers,\n", " offsets,\n", " delta_cutoff_proportion=0.0001):\n", " \"\"\"Implementation of dynacoring.\n", "\n", " Parameters\n", " ----------\n", " data: `pd.DataFrame`\n", " Column names must include at least the keys of `weights`.\n", " weights: `dict`\n", " Map from metric names to their weights for dynascoring.\n", " perf_metric_field_name: `str`\n", " The metric in `weights` to use for performance.\n", " direction_multipliers: `pd.Series` or `None`\n", " If not `None`, then this should have the same structure as\n", " `weights` but with values 1 for no change in direction and `-1`\n", " to change direction for the metric.\n", " offsets: `pd.Series` or `None`\n", " If not `None`, then this should have the same structure as\n", " `weights` and provide an adjustment of some kind for the\n", " values in `weights`.\n", " delta_cutoff_proportion: float\n", " Default of 0.0001. This value controls the smallest scoring\n", " distinction that we retain.\n", "\n", " Returns\n", " -------\n", " pd.DataFrame containing the adjusted scores and a new column\n", " `Dynascore`.\n", "\n", " \"\"\"\n", " converted_data = data.copy(deep=True)\n", " converted_data.sort_values(perf_metric_field_name, inplace=True)\n", "\n", " metrics = weights.index\n", "\n", " # Convert the data:\n", " for metric in metrics:\n", " if direction_multipliers is not None:\n", " converted_data[metric] *= direction_multipliers[metric]\n", " if offsets is not None:\n", " converted_data[metric] += offsets[metric]\n", "\n", " converted_data[\"Dynascore\"] = 0\n", "\n", " # Normalized the weights:\n", " weights = weights / weights.sum()\n", "\n", " # We don't want small denominators to make AMRS super sensitive to\n", " # noise in the model submissions.\n", " delta = converted_data.diff()\n", " delta_threshold = (\n", " converted_data[perf_metric_field_name].max() * delta_cutoff_proportion\n", " )\n", " satisfied_indices = []\n", " for index in range(len(delta[perf_metric_field_name])):\n", " if abs(delta[perf_metric_field_name][index]) > delta_threshold:\n", " satisfied_indices.append(index)\n", "\n", " for metric in metrics:\n", " AMRS = (\n", " delta[metric][satisfied_indices].abs() / delta[perf_metric_field_name][satisfied_indices]\n", " ).mean(skipna=True)\n", " converted_data[metric] = converted_data[metric] / abs(AMRS)\n", " converted_data[\"Dynascore\"] += converted_data[metric] * weights.get(\n", " metric, 0\n", " )\n", "\n", " return converted_data.sort_values(\"Dynascore\", ascending=False)" ] }, { "cell_type": "markdown", "id": "7efcd93f-cfd9-4751-b8bb-9ad20224ffdf", "metadata": {}, "source": [ "## Example" ] }, { "cell_type": "markdown", "id": "941e32aa-9f27-4289-a48c-523ccf974117", "metadata": {}, "source": [ "These numbers are from [Ma et al. 2021](https://papers.nips.cc/paper/2021/hash/55b1927fdafef39c48e5b73b5d61ea60-Abstract.html), Table 1, top (NLI example). The output scores are somewhat different from the paper, I assume because the paper's scoring was done based on the unrounded values." ] }, { "cell_type": "code", "execution_count": 4, "id": "e8878793-aad3-4755-8dec-81d2223464d2", "metadata": {}, "outputs": [], "source": [ "data = {\n", " \"Model\": [\"DeBERTa\", \"RoBERTa\", \"ALBERT\", \"T5\", \"BERT\", \"Majority Baseline\", \"FastText\"],\n", " \"Perf\": [69.54, 69.07, 67.29, 67.16, 64.82, 32.41, 31.29],\n", " \"Throughput\": [7.41, 9.23, 9.60, 7.10, 9.39, 77.33, 73.94],\n", " \"Memory\": [5.71, 4.82, 2.18, 10.62, 4.13, 1.15, 2.20],\n", " \"Fairness\": [91.97, 90.94, 89.94, 91.89, 92.11, 100.00, 83.23],\n", " \"Robustness\": [75.70, 74.82, 74.12, 73.47, 66.38, 100.00, 69.14]\n", "}\n", "\n", "data = pd.DataFrame(data).set_index(\"Model\")" ] }, { "cell_type": "markdown", "id": "41b34cba-041d-4ccc-9c05-be1def915657", "metadata": {}, "source": [ "Here's a look at a full set-up; the required pieces are just `weights` and `perf_metric_field_name`:" ] }, { "cell_type": "code", "execution_count": 5, "id": "2c8f1781-cd7a-4d44-beec-7971508a4c98", "metadata": {}, "outputs": [], "source": [ "# The implementation normalizes the weights:\n", "weights = pd.Series({\n", " \"Perf\": 4,\n", " \"Throughput\": 1,\n", " \"Memory\": 1,\n", " \"Fairness\": 1,\n", " \"Robustness\": 1})\n", "\n", "perf_metric_field_name = \"Perf\"\n", "\n", "# All our metrics are ones we want to increase, so these values\n", "# are all 1. We could use -1 to reverse direction.\n", "direction_multipliers = pd.Series(\n", " {'Perf': 1,\n", " 'Throughput': 1,\n", " 'Memory': 1,\n", " \"Fairness\": 1,\n", " \"Robustness\": 1})\n", "\n", "offsets = pd.Series(\n", " {'Perf': 0,\n", " 'Throughput': 0,\n", " 'Memory': 16, # (16GB - memory used), as in the paper\n", " \"Fairness\": 0,\n", " \"Robustness\": 0})" ] }, { "cell_type": "markdown", "id": "2138d4bf-0414-420d-ae03-44bf8378e72e", "metadata": {}, "source": [ "Example run:" ] }, { "cell_type": "code", "execution_count": 6, "id": "daeb818e-0f04-4b53-b294-67497a8cc644", "metadata": {}, "outputs": [], "source": [ "scored = dynascore(\n", " data,\n", " weights,\n", " perf_metric_field_name=perf_metric_field_name,\n", " direction_multipliers=direction_multipliers,\n", " offsets=offsets,\n", " delta_cutoff_proportion=0.0001)" ] }, { "cell_type": "code", "execution_count": 7, "id": "b8f56fbb-e411-4252-bfd6-2de22015f89a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PerfThroughputMemoryFairnessRobustnessDynascore
Model
DeBERTa69.541.5115941.80658716.68947011.68017038.730978
RoBERTa69.071.8828631.73252716.50256011.54439038.492792
ALBERT67.291.9583401.51284016.32109311.43638337.548582
T567.161.4483562.21517116.67495311.33609137.539321
BERT64.821.9155021.67510916.71487510.24213636.228453
Majority Baseline32.4115.7748401.42712918.14664615.42955122.552271
FastText31.2915.0833011.51450415.10345310.66799220.941156
\n", "
" ], "text/plain": [ " Perf Throughput Memory Fairness Robustness \\\n", "Model \n", "DeBERTa 69.54 1.511594 1.806587 16.689470 11.680170 \n", "RoBERTa 69.07 1.882863 1.732527 16.502560 11.544390 \n", "ALBERT 67.29 1.958340 1.512840 16.321093 11.436383 \n", "T5 67.16 1.448356 2.215171 16.674953 11.336091 \n", "BERT 64.82 1.915502 1.675109 16.714875 10.242136 \n", "Majority Baseline 32.41 15.774840 1.427129 18.146646 15.429551 \n", "FastText 31.29 15.083301 1.514504 15.103453 10.667992 \n", "\n", " Dynascore \n", "Model \n", "DeBERTa 38.730978 \n", "RoBERTa 38.492792 \n", "ALBERT 37.548582 \n", "T5 37.539321 \n", "BERT 36.228453 \n", "Majority Baseline 22.552271 \n", "FastText 20.941156 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "scored" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 5 }