{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Deep Transition Dependency Parser in PyTorch\n", "\n", "- In this problem set, you will implement a deep transition dependency parser in [PyTorch](https://pytorch.org).\n", "\n", "- You will see how more complicated network architectures can be used to effectively solve a structured prediction problem.\n", "\n", "You will:\n", "\n", "- Implement an arc-standard transition-based dependency parser in PyTorch\n", "- Implement various methods of computing word embeddings\n", "- Implement neural network components for choosing actions and combining stack elements\n", "- Train your network to parse English and Norwegian sentences" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 0. Setup\n", "\n", "In order to develop this assignment, you will need [python 3.6](https://www.python.org/downloads/) and the following libraries. Most if not all of these are part of [anaconda](https://www.continuum.io/downloads), so a good starting point would be to install that.\n", "\n", "- [jupyter](http://jupyter.readthedocs.org/en/latest/install.html)\n", "- [numpy](https://docs.scipy.org/doc/numpy/user/install.html)\n", "- [matplotlib](http://matplotlib.org/users/installing.html)\n", "- [nosetests](https://nose.readthedocs.org/en/latest/)\n", "- [pytorch](https://pytorch.org)\n", "\n", "Here is some help on installing packages in python: https://packaging.python.org/installing/. You can use ```pip --user``` to install locally without sudo." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## About this assignment\n", "\n", "- This is a Jupyter notebook. You can execute cell blocks by pressing control-enter.\n", "- Most of your coding will be in the python source files in the directory ```gtnlplib```.\n", "- The directory ```tests``` contains unit tests that will be used to grade your assignment, using ```nosetests```. You should run them as you work on the assignment to see that you're on the right track. You are free to look at their source code, if that helps -- though most of the relevant code is also here in this notebook. Learn more about running unit tests at http://pythontesting.net/framework/nose/nose-introduction/\n", "- You may want to add more tests, but that is completely optional. \n", "- **To submit this assignment, run the script ```make-submission.sh```, and submit the tarball ```pset3-submission.tgz``` on Canvas.**" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import torch\n", "import torch.optim as optim\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.autograd as ag\n", "\n", "import nose\n", "import numpy as np\n", "\n", "from imp import reload" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "My library versions\n", "numpy: 1.13.3\n", "nose: 1.3.7\n", "torch: 0.3.0.post4\n" ] } ], "source": [ "print('My library versions')\n", "\n", "print('numpy: {}'.format(np.__version__))\n", "print('nose: {}'.format(nose.__version__))\n", "print('torch: {}'.format(torch.__version__))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To test whether your libraries are the right version, run:\n", "\n", "`nosetests tests/test_environment.py`" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ".\r\n", "----------------------------------------------------------------------\r\n", "Ran 1 test in 0.001s\r\n", "\r\n", "OK\r\n" ] } ], "source": [ "# use ! to run shell commands in notebook\n", "! nosetests tests/test_environment.py" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import gtnlplib.parsing as parsing\n", "import gtnlplib.data_tools as data_tools\n", "import gtnlplib.constants as consts\n", "import gtnlplib.evaluation as evaluation\n", "import gtnlplib.utils as utils\n", "import gtnlplib.feat_extractors as feat_extractors\n", "import gtnlplib.neural_net as neural_net" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Read in the datasets\n", "reload(data_tools)\n", "en_dataset = data_tools.Dataset(consts.EN_TRAIN_FILE, consts.EN_DEV_FILE, consts.EN_TEST_FILE)\n", "nr_dataset = data_tools.Dataset(consts.NR_TRAIN_FILE, consts.NR_DEV_FILE, consts.NR_TEST_FILE)\n", "\n", "# Assign each word a unique index, including two special tokens needed for parsing logic\n", "word_to_ix_en = { word: i for i, word in enumerate(en_dataset.vocab) }\n", "word_to_ix_nr = { word: i for i, word in enumerate(nr_dataset.vocab) }" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Some constants to keep around\n", "LSTM_NUM_LAYERS = 1\n", "TEST_EMBEDDING_DIM = 4\n", "WORD_EMBEDDING_DIM = 64\n", "STACK_EMBEDDING_DIM = 100\n", "NUM_FEATURES = 3\n", "\n", "# Hyperparameters\n", "ETA_0 = 0.01\n", "DROPOUT = 0.0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# High-Level Overview of the Parser\n", "Be sure that you have reviewed the notes on transition-based dependency parsing, and are familiar with the relevant terminology.\n", "Parsing will proceed as follows:\n", "* Initialize your parsing stack and input buffer.\n", "* At each step, until the parse is done:\n", " * Extract some features. We will start with simple features, but these can be anything: words in the sentence, the configuration of the stack, the configuration of the input buffer, the previous action, etc.\n", " * Send these features through a feed-forward (FF) network to get a probability distribution over actions (`SHIFT`, `ARC_L`, `ARC_R`). The next action you choose is the one with the highest probability.\n", " * If the action is an arc- operation, you use a neural network to combine the two items in the operation and get a dense output to place back on the input buffer." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The key classes you will fill in code for are:\n", "* Feature extraction in `feat_extractors.py`\n", "* The `ParserState` class, which keeps track of the input buffer and parse stack, and offers a public interface for doing the parsing actions to update the state\n", "* The `TransitionParser` class, which is a PyTorch module where the core parsing logic resides, in `parsing.py`.\n", "* The neural network components in `neural_net.py`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The network components are compartmentalized as follows:\n", "* `TransitionParser`, the base component that contains and coordinates the other substitutable components\n", "\n", "* Embedding Lookup: You will implement three flavors of embeddings. These embeddings are used to initialize the input buffer, and will be shifted on the stack / serve as inputs to the combiner networks.\n", " - `VanillaWordEmbedding` just gets embeddings from a lookup table, as we used in pset 2.\n", " - `BiLSTMWordEmbedding` will run a sequence model in both directions over the sentence. The hidden state at step t is the embedding for the `t`-th word of the sentence.\n", " - `SuffixAndWordEmbedding` gets embeddings for words as in the vanilla embeddings, and also gets embeddings for word suffixes, and concatenates t\n", "* Action Choosing: You will implement two action choosing components:\n", " - `FFActionChooser` is a simple feed-forward neural network that outputs log probabilities over the three actions given the extracted features as input.\n", " - `LSTMActionChooser` applies a sequence model that takes the hidden state of the previous action decision as input.\n", "\n", "* Combiners: You will implement two combiners, which are the network components that take the two embeddings of the items in an arc- operation and creates a single vector.\n", " - `FFCombiner` takes the two input embeddings and gives a dense output\n", " - `LSTMCombiner` applies a sequence model, where the output embedding is the hidden state of the next timestep." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parsing example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following is how the input buffer and stack look at each step of a parse, up to the first arc. The input sentence is \"the dog ran away\". Our action chooser network takes the top element of the stack, the top element of the input buffer, plus a one-token \"lookahead\" in the input buffer. $C(x,y)$ refers to calling our combiner network on arguments $x, y$. Also let $A$ be the set of actions: $\\{ \\text{SHIFT}, \\text{ARC-L}, \\text{ARC-R} \\}$, and let $q_w$ be the embedding for word $w$.\n", "\n", "1. \n", " * Input Buffer: $\\left[ q_\\text{the}, q_\\text{dog}, q_\\text{ran}, q_\\text{away}, q_\\text{END-INPUT} \\right]$\n", " * Stack: $\\left[ q_\\text{ROOT} \\right]$\n", " * Action: $ \\text{argmax}_{a \\in A} \\ \\text{ActionChooser}(q_\\text{ROOT}, q_\\text{the}, \\overbrace{q_\\text{dog}}^\\text{lookahead}) \\Rightarrow \\text{SHIFT}$\n", " \n", "2.\n", " * Input Buffer: $\\left[ q_\\text{dog}, q_\\text{ran}, q_\\text{away}, q_\\text{END-INPUT} \\right]$\n", " * Stack: $\\left[ q_\\text{ROOT}, q_\\text{the} \\right]$\n", " * Action: $ \\text{argmax}_{a \\in A} \\ \\text{ActionChooser}(q_\\text{the}, q_\\text{dog}, q_\\text{ran}) \\Rightarrow \\text{ARC-L}$\n", " \n", "3.\n", " * Input Buffer: $\\left[C(q_\\text{dog}, q_\\text{the}), q_\\text{ran}, q_\\text{away}, q_\\text{END-INPUT} \\right]$\n", " * Stack: $\\left[ q_\\text{ROOT} \\right]$\n", " \n", "This is a partial picture of parsing - we keep more than just the embedding on the stack and input buffer. We also keep the word and its position in the sentence so that when we create an arc, we know what edge was just created.\n", "So, for example, the initial input buffer really looks like\n", "\n", "$$ \\left[ (\\text{the}, 0, q_\\text{the}), (\\text{dog}, 1, q_\\text{dog}), (\\text{ran}, 2, q_\\text{ran}), (\\text{away}, 3, q_\\text{away}), (\\text{END-INPUT}, 4, q_\\text{END-INPUT}) \\right] $$\n", "\n", "Before beginning, I recommend completing the parse by hand, drawing the input buffer and stack at each step, and explicity listing the arguments to the action chooser." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1. Managing and Updating the Parser State \n", "### (4650: 2.5 points, 7650: 1.5 points)\n", "\n", "In this part of the assignment, you will work with the ParserState class, that keeps track of the parser's input buffer and stack." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 1.1: Implementing Arc\n", "\n", "#### 1.1a: Get arc components (0.25 points)\n", "You will implement the generalized arc- operation of the `ParserState` in `parsing.py`, in the function `_arc`, in two parts.\n", "\n", "First, fill in `_get_arc_components` in `parsing.py` to select the head and modifier according to the action passed in. This method should also remove the items from the stack and input buffer.\n", "Arc actions follow the arc-standard procedure, from section 10.3.1 in the course notes.\n", "\n", "- **Test**: ` test_parser.py:test_get_arc_components_d1_1a`" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(parsing)\n", "test_sentence = \"The man ran away\".split()\n", "parser_state = parsing.ParserState(test_sentence + [consts.END_OF_INPUT_TOK], \n", " [None] * (len(test_sentence)+1),\n", " utils.DummyCombiner())" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stack: ['', 'The', 'man']\n", "Input Buffer: ['ran', 'away', '']\n", "\n", "StackEntry(headword='ran', headword_pos=2, embedding=None) StackEntry(headword='man', headword_pos=1, embedding=None)\n", "StackEntry(headword='The', headword_pos=0, embedding=None) StackEntry(headword='away', headword_pos=3, embedding=None)\n" ] } ], "source": [ "parser_state.shift()\n", "parser_state.shift()\n", "print(parser_state)\n", "\n", "head, modifier = parser_state._get_arc_components(consts.Actions.ARC_L)\n", "print(head, modifier)\n", "\n", "head, modifier = parser_state._get_arc_components(consts.Actions.ARC_R)\n", "print(head, modifier)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.1b: Create the arc (0.25 points)\n", "Now, fill in `_create_arc` in `parsing.py` to use the `ParserState`'s `combiner` component to **combine** the passed in head and modifier, put the combination on the input buffer, and create a new dependency graph edge. At this point, we are just using a dummy combiner so we can test the logic.\n", "\n", "You will want to familiarize yourself with the `StackEntry` and `DepGraphEdge` objects used by the `ParserState` object for this one.\n", "\n", "- **Test**: ` test_parser.py:test_create_arc_d1_1b`" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stack: ['']\n", "Input Buffer: ['The', 'man', 'ran', 'away', '']\n", "\n", "Stack: ['', 'The']\n", "Input Buffer: ['man', 'ran', 'away', '']\n", "\n", "First arc: Head: ('man', 1), Modifier: ('The', 0) \n", "\n", "Stack: ['']\n", "Input Buffer: ['man', 'ran', 'away', '']\n", "\n", "Second arc: Head: ('ran', 2), Modifier: ('man', 1) \n", "\n", "Stack: ['']\n", "Input Buffer: ['ran', 'away', '']\n", "\n" ] } ], "source": [ "reload(parsing)\n", "parser_state = parsing.ParserState(test_sentence + [consts.END_OF_INPUT_TOK], \n", " [None] * (len(test_sentence)+1),\n", " utils.DummyCombiner())\n", "\n", "print(parser_state)\n", "\n", "parser_state.shift()\n", "print(parser_state)\n", "\n", "arc = parser_state.arc_left()\n", "print(\"First arc: Head: {}, Modifier: {}\".format(arc[0], arc[1]), \"\\n\")\n", "print(parser_state)\n", "\n", "parser_state.shift()\n", "arc = parser_state.arc_left()\n", "print(\"Second arc: Head: {}, Modifier: {}\".format(arc[0], arc[1]), \"\\n\")\n", "print(parser_state)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 1.2: Parser Terminating Condition (4650: 1 point, 7650: 0.5 points)\n", "In this short (one line) deliverable, implement `done_parsing()` in `ParserState`. Think about what the input buffer and stack look like at the end of a parse.\n", "\n", "- **Test**: `test_parsing.py:test_stack_terminating_cond_d1_2`" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "False\n", "False\n", "False\n", "True\n" ] } ], "source": [ "reload(parsing)\n", "parser_state = parsing.ParserState(test_sentence + [consts.END_OF_INPUT_TOK], \n", " [None] * (len(test_sentence)+1),\n", " utils.DummyCombiner())\n", "\n", "parser_state.shift()\n", "parser_state.arc_left()\n", "parser_state.shift()\n", "parser_state.arc_left()\n", "\n", "print(parser_state.done_parsing())\n", "\n", "parser_state.shift()\n", "parser_state.arc_right()\n", "print(parser_state.done_parsing())\n", "\n", "parser_state.arc_right()\n", "print(parser_state.done_parsing())\n", "\n", "parser_state.shift()\n", "print(parser_state.done_parsing())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 1.3: Validating parser actions (4650: 1 point, 7650: 0.5 points)\n", "Implement the `_validate_action` method in `parsing.TransitionParser`. This will be used in the prediction setting, when the gold standard is not available. We need to ensure that any action we take is legal. Here are the rules:\n", "\n", "- You cannot shift when the input buffer has <= 2 items on it (including the end of input token), UNLESS the stack is empty.\n", " - **In this case, do `ARC_R` by default.**\n", "- You cannot do an arc- operation when the stack is empty (this will happen after creating an arc with ROOT).\n", " - **In this case, do `SHIFT` by default.**\n", "- You cannot do an arc-left operation when the root token is on top of the stack.\n", " - **In this case, do `SHIFT` or `ARC-R` depending on the state of the input buffer.**\n", " \n", "**Test:**\n", "- `test_parser.py:test_validate_action_d1_3`\n", "\n", "**Make sure you pass the test before you move on. The code blocks below are not meant to be comprehensive.**" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(parsing)\n", "parser_state = parsing.ParserState(test_sentence + [consts.END_OF_INPUT_TOK], \n", " [None] * (len(test_sentence)+1),\n", " utils.DummyCombiner())\n", "ix_to_action = consts.Actions.ix_to_action" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stack: ['']\n", "Input Buffer: ['The', 'man', 'ran', 'away', '']\n", "\n", "Chosen action: ARC_L, Valid action: SHIFT\n", "\n", "Stack: ['', 'The']\n", "Input Buffer: ['man', 'ran', 'away', '']\n", "\n", "Chosen action: ARC_L, Valid action: ARC_L\n", "\n", "Stack: ['', 'The', 'man', 'ran']\n", "Input Buffer: ['away', '']\n", "\n", "Chosen action: SHIFT, Valid action: ARC_R\n", "\n" ] } ], "source": [ "print(parser_state)\n", "act_to_do = consts.Actions.ARC_L\n", "valid_action = parser_state._validate_action(act_to_do)\n", "print(\"Chosen action: %s, Valid action: %s\\n\" % (ix_to_action[act_to_do], ix_to_action[valid_action]))\n", "\n", "parser_state.shift()\n", "\n", "print(parser_state)\n", "act_to_do = consts.Actions.ARC_L\n", "valid_action = parser_state._validate_action(act_to_do)\n", "print(\"Chosen action: %s, Valid action: %s\\n\" % (ix_to_action[act_to_do], ix_to_action[valid_action]))\n", "\n", "parser_state.shift()\n", "parser_state.shift()\n", "\n", "print(parser_state)\n", "act_to_do = consts.Actions.SHIFT\n", "valid_action = parser_state._validate_action(act_to_do)\n", "print(\"Chosen action: %s, Valid action: %s\\n\" % (ix_to_action[act_to_do], ix_to_action[valid_action]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2. Neural Network for Action Decisions (2 points)\n", "In this part of the assignment, you will use PyTorch to create a neural network which examines the current state of the parse and makes the decision to either shift, arc left, or arc right." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 2.1: Word Embedding Lookup (0.5 points)\n", "Implement the class `VanillaWordEmbedding` in `neural_net.py`\n", "[Here are the docs for Pytorch embeddings](http://pytorch.org/docs/nn.html#embedding)\n", "\n", "Hint: You will have to turn the input, which is a list of strings (the words in the sentence), into a format that your embedding lookup table can take. \n", "\n", "**Test:** `test_parser.py:test_word_embed_lookup_d2_1`" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "3 \n", "\n", "Embedding for 'natural':\n", " Variable containing:\n", "-0.7945 1.7483 -0.6024 -0.8473\n", "[torch.FloatTensor of size 1x4]\n", "\n" ] } ], "source": [ "torch.manual_seed(765) # DO NOT CHANGE\n", "reload(neural_net)\n", "test_sentence = \"natural language processing\".split()\n", "test_word_to_ix = { \"natural\": 0, \"language\": 1, \"processing\": 2 }\n", "\n", "word_embedder = neural_net.VanillaWordEmbedding(test_word_to_ix, TEST_EMBEDDING_DIM)\n", "embeds = word_embedder(test_sentence)\n", "print(type(embeds))\n", "print(len(embeds), \"\\n\")\n", "print(\"Embedding for 'natural':\\n {}\".format(embeds[0]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 2.2: Feature Extraction (0.5 points)\n", "Fill in the `SimpleFeatureExtractor` class in `feat_extractors.py` to give the following 3 features as a list **in this order**:\n", "* The embedding of the top of the stack\n", "* The embedding of the first token in the input buffer\n", "* The embedding of the next token in the input buffer (one-token lookahead)\n", "\n", "If at this point you have not poked around `ParserState` to see how it stores the state, now would be a good time.\n", "\n", "**Test:** `test_parser.py:test_feature_extraction_d2_2`" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Embedding for 'The':\n", " Variable containing:\n", "-1.0276 -0.5631 -0.8923 -0.0583\n", "[torch.FloatTensor of size 1x4]\n", "\n", "Embedding for 'Sound':\n", " Variable containing:\n", " 0.6614 0.2669 0.0617 0.6213\n", "[torch.FloatTensor of size 1x4]\n", "\n", "Embedding for 'and' (from buffer lookahead):\n", " Variable containing:\n", "-0.4212 -0.5107 -1.5727 -0.1232\n", "[torch.FloatTensor of size 1x4]\n", "\n" ] } ], "source": [ "reload(feat_extractors)\n", "torch.manual_seed(1) # DO NOT CHANGE\n", "test_sentence = \"The Sound and the Fury\".split()\n", "test_word_to_ix = { word: i for i, word in enumerate(set(test_sentence)) }\n", "\n", "embedder = neural_net.VanillaWordEmbedding(test_word_to_ix, TEST_EMBEDDING_DIM)\n", "embeds = embedder(test_sentence)\n", "\n", "state = parsing.ParserState(test_sentence, embeds, utils.DummyCombiner())\n", "\n", "state.shift()\n", "feat_extractor = feat_extractors.SimpleFeatureExtractor()\n", "feats = feat_extractor.get_features(state)\n", "\n", "print(\"Embedding for 'The':\\n {}\".format(feats[0]))\n", "print(\"Embedding for 'Sound':\\n {}\".format(feats[1]))\n", "print(\"Embedding for 'and' (from buffer lookahead):\\n {}\".format(feats[2]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 2.3: Feedforward Network for Choosing Actions (0.5 points)\n", "Implement the class `neural_net.FFActionChooser` according to the specification.\n", "You will need to take the list of embeddings passed in (that come from your feature extractor) and concatenate them to one long row vector (size [1 x num actions])\n", "\n", "This network takes as input the features from your feature extractor, concatenates them, runs them through a feedforward network, and outputs log probabilities over actions.\n", "\n", "**Test:** `test_parser.py:test_action_chooser_d2_3`" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", "-1.2443 -0.8323 -1.2844\n", "[torch.FloatTensor of size 1x3]\n", "\n" ] } ], "source": [ "reload(neural_net)\n", "torch.manual_seed(1) # DO NOT CHANGE, you can compare my output below to yours\n", "act_chooser = neural_net.FFActionChooser(TEST_EMBEDDING_DIM * NUM_FEATURES)\n", "feats = [ ag.Variable(torch.randn(1, TEST_EMBEDDING_DIM)) for _ in range(NUM_FEATURES) ] # make some dummy feature embeddings\n", "log_probs = act_chooser(feats)\n", "print(log_probs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 2.4: Network for Combining Stack Items (0.5 points)\n", "Implement the class `neural_net.FFCombiner` according to the specification. \n", "Recall that what this component does is take two embeddings, the head and modifier, during an arc- operation and output a combined embedding (of size [1 x embedding_dim]), which is then pushed back onto the input buffer during parsing.\n", "\n", "**Test:** `test_parser.py:test_combiner_d2_4`" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", " 0.4285 -0.1363 0.4046 0.6006\n", "[torch.FloatTensor of size 1x4]\n", "\n" ] } ], "source": [ "reload(neural_net)\n", "torch.manual_seed(1) # DO NOT CHANGE\n", "combiner = neural_net.FFCombiner(TEST_EMBEDDING_DIM)\n", "\n", "# Again, make dummy inputs\n", "head_feat = ag.Variable(torch.randn(1, TEST_EMBEDDING_DIM))\n", "modifier_feat = ag.Variable(torch.randn(1, TEST_EMBEDDING_DIM))\n", "combined = combiner(head_feat, modifier_feat)\n", "print(combined)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Return of the Parser (2.5 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 3.1: Parser Training Code (2 points)\n", "We will now complete the parser and train it on our data. It is important to understand the difference between the following tasks:\n", "\n", "* Training: Training the model involves passing it sentences along with the correct sequence of actions, and updating weights.\n", "* Evaluation: We can evaluate the parser by passing it sentences along with the correct sequence of actions, and see how many actions it predicts correctly. This is identical to training, except the weights are not updated after making a prediction.\n", "* Prediction: After setting the weights, we give it a raw sentence (no gold-standard actions), and let it follow its own predicted actions to create a dependency graph, which we can compare to the ground truth.\n", "\n", "You will implement the `forward()` function in `gtnlplib.parsing.TransitionParser`.\n", "\n", "At this point, it is necessary to have all of the components from part 2 in place for constructing the parser.\n", "\n", "The parsing logic is roughly as follows:\n", "* Loop until parsing state is in its terminating state (deliverable 1.2)\n", "* Get the features from the parsing state (deliverable 2.2)\n", "* Send them through your action chooser network to get log probabilities over actions (deliverable 2.3)\n", "* If you have `gold_actions`, do them. Otherwise (when predicting), take the argmax of your log probabilities, validate the action (deliverable 1.3), and do that. An argmax function is provided for you in `utils.argmax`.\n", "\n", "Make sure to keep track of the things that the function wants to keep track of\n", "* Do all of your actions by calling the appropriate function on your `parser_state`\n", "* Append each output autograd.Variable from your action_chooser to the outputs list\n", "* Append each action you do to `actions_done`\n", "* Build the set of dependency edges as you go\n", "\n", "**Tests:**\n", "- `test_parser.py:test_parse_logic_d3_1`\n", "- `test_parser.py:test_predict_after_train_d3_1`" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "test_sentence = \"The man ran away\".split()\n", "test_word_to_ix = { word: i for i, word in enumerate(set(test_sentence)) }\n", "test_word_to_ix[consts.END_OF_INPUT_TOK] = len(test_word_to_ix)\n", "test_sentence_vocab = set(test_sentence)\n", "gold_actions = [\"SHIFT\", \"ARC_L\", \"SHIFT\", \"ARC_L\", \"SHIFT\", \"ARC_R\", \"ARC_R\", \"SHIFT\"]" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{DepGraphEdge(head=('ran', 2), modifier=('away', 3)), DepGraphEdge(head=('man', 1), modifier=('The', 0)), DepGraphEdge(head=('ran', 2), modifier=('man', 1)), DepGraphEdge(head=('', -1), modifier=('ran', 2))}\n", "[0, 1, 0, 1, 0, 2, 2, 0]\n" ] } ], "source": [ "reload(parsing)\n", "torch.manual_seed(1)\n", "feat_extractor = feat_extractors.SimpleFeatureExtractor()\n", "word_embedding_lookup = neural_net.VanillaWordEmbedding(test_word_to_ix, STACK_EMBEDDING_DIM)\n", "action_chooser = neural_net.FFActionChooser(STACK_EMBEDDING_DIM * NUM_FEATURES)\n", "combiner_network = neural_net.FFCombiner(STACK_EMBEDDING_DIM)\n", "parser = parsing.TransitionParser(feat_extractor, word_embedding_lookup,\n", " action_chooser, combiner_network)\n", "output, depgraph, actions_done = parser(test_sentence, gold_actions)\n", "print(depgraph)\n", "print(actions_done)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Now Train the Parser!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Training your parser may take some time. On the test below, I get about 5 seconds per loop on an i7 processor.\n", "\n", "There are 10,000 training sentences, so multiply this measurement by 100 to get your training time." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def train_parser(parser, optimizer, dataset, n_epochs=1, n_train_insts=1000):\n", " for epoch in range(n_epochs):\n", " print(\"Epoch {}\".format(epoch+1))\n", "\n", " parser.train() # turn on dropout layers if they are there\n", " parsing.train(dataset.training_data[:n_train_insts], parser, optimizer, verbose=True)\n", "\n", " print(\"Dev Evaluation\")\n", " parser.eval() # turn them off for evaluation\n", " parsing.evaluate(dataset.dev_data, parser, verbose=True)\n", " print(\"F-Score: {}\".format(evaluation.compute_metric(parser, dataset.dev_data, evaluation.fscore)))\n", " print(\"Attachment Score: {}\".format(evaluation.compute_attachment(parser, dataset.dev_data)))\n", " print(\"\\n\")" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(parsing)\n", "torch.manual_seed(1)\n", "feat_extractor = feat_extractors.SimpleFeatureExtractor()\n", "word_embedding_lookup = neural_net.VanillaWordEmbedding(word_to_ix_en, STACK_EMBEDDING_DIM)\n", "action_chooser = neural_net.FFActionChooser(STACK_EMBEDDING_DIM * NUM_FEATURES)\n", "combiner_network = neural_net.FFCombiner(STACK_EMBEDDING_DIM)\n", "parser = parsing.TransitionParser(feat_extractor, word_embedding_lookup,\n", " action_chooser, combiner_network)\n", "optimizer = optim.SGD(parser.parameters(), lr=ETA_0)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of instances: 100 Number of network actions: 4836\n", "Acc: 0.694995864350703 Loss: 33.874727449417115\n", "Number of instances: 100 Number of network actions: 4836\n", "Acc: 0.835814722911497 Loss: 18.836619388461113\n", "Number of instances: 100 Number of network actions: 4836\n", "Acc: 0.9083953680727874 Loss: 11.008044768869876\n", "Number of instances: 100 Number of network actions: 4836\n", "Acc: 0.9408602150537635 Loss: 7.201295542120934\n", "1 loop, best of 3: 3.22 s per loop\n" ] } ], "source": [ "%%timeit\n", "torch.manual_seed(1)\n", "parsing.train(en_dataset.training_data[:100], parser, optimizer, verbose=True)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1\n", "Number of instances: 1000 Number of network actions: 44560\n", "Acc: 0.822105026929982 Loss: 20.13014102517441\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 15846\n", "Acc: 0.8085321216710842 Loss: 16.59423623385629\n", "F-Score: 0.4491140169714535\n", "Attachment Score: 0.4321595355294712\n", "\n", "\n" ] } ], "source": [ "# train the parser for a while here.\n", "# Shouldn't take *too* long, even on a laptop\n", "torch.manual_seed(1)\n", "train_parser(parser, optimizer, en_dataset, n_train_insts=1000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 3.2: Test Data Predictions (0.25 points)\n", "Run the code below to output your predictions on the test data and dev data. You can run the dev test to verify you are correct up to this point. The test data evaluation is for us.\n", "\n", "**Test**: `test_parser.py:test_dev_d3_2_english`" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dev_sentences = [ sentence for sentence, _ in en_dataset.dev_data ]\n", "evaluation.output_preds(consts.EN_D3_2_DEV_FILENAME, parser, dev_sentences)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(consts.EN_D3_2_TEST_FILENAME, parser, en_dataset.test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 3.3: Dependency parsing in Norwegian (0.25 points)\n", "Run the code below to output your predictions on the **norwegian** test data and dev data. You can run the dev test to verify you are correct up to this point. The test data evaluation is for us.\n", "\n", "**Test**: `test_parser.py:test_dev_d3_3_norwegian`" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#first, make the parser\n", "reload(parsing)\n", "torch.manual_seed(1)\n", "feat_extractor_nr = feat_extractors.SimpleFeatureExtractor()\n", "word_embedding_lookup_nr = neural_net.VanillaWordEmbedding(word_to_ix_nr, STACK_EMBEDDING_DIM)\n", "action_chooser_nr = neural_net.FFActionChooser(STACK_EMBEDDING_DIM * NUM_FEATURES)\n", "combiner_network_nr = neural_net.FFCombiner(STACK_EMBEDDING_DIM)\n", "parser_nr = parsing.TransitionParser(feat_extractor_nr, word_embedding_lookup_nr,\n", " action_chooser_nr, combiner_network_nr)\n", "optimizer_nr = optim.SGD(parser_nr.parameters(), lr=ETA_0)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1\n", "Number of instances: 1000 Number of network actions: 30942\n", "Acc: 0.8121323767048025 Loss: 13.992266668349504\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 16028\n", "Acc: 0.8132642874968805 Loss: 14.277417845183802\n", "F-Score: 0.4524270779220721\n", "Attachment Score: 0.4336161716995258\n", "\n", "\n", "Epoch 2\n", "Number of instances: 1000 Number of network actions: 30942\n", "Acc: 0.8784500032318532 Loss: 9.374118270650971\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 16028\n", "Acc: 0.8251185425505365 Loss: 15.825147108596838\n", "F-Score: 0.4604127041161535\n", "Attachment Score: 0.4498377838782131\n", "\n", "\n" ] } ], "source": [ "torch.manual_seed(1)\n", "train_parser(parser_nr, optimizer_nr, nr_dataset, n_epochs=2, n_train_insts=1000)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true, "scrolled": true }, "outputs": [], "source": [ "reload(evaluation)\n", "dev_sentences_nr = [ sentence for sentence, _ in nr_dataset.dev_data ]\n", "evaluation.output_preds(consts.NR_D3_3_DEV_FILENAME, parser_nr, dev_sentences_nr)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(consts.NR_D3_3_TEST_FILENAME, parser_nr, nr_dataset.test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Evaluation and Training Improvements (3 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 4.1: BiLSTM Word Embeddings (0.5 points)\n", "Implement the class `BiLSTMWordEmbedding` in `neural_net.py`.\n", "This class can replace your `VanillaWordEmbedding`.\n", "This class implements a sequence model over the sentence, where the t'th word's embedding is the hidden state at timestep t.\n", "This means that, rather than have our embeddings on the stack only include the semantics of a single word, our embeddings will contain information from all parts of the sentence (the LSTM will, in principle, learn what information is relevant).\n", "\n", "**Test**: `tests/test_parser.py:test_bilstm_word_embeds_d4_1`" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "2 \n", "\n", "Embedding for Noam:\n", " Variable containing:\n", "\n", "Columns 0 to 9 \n", " 0.1056 0.0944 0.0412 0.3668 -0.0773 -0.0040 0.1634 -0.1230 -0.1293 0.1256\n", "\n", "Columns 10 to 19 \n", " 0.1295 0.0352 0.1901 0.0478 0.0433 0.0087 -0.0291 0.1329 0.3459 -0.1165\n", "\n", "Columns 20 to 29 \n", "-0.3131 -0.0233 -0.0261 0.1224 0.2036 -0.1736 -0.2412 0.0233 0.0510 -0.1779\n", "\n", "Columns 30 to 39 \n", "-0.0900 -0.0254 -0.1700 -0.0838 -0.0048 -0.0954 0.1243 -0.2872 -0.0251 0.0371\n", "\n", "Columns 40 to 49 \n", " 0.0771 -0.2720 0.1354 0.1249 0.0127 -0.2583 0.1458 -0.0781 -0.0400 -0.0394\n", "\n", "Columns 50 to 59 \n", "-0.1927 0.0358 0.0909 0.0357 0.1218 0.1649 -0.0762 0.1449 0.2308 -0.0049\n", "\n", "Columns 60 to 69 \n", "-0.2879 -0.0810 0.1228 0.1740 -0.2155 0.1019 0.3102 0.0146 0.3096 0.0209\n", "\n", "Columns 70 to 79 \n", " 0.0164 -0.1165 0.1106 0.2169 0.2170 -0.0204 -0.1604 -0.1576 -0.0712 -0.0304\n", "\n", "Columns 80 to 89 \n", "-0.1149 -0.0915 -0.1964 0.1715 0.0541 -0.1103 -0.3502 -0.1217 0.0280 -0.0770\n", "\n", "Columns 90 to 99 \n", " 0.0541 0.0820 0.1399 0.1348 0.0342 0.1564 -0.1653 -0.0415 0.2266 -0.1426\n", "[torch.FloatTensor of size 1x100]\n", "\n" ] } ], "source": [ "reload(neural_net)\n", "torch.manual_seed(1) # DO NOT CHANGE\n", "test_sentence = \"Noam Chomsky\".split()\n", "test_word_to_ix = { \"Noam\": 0, \"Chomsky\": 1 }\n", "\n", "lstm_word_embedder = neural_net.BiLSTMWordEmbedding(test_word_to_ix,\n", " WORD_EMBEDDING_DIM,\n", " STACK_EMBEDDING_DIM,\n", " num_layers=LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", " \n", "lstm_embeds = lstm_word_embedder(test_sentence)\n", "print(type(lstm_embeds))\n", "print(len(lstm_embeds), \"\\n\")\n", "print(\"Embedding for Noam:\\n {}\".format(lstm_embeds[0]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 4.2: Suffix Embeddings (0.5 points)\n", "We can also try to more explicitly include morphological information by embedding the suffix of a word in addition to the word itself. We approximate the \"suffix\" by just looking at the last two characters of a word.\n", "\n", "First, implement the function `build_suff_to_ix` in `utils.py`. It should take in a `word_to_ix` lookup and return a `suff_to_ix` lookup.\n", "\n", "Then, implement the class `SuffixAndWordEmbedding` in `neural_net.py`.\n", "This class embeds the words and suffixes in a sentence and then concatenates them to form one embedding. \n", "\n", "**Test**: `tests/test_parser.py:test_suff_word_embeds_d4_2`" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false }, "outputs": [], "source": [ "reload(utils)\n", "suff_to_ix_en = utils.build_suff_to_ix(word_to_ix_en)\n", "suff_to_ix_nr = utils.build_suff_to_ix(word_to_ix_nr)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "(1145, 849)" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(suff_to_ix_en), len(suff_to_ix_nr)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(neural_net)\n", "torch.manual_seed(1) # DO NOT CHANGE\n", "test_sentence = \"prefix fixsuf fixinfix\".split()\n", "test_word_to_ix = { \"prefix\": 0, \"fixsuf\": 1 , \"fixinfix\": 2}\n", "test_suff_to_ix = utils.build_suff_to_ix(test_word_to_ix)\n", "\n", "suff_word_embedder = neural_net.SuffixAndWordEmbedding(test_word_to_ix, test_suff_to_ix, TEST_EMBEDDING_DIM)\n", "test_embs = suff_word_embedder(test_sentence)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 0.6614 0.2669 -1.5228 0.3817\n", "[torch.FloatTensor of size 1x4]" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_embs[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 4.3: Pretrained Embeddings (0.5 points)\n", "\n", "Fill in the function `initialize_with_pretrained` in `utils.py`.\n", "\n", "It will take a word embedding lookup component and initialize its lookup table with pretrained embeddings, which are provided. Note that this is only applicable for the Vanilla, BiLSTM, and SuffixAndWord embedding components.\n", "\n", "**Test**: `tests/test_parser.py:test_pretrained_embeddings_d4_3`" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.12429751455783844, -0.11472601443529129, -0.5684014558792114, -0.396965891122818, 0.22938089072704315]\n" ] } ], "source": [ "import pickle\n", "pretrained_embeds = pickle.load(open(consts.PRETRAINED_EMBEDS_FILE, 'rb'))\n", "print(pretrained_embeds['four'][:5])" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true }, "outputs": [], "source": [ "torch.manual_seed(1)\n", "embedder = neural_net.VanillaWordEmbedding(word_to_ix_en,64)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", "-0.4954\n", "-1.1286\n", "-0.4468\n", "-0.1286\n", " 1.0928\n", "[torch.FloatTensor of size 5]" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "embedder.forward(['four'])[0][0,:5]" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", " 0.1243\n", "-0.1147\n", "-0.5684\n", "-0.3970\n", " 0.2294\n", "[torch.FloatTensor of size 5]\n", "\n" ] } ], "source": [ "reload(utils);\n", "utils.initialize_with_pretrained(pretrained_embeds,embedder)\n", "print(embedder.forward(['four'])[0][0,:5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 4.4: Better Arc Component Combination (0.5 points)\n", "Before, in order to combine two embeddings during an arc- operation, we just passed them through a feed-forward network and got a dense output. Now, we will instead use a sequence model of the stack. The combined embedding from an arc- operation is the next time step of an LSTM. Implement `neural_net.LSTMCombiner`.\n", "\n", "**Test**: `tests/test_parser.py:test_lstm_combiner_d4_4`" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false }, "outputs": [], "source": [ "reload(neural_net)\n", "torch.manual_seed(1)\n", "combiner = neural_net.LSTMCombiner(TEST_EMBEDDING_DIM,\n", " num_layers=LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", "head_feat = ag.Variable(torch.randn(1,TEST_EMBEDDING_DIM))\n", "mod_feat = ag.Variable(torch.randn(1,TEST_EMBEDDING_DIM))" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 0.0532 -0.1534 0.1484 -0.0595\n", "[torch.FloatTensor of size 1x4]" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "combined = combiner(head_feat, mod_feat)\n", "combined" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 4.5: Better action choosing (0.5 points)\n", "Instead of choosing the action from the combiner output independently at each time step, let's use an LSTM to predict the action. This way, past actions can influence the current decision directly. \n", "\n", "Implement `neural_net.LSTMActionChooser`. Use a linear layer to predict the action from the LSTM hidden state.\n", "\n", "**Test**: `tests/test_parser.py:test_lstm_action_chooser_d4_5`" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(neural_net)\n", "torch.manual_seed(1)\n", "action_chooser = neural_net.LSTMActionChooser(TEST_EMBEDDING_DIM * NUM_FEATURES,\n", " LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", "feats = [ag.Variable(torch.randn(1, TEST_EMBEDDING_DIM)) for _ in range(NUM_FEATURES)]" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", "-1.0328 -1.1798 -1.0887\n", "[torch.FloatTensor of size 1x3]" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "output = action_chooser(feats)\n", "output" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Retrain with the new components" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(neural_net)\n", "reload(parsing)\n", "torch.manual_seed(1)\n", "stack_dim = STACK_EMBEDDING_DIM\n", "feat_extractor = feat_extractors.SimpleFeatureExtractor()\n", "#BiLSTM word embeddings will probably work best, but feel free to experiment with the others you developed\n", "word_embedding_lookup = neural_net.BiLSTMWordEmbedding(word_to_ix_en,\n", " WORD_EMBEDDING_DIM,\n", " STACK_EMBEDDING_DIM,\n", " num_layers=LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", "utils.initialize_with_pretrained(pretrained_embeds, word_embedding_lookup)\n", "action_chooser = neural_net.LSTMActionChooser(STACK_EMBEDDING_DIM * NUM_FEATURES,\n", " LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", "combiner = neural_net.LSTMCombiner(STACK_EMBEDDING_DIM,\n", " num_layers=LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", "parser = parsing.TransitionParser(feat_extractor, word_embedding_lookup,\n", " action_chooser, combiner)\n", "optimizer = optim.SGD(parser.parameters(), lr=ETA_0)" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1\n", "Number of instances: 1000 Number of network actions: 44560\n", "Acc: 0.8002917414721723 Loss: 20.580642379820347\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 15846\n", "Acc: 0.8632462451091758 Loss: 10.602646742164792\n", "F-Score: 0.578173258242757\n", "Attachment Score: 0.5754133535277042\n", "\n", "\n", "Epoch 2\n", "Number of instances: 1000 Number of network actions: 44560\n", "Acc: 0.8842908438061041 Loss: 12.526566296416334\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 15846\n", "Acc: 0.8797803862173419 Loss: 9.533563165748845\n", "F-Score: 0.6145970679576657\n", "Attachment Score: 0.6142875173545375\n", "\n", "\n" ] } ], "source": [ "# The LSTMs will make this take longer, probably just a few minutes\n", "train_parser(parser, optimizer, en_dataset, n_epochs=2, n_train_insts=1000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 4.6: Test Predictions: English (0.25 points)\n", "\n", "**Test**: `tests/test_parser.py:test_dev_preds_d4_6_english`" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dev_sentences = [ sentence for sentence, _ in en_dataset.dev_data ]\n", "evaluation.output_preds(consts.EN_D4_6_DEV_FILENAME, parser, dev_sentences)" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(consts.EN_D4_6_TEST_FILENAME, parser, en_dataset.test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deliverable 4.7: Test Predictions: Norwegian (0.25 points)\n", "\n", "**Test**: `tests/test_parser.py:test_dev_preds_d4_7_norwegian`" ] }, { "cell_type": "code", "execution_count": 62, "metadata": { "collapsed": false }, "outputs": [], "source": [ "torch.manual_seed(1)\n", "feat_extractor_nr = feat_extractors.SimpleFeatureExtractor()\n", "#BiLSTM word embeddings will probably work best, but feel free to experiment with the others you developed\n", "word_embedding_lookup_nr = neural_net.BiLSTMWordEmbedding(word_to_ix_nr,\n", " WORD_EMBEDDING_DIM,\n", " STACK_EMBEDDING_DIM,\n", " num_layers=LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", "action_chooser_nr = neural_net.FFActionChooser(STACK_EMBEDDING_DIM * NUM_FEATURES)\n", "combiner_nr = neural_net.LSTMCombiner(STACK_EMBEDDING_DIM,\n", " num_layers=LSTM_NUM_LAYERS,\n", " dropout=DROPOUT)\n", "parser_nr = parsing.TransitionParser(feat_extractor_nr, word_embedding_lookup_nr,\n", " action_chooser_nr, combiner_nr)\n", "optimizer_nr = optim.SGD(parser_nr.parameters(), lr=ETA_0)" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1\n", "Number of instances: 1000 Number of network actions: 30942\n", "Acc: 0.8080279232111692 Loss: 13.581073675002903\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 16028\n", "Acc: 0.8359121537309708 Loss: 11.764973430713999\n", "F-Score: 0.5047176037981329\n", "Attachment Score: 0.4774145245819815\n", "\n", "\n", "Epoch 2\n", "Number of instances: 1000 Number of network actions: 30942\n", "Acc: 0.8795811518324608 Loss: 8.734120592931285\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 16028\n", "Acc: 0.8588719740454205 Loss: 10.630846565972249\n", "F-Score: 0.5493485293478391\n", "Attachment Score: 0.5243324182680309\n", "\n", "\n", "Epoch 3\n", "Number of instances: 1000 Number of network actions: 30942\n", "Acc: 0.9154224032059983 Loss: 6.463412301494274\n", "Dev Evaluation\n", "Number of instances: 501 Number of network actions: 16028\n", "Acc: 0.8622410781133018 Loss: 11.003425202541566\n", "F-Score: 0.5542113780643677\n", "Attachment Score: 0.5308210631395058\n", "\n", "\n" ] } ], "source": [ "train_parser(parser_nr, optimizer_nr, nr_dataset, n_epochs=3, n_train_insts=1000)" ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dev_sentences_nr = [ sentence for sentence, _ in nr_dataset.dev_data ]\n", "evaluation.output_preds(consts.NR_D4_7_DEV_FILENAME, parser_nr, dev_sentences_nr)" ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(consts.NR_D4_7_TEST_FILENAME, parser_nr, nr_dataset.test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5. Bakeoff! (2 points)\n", "\n", "\n", "We will have another Kaggle bakeoff for this problem set. The links will be put on canvas.\n", "\n", "Try to implement new features and tune your network's architecture and hyperparameters to get the best network.\n", "Section 3 of [this paper](https://pdfs.semanticscholar.org/55b8/1991fbb025038d98e8c71acf7dc2b78ee5e9.pdfhttps://pdfs.semanticscholar.org/55b8/1991fbb025038d98e8c71acf7dc2b78ee5e9.pdf) may help out with hyper parameter tuning if you are new to neural networks.\n", "To get very competitive, it may be necessary to train for a large amount of time (leaving it running overnight should be fine). Here are some suggestions.\n", "* Tune your learning rate.\n", "* Tune your other hyperparameters.\n", "* Try different optimizers. torch.optim has a ton of different training algorithms. SGD was used in this pset because it is fast, but it is the most vanilla of them. Trying new ones, like Adam, will almost certainly boost performance\n", "* Try adding regularization to your network if you see evidence that it is overfitting. This can be done with:\n", " * L2 regularization using the [weight decay argument](http://pytorch.org/docs/master/optim.html#torch.optim.SGD)\n", " * adding dropout (already an input argument to some of the neural net components)\n", " * implement early stopping (stop training if dev performance on some metric doesn't improve for k epochs)\n", "* Try customizing any of the 3 components (word embeddings, action choosing, combining) in clever ways. You can create new classes that expose the same public interface and use them here (just leave your required ones untouched). Building word embeddings from characters using an RNN or convolutional layer may help.\n", "* Try new features. Write new classes that expose the same public interface as SimpleFeatureExtractor. Try looking further into stack history, or more input buffer lookahead, or features based on the action sequence. The possibilities are endless.\n", "* Check out [this book](http://www.deeplearningbook.org/), which is undoubtedly the best deep learning book (and it is free online!) which has great information on regularization, optimization, and different network architectures.\n", "\n", "**Tests**: \n", "- `tests/test_parser.py:test_dev_preds_bakeoff_d5_1_english`\n", "- `tests/test_parser.py:test_dev_preds_bakeoff_d5_2_norwegian`\n", "\n", "**Rubric**:\n", "English dev:\n", "- $\\geq$ 0.74: 0.15 points\n", "- $\\geq$ 0.75: 0.25 points\n", "- $\\geq$ 0.76: 0.5 points\n", "\n", "English test:\n", "- $\\geq$ 0.70: 0.15 points\n", "- $\\geq$ 0.71: 0.25 points\n", "- $\\geq$ 0.73: 0.5 points\n", "\n", "Norwegian dev:\n", "- $\\geq$ 0.66: 0.15 points\n", "- $\\geq$ 0.70: 0.25 points\n", "- $\\geq$ 0.71: 0.5 points\n", "\n", "Norwegian test:\n", "- $\\geq$ 0.66: 0.15 points\n", "- $\\geq$ 0.69: 0.25 points\n", "- $\\geq$ 0.70: 0.5 points\n", "\n", "**Extra credit**: \n", "- +0.25 if you beat the best TA/prof system in English or Norwegian\n", "- +0.25 if you are #1 in CS4650 in English or Norwegian\n", "- +0.25 if you are #1 in CS7650 in English or Norwegian\n", "\n", "Current staff best attachment scores are: \n", "- English dev: 0.786\n", "- English test: 0.753\n", "- Norwegian dev: 0.736\n", "- Norwegian test: 0.736" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using Cuda\n", "You can use CUDA to train your network, and you should expect decent speedup if you have a GPU and the CUDA toolkit installed.\n", "If you want to use CUDA in this assignment, change the HAVE_CUDA variable to True in constants.py, and call `.to_cuda()` on your parser. You may also need to reconfigure your Embedding layers if you didn't consider cuda before.\n", "\n", "We are not officially supporting CUDA though. If you have problems installing or running CUDA, please just use the CPU, we cannot help you debug it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Set your hyperparameters here\n", "# e.g learning rate, regularization, lr annealing, dimensionality of embeddings, number of epochs, early stopping etc." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Make your parser here\n", "# name your TransitionParser bakeoff_parser to output your predictions below\n", "# bakeoff_parser_en = TransitionParser(...)\n", "\n", "# Also, choose an optimizer.\n", "# bakeoff_optimizer_en = optim...." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# train for bakeoff\n", "train_parser(bakeoff_parser_en, bakeoff_optimizer_en, en_dataset, n_epochs=5, n_train_insts=10000)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(\"bakeoff-dev-en.preds\", bakeoff_parser_en, en_dataset.dev_data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(\"bakeoff-test-en.preds\", bakeoff_parser_en, en_dataset.test_data)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#use this to output predictions in kaggle-ready format\n", "evaluation.kaggle_output(\"KAGGLE-bakeoff-preds-en.csv\", bakeoff_parser_en, en_dataset.test_data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Now make your norwegian parser if necessary\n", "# name your TransitionParser bakeoff_parser to output your predictions below\n", "# bakeoff_parser_nr = TransitionParser(...)\n", "\n", "# Also, choose an optimizer.\n", "# bakeoff_optimizer_nr = optim...." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# train for bakeoff\n", "train_parser(bakeoff_parser_nr, bakeoff_optimizer_nr, nr_dataset, n_epochs=5, n_train_insts=10000)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(\"bakeoff-dev-nr.preds\", bakeoff_parser_nr, nr_dataset.dev_data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.output_preds(\"bakeoff-test-nr.preds\", bakeoff_parser_nr, nr_dataset.test_data)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": true }, "outputs": [], "source": [ "evaluation.kaggle_output(\"KAGGLE-bakeoff-preds-nr.csv\", bakeoff_parser_nr, nr_dataset.test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6. 7650 only: Research question (1 point)\n", "\n", "Describe a paper that uses dependency trees for some downstream task. Your response should answer the following questions, with 1-2 sentences each.\n", "\n", "1. What is the task that is being solved?\n", "2. Briefly (one sentence) explain the metric for success on this task.\n", "3. Why are dependency features expected to help with this task?\n", "4. How are dependency features incorporated into the solution?\n", "5. Does the paper evaluate whether dependency features improve performance on the downstream task? If so, what is their impact? If not, why not?\n", "\n", "You should select a paper from 2008-2018 at one of the following venues: ACL, NAACL, EACL, EMNLP, TACL. Exceptions from this list will be considered on a case-by-case basis, please post a private question to Piazza.\n", "\n", "Here are some suggested papers. Free downloads of all these papers can be found by searching online.\n", "\n", "- Rajpurkar, Pranav, Jian Zhang, Konstantin Lopyrev, and Percy Liang. \"SQuAD: 100,000+ Questions for Machine Comprehension of Text.\"\n", "- Stanovsky, Gabriel, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. \"Integrating Deep Linguistic Features in Factuality Prediction over Unified Datasets.\" In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), vol. 2, pp. 352-357. 2017.\n", "- Xu, Yan, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. \"Classifying relations via long short term memory networks along shortest dependency paths.\" In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1785-1794. 2015.\n", "- Madaan, Aman, Ashish Mittal, G. Ramakrishnan Mausam, Ganesh Ramakrishnan, and Sunita Sarawagi. \"Numerical Relation Extraction with Minimal Supervision.\" In AAAI, pp. 2764-2771. 2016.\n", "- Lee, Kenton, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. \"Event detection and factuality assessment with non-expert supervision.\" In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1643-1648. 2015.\n", "- Kim, Sun, Haibin Liu, Lana Yeganova, and W. John Wilbur. \"Extracting drug–drug interactions from literature using a rich feature-based linear kernel approach.\" Journal of biomedical informatics 55 (2015): 23-30.\n", "- Shwartz, Vered, Yoav Goldberg, and Ido Dagan. \"Improving Hypernymy Detection with an Integrated Path-based and Distributional Method.\" In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 2389-2398. 2016.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 2 }