{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "ZX6M7xdN4B0e" }, "source": [ "# CS 124 Tutorial: Regular Expressions\n", "\n", "Based on the `CS 124: Jupyter and Python Tutorial` created by \n", "`Krishna Patel (Winter 2020)`, and updated by `Bryan Kim (Winter 2021)` and \n", "`Dilara Soylu (Winter 2022)`." ] }, { "cell_type": "markdown", "metadata": { "id": "_mygv6OHty4i" }, "source": [ "\n", "## Overview\n", "\n", "In this tutorial, we will walk you through some `Regular Expression` examples as\n", "a preparation for our first assignments, `PA 1`. \n", "This tutorial assumes that you have completed the following from the \n", "[PA 0 repository](https://github.com/cs124/pa0-python-jupyter-tutorial) and you \n", "are familiar with `Python`:\n", "* Setup instructions for your machine\n", "* [Jupyter Notebook Tutorial](https://github.com/cs124/pa0-python-jupyter-tutorial/blob/main/jupyter_tutorial.ipynb)" ] }, { "cell_type": "markdown", "metadata": { "id": "ZxV1KeAABK7x" }, "source": [ "\n", "## Contents\n", "\n", "1. [Environment Check](#environment_check)\n", "2. [`Regular Expressions` Exercises](#regular_expressions_exercises)\n", "3. [Answers](#answers)\n", "4. [Next Steps](#next_steps)" ] }, { "cell_type": "markdown", "metadata": { "id": "vH9-1RhhBgvp" }, "source": [ "\n", "### Environment Check\n", "\n", "Let's ensure that we are running our notebook in the correct environment." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "jw6sS9SHtsM1" }, "outputs": [], "source": [ "# Check the name of the conda environment\n", "import os\n", "assert os.environ['CONDA_DEFAULT_ENV'] == \"cs124\"\n", "\n", "# Check that the Python version is 3.8\n", "import sys\n", "assert sys.version_info.major == 3 and sys.version_info.minor == 10" ] }, { "cell_type": "markdown", "metadata": { "id": "A3c-AC3gtsM1", "pycharm": { "name": "#%% md\n" } }, "source": [ "If the above cell causes an error, it means that you are using the wrong \n", "environment or `Python` version!\n", "If this is the case, please follow the troubleshotting steps shared in the \n", "[Jupyter Notebook Tutorial](https://github.com/cs124/pa0-python-jupyter-tutorial/blob/main/jupyter_tutorial.ipynb)." ] }, { "cell_type": "markdown", "metadata": { "id": "30GSRCf0CnIB" }, "source": [ "\n", "## `Regular Expressions` Exercises\n", "\n", "`Regular Expressions` (`RegEx`) are usually used to search for patterns in \n", "strings, or check if a string matches a pattern. \n", "`Python` has a regular expression module that helps us execute regular \n", "expressions on bodies of text. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "b_nGqyQjC_dH" }, "outputs": [], "source": [ "# Import the standard Python standard RegEx library\n", "import re" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4jzHvdlztsM4", "outputId": "6f2f504b-4006-4675-b91b-b75b372e9cc5", "pycharm": { "name": "#%%\n" }, "scrolled": true }, "outputs": [], "source": [ "# As an example, we can use a comma as our RegEx pattern and use this pattern\n", "# to split a string\n", "\n", "input_str = \"a::b,c.d,e;:f,g\"\n", "\n", "# It is a good habit to mark RegEx patterns with the \"r\" prefix. In this\n", "# case it doesn't matter, but \"r\" is needed for the RegEx to be read\n", "# correctly when using special RegEx characters like \\b, \\w, etc. \n", "\n", "# This pattern matches a single comma\n", "pattern = r\",\"\n", "\n", "# re.split splits the input string at the matching patterns\n", "tokens = re.split(pattern, input_str)\n", "\n", "tokens" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "NnbEgBtotsM4", "outputId": "5c4aad9a-532c-4bac-bb71-34fb3dbfe215", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# We could be a bit fancier, and allow our pattern to be any character in\n", "# a set. Bracket notation [] indicates that we can match any of the characters\n", "# in the brackets.\n", "\n", "# This matches any ONE character in the set(a period, comma, semicolon, or\n", "# colon). Note that although the period (\".\") is a special character in RegExes, we\n", "# do not need to escpape using a backslash in this case because it is in a character class\n", "# which is denoted using the [ ]. More generally, any character other than ^, -, \\, or ] \n", "# is interpreted as a literal in a character class and does not need to be escaped.\n", "pattern = r\"[.,;:]\"\n", "\n", "tokens = re.split(pattern, input_str)\n", "\n", "tokens" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "EGkW2qmJtsM4", "outputId": "ee297623-ebd9-4b4e-eb7a-4d24cdb3dcdb", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# We could even use special operators to describe more specific patterns.\n", "# For example, the \"+\" operator means that it will match the object to its left\n", "# at least once, but possibly multiple times. Note that if the object to the left\n", "# is a set, it could be a different character from that set each time.\n", "\n", "# This matches any sequence of one or more characters from the set\n", "# [.,;:].\n", "pattern = r\"[.,;:]+\"\n", "\n", "tokens = re.split(pattern, input_str)\n", "\n", "tokens" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "gUzwPvFrtsM4", "outputId": "d27da824-926f-4fb0-d341-33dcc991b7c5", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# We can also use RegExes to find all the times a specific pattern appears\n", "# in a string. For example, what if we wanted to find all the instances of\n", "# the word \"dog\" in a text.\n", "\n", "text = \"\"\"F\n", "I love my dog Spot! Spot is the best dog in the world. He likes playing\n", "with other dogs at the park. But he doesn't like cats, he is scared of them.\n", "Today I will take him to the dog park. One time he saw a cat there and got so\n", "scared he wanted to go home.\n", "\"\"\"\n", "\n", "# This matches just the 3-character sequence \"dog\"\n", "pattern = r\"dog\"\n", "\n", "tokens = re.findall(pattern, text)\n", "\n", "tokens" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "bw1_bRYPtsM6", "outputId": "6fc5e29e-7e95-46bb-f1db-1808fcc51785", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# There are also other operators we can use, like ? to match the object to its\n", "# left 0 or 1 times (in other words, it is optional).\n", "\n", "# This matches \"dog\" followed by an optional s.\n", "pattern = r\"dog[s]?\"\n", "\n", "tokens = re.findall(pattern, text)\n", "\n", "tokens" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "hclcyyyvtsM6", "outputId": "f4fa19d1-e6b5-4e42-fbe1-79c423b5bee3", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Or we can match multiple possibilities (i.e. A or B)\n", "\n", "# This matches \"dog\" or \"cat\".\n", "pattern = r\"(dog|cat)\"\n", "\n", "tokens = re.findall(pattern, text)\n", "\n", "tokens" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "aoagV8iUtsM7", "outputId": "766c6977-04c4-4509-ea08-4a60bc5b99da", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# A particularly common operator is the star \"*\" operator. It matches 0 or more\n", "# of the object to its left. For example, we can use it to match any word that\n", "# starts with an a (an a or A at the start of a word, followed by 0 or more of\n", "# any letter).\n", "\n", "# We can use the \\b symbol to match the start of a word.\n", "# and the \\w symbol to match a letter.\n", "# See https://docs.python.org/3/library/re.html for details and other\n", "# special symbols.\n", "\n", "# This matches any word starting with an a or A\n", "pattern = r\"\\b[aA]\\w*\"\n", "\n", "tokens = re.findall(pattern, text)\n", "\n", "tokens" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fUjm3sZECarI", "outputId": "3a3f07ba-16c3-4d85-fae6-3e8d429e2a84" }, "outputs": [], "source": [ "# Beyond the re.findall() function, we also often use re.search() if we want\n", "# more flexible control over how we match/search\n", "# (example from https://www.w3schools.com/python/python_regex.asp)\n", "\n", "# re.search() is slightly more complicated than re.findall(), because it\n", "# doesn't just return a list of matches as strings. It instead returns a Python\n", "# match object that contains more detailed information about the match.\n", "\n", "# NOTE: period (\".\") is a special character indicating any character (except\n", "# a newline). \"^\" matches the start of a string and \"$\" matches the end of a\n", "# string.\n", "\n", "text = r\"The rain in Spain\"\n", "match = re.search(r\"^The.*Spain$\", text)\n", "\n", "# Print the match object\n", "print(match)\n", "\n", "# Find the first (and only) match\n", "print(match[0])\n", "\n", "# Get the original string back\n", "print(match.string)\n", "\n", "# Get the group (the entire part of the string where the match happened)\n", "print(match.group())\n", "\n", "# Find the span, the tuple (start_index, end_index).\n", "# In other words the positions (from the start of the string, counting from\n", "# 0), where the match starts and ends.\n", "print(\"span: \" + str(match.span()))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "-Jjf6PbpC9lA" }, "outputs": [], "source": [ "# We can also use capture groups (in parentheses) to indicate\n", "# parts of the match that we want to save so that we can use them separately\n", "\n", "# Using your knowledge of RegExes from the previous examples, what does this\n", "# pattern mean? What do you expect it to do/match from text? What parts of text\n", "# will fall in each of the 3 capture groups?\n", "ai_match = re.search(r\"(.*?)ai(.*?)ai(.*)\", text)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "fXvBtEdVDhf1", "outputId": "eeeb4644-b25d-4dd3-c570-348925aa2feb" }, "outputs": [], "source": [ "# Let's check if your guesses were correct:\n", "\n", "# Entire match\n", "ai_match.group(0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "B65YzEsFEVTP", "outputId": "bc1c4bb6-78d7-4ba9-dc79-64fc58d956a0" }, "outputs": [], "source": [ "# First capturing group \n", "ai_match.group(1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "B7dRS1RSEurW", "outputId": "88f279d6-b939-44d5-ef8a-8bbb86de10bb" }, "outputs": [], "source": [ "# Second capturing group\n", "ai_match.group(2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "rDqEUeNhE2sT", "outputId": "1d8240e2-63c8-467b-af41-7b14de950d18" }, "outputs": [], "source": [ "# Third capturing group\n", "ai_match.group(3)" ] }, { "cell_type": "markdown", "metadata": { "id": "DC7jMTXdJwuQ" }, "source": [ "Let's try a more difficult example: \n", "\n", "__HTML Content Extraction__ Try to extract the inside of an HTML Tag that \n", "follows the following rules:\n", "\n", "* Start tags start with \"<,\" have at least 1 alpha numeric character\n", "and then end with \">\"\n", "* End tags start with \"\"\n", "* There can be any character between the two tags\n", "\n", "Here are some examples example of the text:\n", "\n", "``this is what we want to extract``\n", "\n", "don't want`

`what we want `

` don't want\n", "\n", "---\n", "\n", "If you want to make it harder make sure you pass this test case, since\n", "technically HTML tags must match (the contents of the start tag must match\n", "the contents of the end tag):\n", "\n", "``this is what we want to extract `` `` \n", "\n", "You should extract: this is what we want to extract ``\n", "\n", "This problem was inspired by the `Regex Examples` in this \n", "[`link`](https://www.sitepoint.com/demystifying-regex-with-practical-examples/)." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "tejzB5MRLNDS" }, "outputs": [], "source": [ "# Try to solve this here\n", "\n", "test_str1 = \"this is what we want to extract\"\n", "test_str2 = \"don't want

what we want

don't want\"\n", "\n", "hard_case = \"this is what we want to extract \"" ] }, { "cell_type": "markdown", "metadata": { "id": "sNynBmNuLaP6" }, "source": [ "Solved it? You can find the answer in the `Solutions` section." ] }, { "cell_type": "markdown", "metadata": { "id": "UydjsI7wIHRG" }, "source": [ "Let's try a longer example: \n", "\n", "__URL Matching__ Try to match a URL that follows the following rules:\n", "\n", "* the URL must start with http or https followed by ://\n", "* the domain name can only be alphanumeric or contain \".\" or \"-\"\n", "* can contain a port specification (http://abc.com:80) (you can assume ports\n", "go from 0 to 99)\n", "* after the port, the URL can contain any number of alphanumeric digits,\n", "dots and hyphens\n", "\n", "This problem was inspired by the `Regex Examples` in this \n", "[`link`](https://www.sitepoint.com/demystifying-regex-with-practical-examples/)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8m69MJFNeF57" }, "outputs": [], "source": [ "# Try to solve this here\n", "\n", "test_str1 = \"http://www.google.com\"\n", "test_str2 = \"https://www.gmail.com:88/hello-hi\"\n", "test_str3 = \"http://abd-fh.8rhgyt.org:90/h-\"\n" ] }, { "cell_type": "markdown", "metadata": { "id": "pysyOzmBC3w7" }, "source": [ "\n", "## Answers\n", "\n", "You can find the answers to the last two exercises below." ] }, { "cell_type": "markdown", "metadata": { "id": "gOiLW0SIDO3Y" }, "source": [ "__HTML Content Extraction__" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "c7IhYOBvDnvi", "outputId": "7aed45b6-374a-45a2-8192-706f13c1b142" }, "outputs": [], "source": [ "import re\n", "test_str1 = \"this is what we want to extract\"\n", "test_str2 = \"don't want

what we want

don't want\"\n", "\n", "# Answer 1 -- works on first 2 test strings\n", "match = re.search(r\"<[\\w]+>(.*?)\", test_str1)\n", "\n", "match.group(1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "E0C8UfzqDo8W", "outputId": "5e54bb7c-9b61-4c8f-f21c-b7d742171295" }, "outputs": [], "source": [ "hard_test = \"this is what we want to extract \"\n", "\n", "# Answer 2 -- works on hard case as well\n", "match = re.search(r\"<([\\w]+)>(.*?)<\\/\\1>\", hard_test)\n", "\n", "match.group(2)\n", "\n", "# NB: \\1 will match the same text matched by the first capturing group" ] }, { "cell_type": "markdown", "metadata": { "id": "JjWRvntcEGno" }, "source": [ "__URL Matching__" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "xob4vGiqEIEw", "outputId": "cdf59922-560b-4b8c-883a-400bf2589509" }, "outputs": [], "source": [ "import re\n", "test_str1 = \"http://www.google.com\"\n", "test_str2 = \"https://www.gmail.com:88/hello-hi\"\n", "test_str3 = \"http://abd-fh.8rhgyt.org:90/h-\"\n", "\n", "match = re.search(r\"https?://([a-zA-Z0-9-\\.]+)(:[0-9][0-9]?)?/?([a-zA-Z0-9-\\.]*)\", test_str3)\n", "\n", "match.group(0)" ] }, { "cell_type": "markdown", "metadata": { "id": "rtq-S9arHkLQ" }, "source": [ "\n", "## Next Steps" ] }, { "cell_type": "markdown", "metadata": { "id": "m_UznkphtsNB", "pycharm": { "name": "#%% md\n" } }, "source": [ "You can now head start the first assignment!\n", "If you want more practice with regular expressions, you can try the following:\n", "* [learnpython problem](https://www.learnpython.org/en/Regular_Expressions)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "regular_expressions_tutorial.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.19" } }, "nbformat": 4, "nbformat_minor": 4 }