{
 "cells": [
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "#### Import python files from another directory\n",
    "In order to reuse code written in .py files you need to import them.\n",
    "If they are from a distant folder, rather than copy-paste it into the current folder, you can write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.append(\n",
    "    \"../../ex02/template\"\n",
    ")  # This goes up twice in the directories tree (hence in labs)\n",
    "# then goes down to ex02/template where your files from lab 2 are."
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "you can now import your desired files, for example, we can import grid_search.py with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import grid_search  # You then need to call your functions using grid_search.function_name()\n",
    "import grid_search as gs  # You then need to call your functions using gs.function_name()\n",
    "from grid_search import *  # You can call any functions of the file with function_name()\n",
    "\n",
    "# Let's try to call generate_w from grid_search.py:\n",
    "w0, w1 = generate_w(5)\n",
    "print(w0, w1)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "As you can see we are now able to call functions from the grid_search.py file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Useful starting lines\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from helpers import *\n",
    "\n",
    "from test_utils import test\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# 1 Least squares and linear basis functions models\n",
    "## 1.1 Least squares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def least_squares(y, tx):\n",
    "    \"\"\"Calculate the least squares solution.\n",
    "       returns mse, and optimal weights.\n",
    "\n",
    "    Args:\n",
    "        y: numpy array of shape (N,), N is the number of samples.\n",
    "        tx: numpy array of shape (N,D), D is the number of features.\n",
    "\n",
    "    Returns:\n",
    "        w: optimal weights, numpy array of shape(D,), D is the number of features.\n",
    "        mse: scalar.\n",
    "\n",
    "    >>> least_squares(np.array([0.1,0.2]), np.array([[2.3, 3.2], [1., 0.1]]))\n",
    "    (array([ 0.21212121, -0.12121212]), 8.666684749742561e-33)\n",
    "    \"\"\"\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # least squares: TODO\n",
    "    # returns mse, and optimal weights\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "### You can test your implementation here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test(least_squares)\n",
    "# NB:\n",
    "#\n",
    "# Due to precision issues,\n",
    "# the output might not be exactly the same\n",
    "# even if you have implemented the right code.\n",
    "#\n",
    "# For example, the mse output expected to be\n",
    "# 8.666684749742561e-33,\n",
    "# but you might get some other small number\n",
    "# close to zero.\n",
    "#\n",
    "# In this case,\n",
    "# Failing the test doesn't necessarily mean\n",
    "# your implementation is wrong.:)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "### Load the data\n",
    "Here we will reuse the dataset `height_weight_genders.csv` from previous exercise section to check the correctness of your implementation. Please compare it with your previous result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_your_least_squares():\n",
    "    \"\"\"\n",
    "    # this code should compare the optimal weights obtained\n",
    "    # by least squares vs. grid search\n",
    "    \"\"\"\n",
    "    height, weight, gender = load_data_from_ex02(sub_sample=False, add_outlier=False)\n",
    "    x, mean_x, std_x = standardize(height)\n",
    "    y, tx = build_model_data(x, weight)\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # least square or grid search: TODO\n",
    "    # this code should compare the optimal weights obtained\n",
    "    # by least squares vs. grid search\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Test it here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_your_least_squares()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## 1.2 Least squares with a linear basis function model\n",
    "Start from this section, we will use the dataset `dataEx3.csv`.\n",
    "\n",
    "### Implement polynomial basis functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load dataset\n",
    "x, y = load_data()\n",
    "print(\"shape of x {}\".format(x.shape))\n",
    "print(\"shape of y {}\".format(y.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_poly(x, degree):\n",
    "    \"\"\"polynomial basis functions for input data x, for j=0 up to j=degree.\n",
    "\n",
    "    Args:\n",
    "        x: numpy array of shape (N,), N is the number of samples.\n",
    "        degree: integer.\n",
    "\n",
    "    Returns:\n",
    "        poly: numpy array of shape (N,d+1)\n",
    "\n",
    "    >>> build_poly(np.array([0.0, 1.5]), 2)\n",
    "    array([[1.  , 0.  , 0.  ],\n",
    "           [1.  , 1.5 , 2.25]])\n",
    "    \"\"\"\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # polynomial basis function: TODO\n",
    "    # this function should return the matrix formed\n",
    "    # by applying the polynomial basis to the input data\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test(build_poly)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Once your implementation of `build_poly` passes the test, copy it to `build_polynomial.py`\n",
    "Let us play with polynomial regression. Note that we will use your implemented function `compute_mse`. Please copy and paste your implementation from exercise02."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from plots import *\n",
    "\n",
    "\n",
    "def polynomial_regression():\n",
    "    \"\"\"Constructing the polynomial basis function expansion of the data,\n",
    "    and then running least squares regression.\"\"\"\n",
    "    # define parameters\n",
    "    degrees = [1, 3, 7, 12]\n",
    "\n",
    "    # define the structure of the figure\n",
    "    num_row = 2\n",
    "    num_col = 2\n",
    "    f, axs = plt.subplots(num_row, num_col)\n",
    "\n",
    "    for ind, degree in enumerate(degrees):\n",
    "        # ***************************************************\n",
    "        # INSERT YOUR CODE HERE\n",
    "        # form the data to do polynomial regression.: TODO\n",
    "        # ***************************************************\n",
    "        raise NotImplementedError\n",
    "        # ***************************************************\n",
    "        # INSERT YOUR CODE HERE\n",
    "        # least square and calculate RMSE: TODO\n",
    "        # ***************************************************\n",
    "        raise NotImplementedError\n",
    "\n",
    "        print(\n",
    "            \"Processing {i}th experiment, degree={d}, rmse={loss}\".format(\n",
    "                i=ind + 1, d=degree, loss=rmse\n",
    "            )\n",
    "        )\n",
    "        # plot fit\n",
    "        plot_fitted_curve(y, x, weights, degree, axs[ind // num_col][ind % num_col])\n",
    "    plt.tight_layout()\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Run polynomial regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "polynomial_regression()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Your results should look like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "![alt text](visualize_polynomial_regression.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# 2 Evaluating model predication performance\n",
    "\n",
    "Let us show the train and test splits for various polynomial degrees. First of all, please fill in the function `split_data()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_data(x, y, ratio, seed=1):\n",
    "    \"\"\"\n",
    "    split the dataset based on the split ratio. If ratio is 0.8\n",
    "    you will have 80% of your data set dedicated to training\n",
    "    and the rest dedicated to testing. If ratio times the number of samples is not round\n",
    "    you can use np.floor. Also check the documentation for np.random.permutation,\n",
    "    it could be useful.\n",
    "\n",
    "    Args:\n",
    "        x: numpy array of shape (N,), N is the number of samples.\n",
    "        y: numpy array of shape (N,).\n",
    "        ratio: scalar in [0,1]\n",
    "        seed: integer.\n",
    "\n",
    "    Returns:\n",
    "        x_tr: numpy array containing the train data.\n",
    "        x_te: numpy array containing the test data.\n",
    "        y_tr: numpy array containing the train labels.\n",
    "        y_te: numpy array containing the test labels.\n",
    "\n",
    "    >>> split_data(np.arange(13), np.arange(13), 0.8, 1)\n",
    "    (array([ 2,  3,  4, 10,  1,  6,  0,  7, 12,  9]), array([ 8, 11,  5]), array([ 2,  3,  4, 10,  1,  6,  0,  7, 12,  9]), array([ 8, 11,  5]))\n",
    "    \"\"\"\n",
    "    # set seed\n",
    "    np.random.seed(seed)\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # split the data based on the given ratio: TODO\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test(split_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Then, test your `split_data` function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_test_split_demo(x, y, degree, ratio, seed):\n",
    "    \"\"\"polynomial regression with different split ratios and different degrees.\n",
    "\n",
    "    Returns:\n",
    "      x_tr: numpy array\n",
    "      x_te: numpy array\n",
    "      y_tr: numpy array\n",
    "      y_te: numpy array\n",
    "      weights: weights from the least squares optimization\"\"\"\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # split the data, and return train and test data: TODO\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # form train and test data with polynomial basis function: TODO\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # calculate weight through least square: TODO\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # calculate RMSE for train and test data,\n",
    "    # and store them in rmse_tr and rmse_te respectively: TODO\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError\n",
    "    print(\n",
    "        \"proportion={p}, degree={d}, Training RMSE={tr:.3f}, Testing RMSE={te:.3f}\".format(\n",
    "            p=ratio, d=degree, tr=rmse_tr, te=rmse_te\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Demo time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 6\n",
    "degrees = [1, 3, 7, 12]\n",
    "split_ratios = [0.9, 0.7, 0.5, 0.1]\n",
    "\n",
    "# define the structure of the figure\n",
    "num_row = 4\n",
    "num_col = 4\n",
    "axs = plt.subplots(num_row, num_col, figsize=(20, 8))[1]\n",
    "\n",
    "for ind, split_ratio in enumerate(split_ratios):\n",
    "    for ind_d, degree in enumerate(degrees):\n",
    "        x_tr, x_te, y_tr, y_te, w = train_test_split_demo(\n",
    "            x, y, degree, split_ratio, seed\n",
    "        )\n",
    "        plot_fitted_curve(y_tr, x_tr, w, degree, axs[ind_d][ind % num_col])\n",
    "        axs[ind_d][ind].set_title(f\"Degree: {degree}, Split {split_ratio}\")\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Your graph should look like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "![alt text](split_demo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Ridge Regression\n",
    "Please fill in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ridge_regression(y, tx, lambda_):\n",
    "    \"\"\"implement ridge regression.\n",
    "\n",
    "    Args:\n",
    "        y: numpy array of shape (N,), N is the number of samples.\n",
    "        tx: numpy array of shape (N,D), D is the number of features.\n",
    "        lambda_: scalar.\n",
    "\n",
    "    Returns:\n",
    "        w: optimal weights, numpy array of shape(D,), D is the number of features.\n",
    "\n",
    "    >>> ridge_regression(np.array([0.1,0.2]), np.array([[2.3, 3.2], [1., 0.1]]), 0)\n",
    "    array([ 0.21212121, -0.12121212])\n",
    "    >>> ridge_regression(np.array([0.1,0.2]), np.array([[2.3, 3.2], [1., 0.1]]), 1)\n",
    "    array([0.03947092, 0.00319628])\n",
    "    \"\"\"\n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test(ridge_regression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ridge_regression_demo(x, y, degree, ratio, seed):\n",
    "    \"\"\"ridge regression demo.\"\"\"\n",
    "    # define parameter\n",
    "    lambdas = np.logspace(-5, 0, 15)\n",
    "\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # split the data, and return train and test data: TODO\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError\n",
    "    # ***************************************************\n",
    "    # INSERT YOUR CODE HERE\n",
    "    # form train and test data with polynomial basis function: TODO\n",
    "    # ***************************************************\n",
    "    raise NotImplementedError\n",
    "\n",
    "    rmse_tr = []\n",
    "    rmse_te = []\n",
    "    for ind, lambda_ in enumerate(lambdas):\n",
    "        # ***************************************************\n",
    "        # INSERT YOUR CODE HERE\n",
    "        # ridge regression with a given lambda\n",
    "        # ***************************************************\n",
    "        raise NotImplementedError\n",
    "        print(\n",
    "            \"proportion={p}, degree={d}, lambda={l:.3f}, Training RMSE={tr:.3f}, Testing RMSE={te:.3f}\".format(\n",
    "                p=ratio, d=degree, l=lambda_, tr=rmse_tr[ind], te=rmse_te[ind]\n",
    "            )\n",
    "        )\n",
    "    plot_train_test(rmse_tr, rmse_te, lambdas, degree)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "Demo time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 56\n",
    "degree = 7\n",
    "split_ratio = 0.5\n",
    "ridge_regression_demo(x, y, degree, split_ratio, seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "Your plot should look like:"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "![alt text](ridge_regression.png)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}