{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Cristian Leo",
  "title_listed": "The Math Behind Gated Recurrent Units",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/",
  "publish_date": "2024-06-04",
  "corpus_stage": "",
  "notes": "",
  "id": "3f95a2e3c48e",
  "collected_at": "2026-10-02T06:14:59",
  "title": "The Math Behind Gated Recurrent Units | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# The Math Behind Gated Recurrent Units\nDive into advanced deep learning with gated recurrent units (GRUs), understand their mathematics, and implement them from scratch.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nJune 4, 202429 min read\n![Image generated by DALL-E](../images/000.jpg)\nImage generated by DALL-E\nGated Recurrent Units (GRUs) are a powerful type of recurrent neural network (RNN) designed to handle sequential data efficiently. In this article, we'll explore what GRUs are, and their math, and we will build an implementation from scratch in Python.\n* * *\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) and [Computer Science](https://roadmap.sh/computer-science) roadmaps.\n## Index\n[**1: Understanding the Basics**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#6a1a) ∘ [1.1: What are GRUs?](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#919d) ∘ [1.2: Comparison with LSTMs and Vanilla RNNs](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#ffb2) [**2: The Architecture of GRUs**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#9a12) ∘ [2.1: GRU Cell Structure](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#f047) [**3: The Mathematics of GRUs**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#c218) ∘ [3.1: Update Gate](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#798d) ∘ [3.2: Reset Gate](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#760d) ∘ [3.3: Candidate Activation](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#adc2) ∘ [3.4: Step-by-Step Example](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#6c28) [**4: Building GRU From Scratch in Python**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#49b9) ∘ [4.1: Imports and Custom Classes](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#3222) ∘ [4.2: GRU Class](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#4bd1) ∘ [4.3: GRUTrainer Class](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#71bd) ∘ [4.4: TimeSeriesDataset Class](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#3426) ∘ [4.5: Training the GRU Model](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#d983) [**5: Conclusion**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#790b)\n* * *\n## 1: Understanding the Basics\n### 1.1: What are GRUs?\nGRUs, or Gated Recurrent Units, are a type of recurrent neural network (RNN) introduced by [Kyunghyun Cho and his colleagues in 2014](https://arxiv.org/abs/1406.1078). Think of GRUs as a smarter version of traditional RNNs, designed to handle sequences of data more effectively.\nImagine you're trying to learn a song by listening to it repeatedly. A basic RNN might forget the beginning of the song by the time it gets to the end. GRUs solve this problem by using gates that control what information is remembered and what is forgotten.\nGRUs simplify the structure of Long Short-Term Memory (LSTM) networks by merging the input and forget gates into a single update gate and adding a reset gate. This makes them faster to train and easier to work with, while still keeping the ability to remember important information for a long time.\n> **Update Gate** : This gate decides how much of the past information should be carried forward to the future.\n> **Reset Gate** : This gate determines how much of the past information to forget.\nThese gates help GRUs maintain a balance between remembering important details and forgetting unimportant ones, similar to how you might focus on remembering the melody of a song while ignoring the background noise.\nGRUs are great for tasks where data comes in sequences, like predicting the stock market, understanding language, or even generating music. They can learn patterns in data by keeping track of past information and using it to make better predictions. This makes them incredibly useful for any application where understanding the context from previous data points is crucial.\n### 1.2: Comparison with LSTMs and Vanilla RNNs\nTo understand where GRUs fit in, let's compare them with LSTMs and Vanilla RNNs.\n**Vanilla RNNs** Think of Vanilla RNNs as the basic version of recurrent neural networks. They work by passing information from one time step to the next, like a relay race where each runner passes the baton to the next. However, they have a big flaw: _they tend to forget things over long sequences_. This is due to the vanishing gradient problem, which makes it hard for them to learn long-term dependencies in data.\nHere's my previous article about Recurrent Neural Networks, where we build them from scratch:\n> [**The Math Behind Recurrent Neural Networks**](https://towardsdatascience.com/the-math-behind-recurrent-neural-networks-2de4e0098ab8)\n**LSTMs** Long Short-Term Memory Networks were designed to fix this problem. They use a more complex structure with three types of gates: _input, forget, and output gates_. These gates act like a sophisticated filing system, deciding what information to keep, what to update, and what to discard. This allows LSTMs to remember important information for long periods, making them great for tasks where context over many time steps is crucial, like understanding paragraphs of text or recognizing patterns in long time series.\nHere's my previous article about LSTM:\n> [**The Math Behind LSTM**](https://towardsdatascience.com/the-math-behind-lstm-9069b835289d)\n**GRUs** Gated Recurrent Units are a streamlined version of LSTMs. They simplify things by combining the input and forget gates into a single update gate, and they also have a reset gate. This makes GRUs less computationally intensive and faster to train than LSTMs, while still being able to handle long-term dependencies effectively.\n## 2: The Architecture of GRUs\n![Gated Recurrent Unit Diagram - Image by Author](../images/001.png)\nGated Recurrent Unit Diagram - Image by Author\n### 2.1: GRU Cell Structure\nNow, let's take a look inside a single GRU cell. A GRU cell can be thought of as a small control unit in a factory that decides how much of the old information to keep and how much new information to add. It does this through two main components: the update gate and the reset gate. Let's explore these two concepts in more detail.\n**Update Gate:** This gate acts like a smart filter. It determines how much of the past information should be carried forward. If the update gate decides a piece of information is important, it will keep it; otherwise, it will discard it.\n**Reset Gate:** This gate decides how much of the past information to forget. When the reset gate is activated, it allows the cell to ignore parts of the past data, which helps in focusing on new, relevant information.\nEach GRU cell takes in the current input and the previous hidden state (like a memory of what happened before). The update gate is calculated using the current input and the previous hidden state. This helps the cell decide how much of the past information to keep.\nThe reset gate is calculated similarly, using the current input and the previous hidden state. This gate helps decide how much of the past information to forget.\nWith the reset gate applied, a candidate for the new hidden state is created by combining the current input and the previous state. This candidate represents the potential new information that could be added to the hidden state.\nThe update gate then decides the final hidden state by blending the previous hidden state and the candidate activation. This blend ensures that important past information is retained while incorporating relevant new information.\n## 3: The Mathematics of GRUs\nIn this section, we will go over the math behind GRU. I suggest you have the image at the top of the article displaying the GRU architecture, making it easier to see how the data flows and get transformed.\n### 3.1: Update Gate\nAs we said before, the update gate determines how much of the past information should be carried forward to influence the current state. This gate helps the GRU retain important information over long sequences, making it effective for handling sequential data.\nThe update gate _z _t_ ​ is calculated using the current input _x _t_ ​ and the previous hidden state _h _t_ −1​. The formula for the update gate is:\n![Update Gate Formula - Image by Author](../images/002.png)\nUpdate Gate Formula - Image by Author\nHere:\n  * **𝑧_𝑡 i** s the update gate at time step t.__\n  * **𝜎** is the sigmoid activation function, which squashes the output to be between 0 and 1.\n  * **𝑊_𝑧​** is the weight matrix for the update gate.\n  * **ℎ_𝑡−1** is the hidden state from the previous time step.\n  * **𝑥_𝑡​** is the current input.\n  * **[_h** _**t**_**−1​, _x** _**t**_**​]** represents the concatenation of the previous hidden state and the current input.\n\n\nLet's focus for a moment on [_h _t_ −1​, _x _t_ ​], which represents the concatenation of the previous hidden state _h _t_ −1​ and the current input _x _t_ ​. In the concatenation, two vectors are joined end-to-end to form a single larger vector. Suppose:\n  * _h _t_ −1​ is a vector of size _n_.\n  * _x _t_ ​ is a vector of size _m_.\n\n\nThen the concatenated vector [_h _t_ −1​, _x _t_ ​] is a new vector of size _n_ +_m_.\nMathematically, if:\n![Previous hidden state vector \\(left\\) and current input x vector \\(right\\) - Image by Author](../images/003.png)\nPrevious hidden state vector (left) and current input x vector (right) - Image by Author\nThen the concatenated vector [_ht_ −1​, _xt_ ​] is:\n![Concatenated Vector - Image by Author](../images/004.png)\nConcatenated Vector - Image by Author\nThis new vector [_h _t_ −1​, _x _t_ ​] combines all the elements of _h _t_ −1​ followed by all the elements of _x _t_ ​, forming a single column vector of size _n_ +_m_. The concatenation ensures that both the past hidden state and the current input contribute to the calculation.\nLet's now look at the role of the sigmoid function, _σ_ , which is defined as:\n![Sigmoid Function Formula - Image by Author](../images/005.png)\nSigmoid Function Formula - Image by Author\nThis function ensures that the values of the update gate _z _t_ ​ are between 0 and 1, which allows the gate to act as a filter. When _z _t_ ​ is close to 1, it means the GRU should retain a lot of the past information. When _z _t_ ​ is close to 0, it means the GRU should focus more on the new information.\nThink of the update gate as a valve that controls the flow of information from the past to the present. If the valve is wide open (i.e., _z _t_ ​ is close to 1), a lot of the past information is allowed through. If the valve is nearly closed (i.e., _z _t_ ​ is close to 0), very little past information is carried forward.\nThis mechanism allows the GRU to strike a balance between remembering important past information and incorporating new data, which is crucial for tasks like language processing, where the context provided by previous words can be essential for understanding the current word.\n### 3.2: Reset Gate\nThe reset gate in a GRU helps the model decide how much of the past information to forget. It's another crucial component that allows the GRU to manage dependencies over sequences effectively.\nThe reset gate _r _t_ ​ is calculated using the current input _x _t_ ​ and the previous hidden state _h _t_ −1​. The formula for the reset gate is:\n![Reset Gate Formula - Image by Author](../images/006.png)\nReset Gate Formula - Image by Author\nHere:\n  * _**r_t** _​ is the reset gate at time step _t_.\n  * _**σ**_ is the sigmoid activation function, which squashes the output between 0 and 1.\n  * _**W_r** _​ is the weight matrix for the reset gate.\n  * **_h** _**t**_**−1** ​ is the hidden state from the previous time step.\n  * **_x** _**t**_**​** is the current input.\n  * **[_h** _**t**_**−1​, _x** _**t**_**​]** is the concatenation of the previous hidden state and the current input, as described in the previous section.\n\n\nThink of the reset gate as a switch that determines how much of the past should be ignored when considering new information. If the reset gate value _r _t_ ​ is close to 0, it means the past information should be largely forgotten. If _r _t_ ​ is close to 1, the past information is mostly retained.\nIn practical terms, the reset gate allows the GRU to selectively reset its memory, which can be particularly useful when the model needs to process a new sequence of data or when the context changes significantly.\n### 3.3: Candidate Activation\nThe candidate activation in a GRU represents the new information that could potentially update the hidden state. This step integrates the reset gate's decision on how much past information to consider with the current input.\nThe candidate activation _h _t_ ​ is computed as follows:\n![Candidate Activation Formula - Image by Author](../images/007.png)\nCandidate Activation Formula - Image by Author\nHere:\n  * _**h_t** _​ is the candidate activation at time step _t_.\n  * **⁡tanh** is the hyperbolic tangent activation function, which outputs values between -1 and 1.\n  * _**W**_ is the weight matrix for the candidate activation.\n  * **_r** _**t**_**​** is the reset gate value at time step _t_ (_refer to section 3.2)_.\n  * **_h** _**t**_**−1** ​ is the hidden state from the previous time step.\n  * _**x_t** _​ is the current input.\n  * **[**_**rt**_**​∗** _**ht**_**−1​,**_**xt**_**​]** represents the concatenation of the reset-modified previous hidden state and the current input.\n\n\nIn this case, we are replacing the sigmoid function with tanh. The ⁡tanh function is defined as:\n![tanh Function Formula - Image by Author](../images/008.png)\ntanh Function Formula - Image by Author\n​This function outputs values between -1 and 1, allowing the GRU to introduce non-linearity into the model and control the range of the candidate activation.\nThink of the candidate activation as a proposal for the new hidden state, combining the new input with the relevant past information. The reset gate _r _t_ ​ decides how much of the previous hidden state _h _t_ −1​ to include in this proposal.\nIf _r _t_ ​ is close to 0, the reset gate effectively zeros out the past hidden state, meaning the candidate activation relies heavily on the new input _x _t_ ​. If _r _t_ ​ is close to 1, more of the past hidden state is considered, blending it with the new input to form _h _t_ ​.\nThe candidate activation _h _t_ ​ then contributes to the final hidden state _h _t_ ​ for the current time step. The update gate _z _t_ ​ (refer to section 3.1) determines how much of this candidate activation is mixed with the previous hidden state:\n![Updated Hidden State Formula - Image by Author](../images/009.png)\nUpdated Hidden State Formula - Image by Author\nWhere:\n  * **_z** _**t**_**​** is the update gate value at time step _t_ , calculated using the sigmoid function. It decides how much of the previous hidden state _h _t_ −1​ and the candidate activation _h _t_ ​ should contribute to the new hidden state.\n  * **_h** _**t**_**−1​** is the hidden state from the previous time step. It contains information about what the network has learned from the past inputs.\n  * **_h** _**t**_**​(tilde)** is the candidate activation for the current time step, calculated using the reset gate and the current input. It represents the new information that could potentially update the hidden state.\n  * **(1−_z** _**t**_**​)∗_h** _**t**_**−1** ​ controls how much of the previous hidden state is retained. When _z _t_ ​ is close to 0, the previous hidden state _h _t_ −1​ has a larger influence.\n  * **_z** _**t**_**​∗_h** _**t**_ ​(tilde) determines how much of the candidate activation _h _t_ ​ is added to the new hidden state. When _z _t_ ​ is close to 1, the candidate activation _h _t_ ​ has a larger influence.\n\n\n​By adjusting _z _t_ ​, the GRU can smoothly transition between preserving past information and incorporating new information. This balance helps the GRU retain important historical context while adapting to new inputs, making it effective for handling sequential data with varying dependencies.\n### 3.4: Step-by-Step Example\nTo understand how GRUs work in practice, let's walk through a simple example step-by-step. We'll use a small set of made-up numbers to illustrate the process.\nLet's assume:\n  * An input vector at time _t_ , _x _t_ ​= [0.5]\n  * A previous hidden state, _h _t_ −1​ = [0.3]\n  * Weights for the update gate, reset gate, and candidate activation are all 1 for simplicity.\n  * Biases are 0 for simplicity.\n\n\n**Step 1: Calculate the Update Gate** First, we calculate the update gate _z _t_ ​:\n![Update Gate Calculation - Image by Author](../images/010.png)\nUpdate Gate Calculation - Image by Author\nUsing the sigmoid function:\n![Sigmoid Function applied to Update Gate - Image by Author](../images/011.png)\nSigmoid Function applied to Update Gate - Image by Author\n**Step 2: Calculate the Reset Gate** Next, we calculate the reset gate _r _t_ ​:\n![Reset Gate Calculation - Image by Author](../images/012.png)\nReset Gate Calculation - Image by Author\nUsing the sigmoid function again:\n![Sigmoid Function applied to Reset Gate - Image by Author](../images/013.png)\nSigmoid Function applied to Reset Gate - Image by Author\n**Step 3: Calculate the Candidate Activation** Now, we calculate the candidate activation _h _t_ ​:\n![Candidate Activation Calculation - Image by Author](../images/014.png)\nCandidate Activation Calculation - Image by Author\nUsing the tanh function:\n![tanh Function applied to Candidate Activation - Image by Author](../images/body-1d737497b2f941c2f800.png)\ntanh Function applied to Candidate Activation - Image by Author\n**Step 4: Calculate the Final Hidden State** Finally, we calculate the new hidden state _h _t_ ​:\n![Final Hidden State Calculation - Image by Author](../images/body-89ea14df606fc83f6871.png)\nFinal Hidden State Calculation - Image by Author\n## 4: Building GRU From Scratch in Python\nThis section will guide you through implementing a Gated Recurrent Unit (GRU) network from scratch using Python. We will train the model over a flight passenger dataset. We will import the dataset from the `seaborn` library, which is [free for commercial use](https://github.com/mwaskom/seaborn/blob/master/LICENSE.md).\nYou can find all the code we'll cover here:\n> [**models-from-scratch-python/Gated Recurrent Unit/demo.ipynb at main ·...**](https://github.com/cristianleoo/models-from-scratch-python/blob/main/Gated%20Recurrent%20Unit/demo.ipynb)\n### 4.1: Imports and Custom Classes\nFirst, we need to import the necessary libraries and custom classes that will help us build and train our GRU model. We'll use `numpy` for numerical operations, `pandas` for dataset manipulation, and `seaborn` for plotting. Additionally, we'll use custom classes for weight initialization and early stopping.\npython\n\n```\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\n# Custom classes (built from scratch)\nfrom src.model import WeightInitializer\nfrom src.trainer import EarlyStopping\n```\n\nThese custom classes (`WeightInitializer` and `EarlyStopping`) were created in a previous article. The `WeightInitializer` allows us to choose between several weight initialization techniques, and `EarlyStopping` stops the training process when there is no significant improvement in performance, preventing overfitting.\nYou can find the code for `WeightInitializer` and `EarlyStopping` in the article about LSTM:\n> [**The Math Behind LSTM**](https://towardsdatascience.com/the-math-behind-lstm-9069b835289d)\n### 4.2: GRU Class\nHere, we define the `GRU` class, which includes initialization, forward propagation, and backward propagation methods.\n**Initialization** The `__init__` method initializes the GRU with a specified _input size_ , _hidden size_ , and _output size_. It also initializes weights and biases using a custom weight initializer.\npython\n\n```\nclass GRU:\n    def __init__(self, input_size, hidden_size, output_size, init_method='xavier'):\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.output_size = output_size\n        self.weight_initializer = WeightInitializer(method=init_method)\n\n        # Initialize weights\n        self.wz = self.weight_initializer.initialize((hidden_size, hidden_size + input_size))\n        self.wr = self.weight_initializer.initialize((hidden_size, hidden_size + input_size))\n        self.wh = self.weight_initializer.initialize((hidden_size, hidden_size + input_size))\n\n        # Initialize biases\n        self.bz = np.zeros((hidden_size, 1))\n        self.br = np.zeros((hidden_size, 1))\n        self.bh = np.zeros((hidden_size, 1))\n\n        # Initialize output layer weights and biases\n        self.why = self.weight_initializer.initialize((output_size, hidden_size))\n        self.by = np.zeros((output_size, 1))\n```\n\nFirst, the initialization stores the sizes of the input, hidden, and output layers as class attributes. It also creates an instance of a WeightInitializer, using the specified method to initialize the weights, in this case, Xavier initialization.\nNext, it initializes the weights for the update gate (`wz`), reset gate (`wr`), and candidate hidden state (`wh`). Each of these weights is a matrix that combines the hidden state and input dimensions.\nAfter that, it initializes the biases for the update gate (`bz`), reset gate (`br`), and candidate hidden state (`bh`). These biases are vectors filled with zeros, one for each hidden unit.\nFinally, it initializes the weights and biases for the output layer. The weights (`why`) are a matrix connecting the hidden state to the output, and the biases (`by`) are a vector filled with zeros, one for each output unit.\n**Activation Functions** Next, we define static methods for the sigmoid function and its derivative, and the derivative of the tanh function.\nless\n\n```\n    @staticmethod\n    def sigmoid(z):\n        return 1 / (1 + np.exp(-z))\n\n    @staticmethod\n    def dsigmoid(y):\n        return y * (1 - y)\n\n    @staticmethod\n    def dtanh(y):\n        return 1 - y * y\n```\n\n**Forward Propagation** The `forward` method processes the input through the GRU cell for each time step, calculating the reset and update gates, candidate activation, and the new hidden state.\npython\n\n```\n    def forward(self, x):\n        caches = []\n        h_prev = np.zeros((self.hidden_size, 1))\n        h = h_prev\n\n        for t in range(x.shape[0]):\n            x_t = x[t].reshape(-1, 1)\n            combined = np.vstack((h_prev, x_t))\n\n            z = self.sigmoid(np.dot(self.wz, combined) + self.bz)\n            r = self.sigmoid(np.dot(self.wr, combined) + self.br)\n            combined_r = np.vstack((r * h_prev, x_t))\n            h_ = np.tanh(np.dot(self.wh, combined_r) + self.bh)\n\n            h = (1 - z) * h_prev + z * h_\n\n            cache = (h_prev, z, r, h_, x_t, combined, combined_r, h)\n            caches.append(cache)\n\n            h_prev = h\n\n        y = np.dot(self.why, h) + self.by\n        return y, caches\n```\n\nFirst, it initializes an empty list called `caches` to store intermediate values needed for backpropagation later. It also initializes the previous hidden state `h_prev` as a column vector of zeros with a size equal to the hidden layer size.\npython\n\n```\ncaches = []\nh_prev = np.zeros((self.hidden_size, 1))\nh = h_prev\n```\n\nThen, for each time step in the input sequence, it extracts the current input `x_t` and reshapes it into a column vector. It concatenates this input with the previous hidden state to form a combined input.\npython\n\n```\nfor t in range(x.shape[0]):\n    x_t = x[t].reshape(-1, 1)\n    combined = np.vstack((h_prev, x_t))\n```\n\nNext, it calculates the update gate `z` by applying the sigmoid function to the dot product of the update gate weights and the combined input, plus the update gate bias. Similarly, it calculates the reset gate `r` by applying the sigmoid function to the dot product of the reset gate weights and the combined input, plus the reset gate bias.\npython\n\n```\nz = self.sigmoid(np.dot(self.wz, combined) + self.bz)\nr = self.sigmoid(np.dot(self.wr, combined) + self.br)\n```\n\nThe reset gate `r` is then used to modify the previous hidden state, which is concatenated with the current input to form `combined_r`. This combined input is then used to calculate the candidate hidden state `h_` by applying the tanh function to the dot product of the candidate hidden state weights and `combined_r`, plus the candidate hidden state bias.\nini\n\n```\ncombined_r = np.vstack((r * h_prev, x_t))\nh_ = np.tanh(np.dot(self.wh, combined_r) + self.bh)\n```\n\nThe new hidden state `h` is computed as a combination of the previous hidden state and the candidate hidden state, weighted by the update gate `z`.\nini\n\n```\nh = (1 - z) * h_prev + z * h_\n```\n\nAll these intermediate values (`h_prev`, `z`, `r`, `h_`, `x_t`, `combined`, `combined_r`, and `h`) are stored in a tuple called `cache`, which is appended to the `caches` list.\ncpp\n\n```\ncache = (h_prev, z, r, h_, x_t, combined, combined_r, h)\ncaches.append(cache)\n```\n\nThe loop updates the previous hidden state `h_prev` to the current hidden state `h` for the next time step.\npython\n\n```\nh_prev = h\n```\n\nAfter processing all time steps, the method calculates the final output `y` by applying the output layer weights and biases to the last hidden state.\npython\n\n```\ny = np.dot(self.why, h) + self.by\n```\n\nFinally, the method returns the output `y` and the list of caches `caches`.\npython\n\n```\nreturn y, caches\n```\n\n**Backward Propagation** The `backward` method calculates gradients using the caches from the forward pass and performs gradient clipping to prevent exploding gradients.\npython\n\n```\n    def backward(self, dy, caches, clip_value=1.0):\n        \"\"\"\n        Backward pass through the GRU network.\n\n        Parameters:\n        - dy: np.ndarray, gradient of the loss with respect to the output\n        - caches: list, caches from the forward pass\n        - clip_value: float, value to clip gradients to (default: 1.0)\n\n        Returns:\n        - tuple, gradients of the loss with respect to the parameters\n        \"\"\"\n        # Initialize gradients\n        dWz, dWr, dWh = [np.zeros_like(w) for w in (self.wz, self.wr, self.wh)]\n        dbz, dbr, dbh = [np.zeros_like(b) for b in (self.bz, self.br, self.bh)]\n        dWhy = np.zeros_like(self.why)\n        dby = np.zeros_like(self.by)\n\n        # Ensure dy is reshaped to match output size\n        dy = dy.reshape(self.output_size, -1)\n        dh_next = np.zeros((self.hidden_size, 1))  # shape must match hidden_size\n\n        # Backpropagate through time\n        for cache in reversed(caches):\n            h_prev, z, r, h_, x_t, combined, combined_r, h = cache\n\n            # Add gradient from next step to current output gradient\n            dh = np.dot(self.why.T, dy) + dh_next\n            dh_ = dh * z * self.dtanh(h_)\n            dz = dh * (h_ - h_prev) * self.dsigmoid(z)\n            dr = np.dot(self.wh[:, :self.hidden_size].T, dh_) * h_prev * self.dsigmoid(r)\n\n            # Update gradients with respect to weights and biases\n            dWz += np.dot(dz, combined.T)\n            dWr += np.dot(dr, combined.T)\n            dWh += np.dot(dh_, combined_r.T)\n\n            dbz += dz.sum(axis=1, keepdims=True)\n            dbr += dr.sum(axis=1, keepdims=True)\n            dbh += dh_.sum(axis=1, keepdims=True)\n\n            # Compute the gradient for the previous time step\n            dh_prev = dh * (1 - z) + np.dot(self.wz[:, :self.hidden_size].T, dz) + np.dot(self.wr[:, :self.hidden_size].T, dr)\n\n            dh_next = dh_prev\n\n        # Update output layer weights and biases\n        dWhy += np.dot(dy, h.T)\n        dby += dy\n\n        gradients = (dWz, dWr, dWh, dbz, dbr, dbh, dWhy, dby)\n\n        # Gradient clipping\n        for i in range(len(gradients)):\n            np.clip(gradients[i], -clip_value, clip_value, out=gradients[i])\n\n        return gradients\n```\n\nFirst, the method initializes gradients for the _update gate_ weights (`dWz`), reset gate weights (`dWr`), candidate hidden state weights (`dWh`), and their respective biases (`dbz`, `dbr`, `dbh`). It also initializes gradients for the output layer weights (`dWhy`) and biases (`dby`). All these gradients are initialized to arrays of zeros with the same shape as the corresponding weights and biases.\npython\n\n```\ndWz, dWr, dWh = [np.zeros_like(w) for w in (self.wz, self.wr, self.wh)]\ndbz, dbr, dbh = [np.zeros_like(b) for b in (self.bz, self.br, self.bh)]\ndWhy = np.zeros_like(self.why)\ndby = np.zeros_like(self.by)\n```\n\nNext, the method reshapes the gradient of the loss with respect to the output (`dy`) to ensure it matches the output size. It also initializes `dh_next`, the gradient of the loss with respect to the next hidden state, to a column vector of zeros with a size equal to the hidden layer size.\npython\n\n```\ndy = dy.reshape(self.output_size, -1)\ndh_next = np.zeros((self.hidden_size, 1))\n```\n\nThe method then iterates backward through the stored caches from the forward pass. For each cache, which contains intermediate values from a specific time step, it extracts the cached values: `h_prev`, `z`, `r`, `h_`, `x_t`, `combined`, `combined_r`, and `h`.\npython\n\n```\nfor cache in reversed(caches):\n    h_prev, z, r, h_, x_t, combined, combined_r, h = cache\n```\n\nIt calculates the gradient of the loss with respect to the hidden state (`dh`) by adding the gradient from the next time step (`dh_next`) to the dot product of the output layer weights transpose and `dy`. It then computes the gradients of the candidate hidden state (`dh_`), update gate (`dz`), and reset gate (`dr`).\npython\n\n```\n# Add gradient from next step to current output gradient\ndh = np.dot(self.why.T, dy) + dh_next\ndh_ = dh * z * self.dtanh(h_)\ndz = dh * (h_ - h_prev) * self.dsigmoid(z)\ndr = np.dot(self.wh[:, :self.hidden_size].T, dh_) * h_prev * self.dsigmoid(r)\n```\n\nIn this step, we update the gradients of the weights and biases. Specifically, it computes the gradients with respect to the combined input (`dWz`, `dWr`) and the combined reset input (`dWh`). It also updates the gradients of the biases (`dbz`, `dbr`, `dbh`).\npython\n\n```\n# Update gradients with respect to weights and biases\ndWz += np.dot(dz, combined.T)\ndWr += np.dot(dr, combined.T)\ndWh += np.dot(dh_, combined_r.T)\n\ndbz += dz.sum(axis=1, keepdims=True)\ndbr += dr.sum(axis=1, keepdims=True)\ndbh += dh_.sum(axis=1, keepdims=True)\n```\n\nAfter that, it calculates the gradient of the loss with respect to the previous hidden state (`dh_prev`) to propagate the gradient back to the previous time step. The gradient `dh_prev` is a combination of the gradient of the loss with respect to the current hidden state and the contributions from the update gate and reset gate weights.\npython\n\n```\n# Compute the gradient for the previous time step\ndh_prev = dh * (1 - z) + np.dot(self.wz[:, :self.hidden_size].T, dz) + np.dot(self.wr[:, :self.hidden_size].T, dr)\n```\n\nThis process continues for each time step, updating `dh_next` with `dh_prev` for the next iteration.\npython\n\n```\ndh_next = dh_prev\n```\n\nOnce the backpropagation through time is complete, the method updates the gradients of the output layer weights (`dWhy`) and biases (`dby`) using the gradient of the loss with respect to the output.\npython\n\n```\n# Update output layer weights and biases\ndWhy += np.dot(dy, h.T)\ndby += dy\n```\n\nFinally, it clips the gradients to a specified value (`clip_value`) to prevent exploding gradients, ensuring that all gradient values are within the range [-clip_value, clip_value]. The method returns the computed gradients for all parameters as a tuple.\npython\n\n```\ngradients = (dWz, dWr, dWh, dbz, dbr, dbh, dWhy, dby)\n\n# Gradient clipping\nfor i in range(len(gradients)):\n    np.clip(gradients[i], -clip_value, clip_value, out=gradients[i])\n\nreturn gradients\n```\n\n**Updating Parameters** The `update_params` method applies the gradients to update the model parameters.\npython\n\n```\n    def update_params(self, grads, learning_rate):\n        dWz, dWr, dWh, dbz, dbr, dbh, dWhy, dby = grads\n\n        self.wz -= learning_rate * dWz\n        self.wr -= learning_rate * dWr\n        self.wh -= learning_rate * dWh\n\n        self.bz -= learning_rate * dbz\n        self.br -= learning_rate * dbr\n        self.bh -= learning_rate * dbh\n\n        self.why -= learning_rate * dWhy\n        self.by -= learning_rate * dby\n```\n\nFirst, the method unpacks the `grads` tuple into individual gradient variables: `dWz`, `dWr`, `dWh`, `dbz`, `dbr`, `dbh`, `dWhy`, and `dby`. These represent the gradients of the loss with respect to the update gate weights, reset gate weights, candidate hidden state weights, their corresponding biases, and the output layer weights and biases, respectively.\nThen, for each parameter, the method performs a gradient descent step. Specifically, it subtracts the product of the gradient and the learning rate from the current value of the parameter. This operation is performed for the update gate weights (`wz`), reset gate weights (`wr`), candidate hidden state weights (`wh`), their biases (`bz`, `br`, `bh`), and the output layer weights (`why`) and biases (`by`).\nTherefore, this method adjusts the parameters of the GRU network in the direction that reduces the loss. The learning rate controls how big each adjustment step is, ensuring that the parameters are updated incrementally based on the gradients calculated during backpropagation. This process is repeated for each parameter, resulting in an updated set of weights and biases for the network.\n### 4.3: GRUTrainer Class\nThe `GRUTrainer` class handles the training process, including forward and backward passes, and updates the model parameters.\n**Initialization** The `__init__` method initializes the trainer with the GRU model, learning rate, and early stopping criteria.\npython\n\n```\nclass GRUTrainer:\n    def __init__(self, model, learning_rate=0.01, patience=7, verbose=True, delta=0):\n        self.model = model\n        self.learning_rate = learning_rate\n        self.train_losses = []\n        self.val_losses = []\n        self.early_stopping = EarlyStopping(patience, verbose, delta)\n```\n\nThis class takes four parameters: `model`, which is the GRU model to be trained; `learning_rate`, which specifies the step size for updating the model's parameters during training (default is 0.01); `patience`, which defines how many epochs to wait for an improvement in validation loss before stopping the training early (default is 7); and `verbose`, a boolean that controls whether to print training information (default is True). It also accepts `delta`, which is the minimum change in validation loss to qualify as an improvement (default is 0).\nInside the `__init__` method, it sets these parameters as attributes of the `GRUTrainer` instance. It initializes an empty list `train_losses` to keep track of the training loss for each epoch and another empty list `val_losses` to store the validation loss for each epoch.\nFinally, it initializes an instance of `EarlyStopping` with the given `patience`, `verbose`, and `delta` parameters. This instance will be used to monitor the validation loss and determine when to stop training if the loss does not improve within the specified _patience_ period.\n**Training Method** The `train` method trains the GRU model on the training data and optionally validates it on a separate validation set.\npython\n\n```\n    def train(self, X_train, y_train, X_val=None, y_val=None, epochs=10, batch_size=1, clip_value=1.0):\n        for epoch in range(epochs):\n            epoch_losses = []\n            for i in range(0, len(X_train), batch_size):\n                batch_X = X_train[i:i + batch_size]\n                batch_y = y_train[i:i + batch_size]\n                losses = []\n\n                for x, y_true in zip(batch_X, batch_y):\n                    y_pred, caches = self.model.forward(x)\n                    loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))\n                    losses.append(loss)\n\n                    dy = y_pred - y_true.reshape(-1, 1)\n                    grads = self.model.backward(dy, caches, clip_value=clip_value)\n                    self.model.update_params(grads, self.learning_rate)\n\n                batch_loss = np.mean(losses)\n                epoch_losses.append(batch_loss)\n\n            avg_epoch_loss = np.mean(epoch_losses)\n            self.train_losses.append(avg_epoch_loss)\n\n            if X_val is not None and y_val is not None:\n                val_loss = self.validate(X_val, y_val)\n                self.val_losses.append(val_loss)\n\n                if epoch % 10 == 0:\n                    print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}, Val Loss: {val_loss:.5f}')\n\n                self.early_stopping(val_loss)\n                if self.early_stopping.early_stop:\n                    print(\"Early stopping\")\n                    break\n            else:\n                print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}')\n```\n\nFirst, the method iterates through the number of epochs specified. For each epoch, it initializes an empty list `epoch_losses` to store the loss for each batch.\npython\n\n```\nfor epoch in range(epochs):\n    epoch_losses = []\n```\n\nWithin each epoch, it iterates over the training data in mini-batches of size `batch_size`. For each batch, it extracts the current batch of inputs (`batch_X`) and their corresponding labels (`batch_y`). It initializes an empty list `losses` to store the loss for each sample in the batch.\npython\n\n```\nfor i in range(0, len(X_train), batch_size):\n    batch_X = X_train[i:i + batch_size]\n    batch_y = y_train[i:i + batch_size]\n    losses = []\n```\n\nFor each sample in the current batch, it performs the forward pass by calling the `forward` method of the GRU model, which returns the predicted output (`y_pred`) and the caches containing intermediate values needed for backpropagation. It then computes the loss between the predicted output and the true label (`y_true`), reshaped to match the prediction's shape, and appends this loss to the `losses` list.\npython\n\n```\nfor x, y_true in zip(batch_X, batch_y):\n    y_pred, caches = self.model.forward(x)\n    loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))\n    losses.append(loss)\n```\n\nNext, it computes the gradient of the loss with respect to the output (`dy`) and performs backpropagation by calling the `backward` method of the GRU model, which returns the gradients for the model's parameters. It then updates the model's parameters using these gradients by calling the `update_params` method with the computed gradients and the learning rate.\npython\n\n```\ndy = y_pred - y_true.reshape(-1, 1)\ngrads = self.model.backward(dy, caches, clip_value=clip_value)\nself.model.update_params(grads, self.learning_rate)\n```\n\nAfter processing all samples in the current batch, it calculates the mean loss for the batch (`batch_loss`) and appends it to the `epoch_losses` list.\ngo\n\n```\nbatch_loss = np.mean(losses)\nepoch_losses.append(batch_loss)\n```\n\nOnce all batches are processed for the current epoch, it calculates the average loss for the epoch (`avg_epoch_loss`) and appends it to the `train_losses` list.\npython\n\n```\navg_epoch_loss = np.mean(epoch_losses)\nself.train_losses.append(avg_epoch_loss)\n```\n\nIf validation data is provided (`X_val`, `y_val`), the method validates the model by calling the `validate` method, which returns the validation loss (`val_loss`). This validation loss is appended to the `val_losses` list.\npython\n\n```\nif X_val is not None and y_val is not None:\n    val_loss = self.validate(X_val, y_val)\n    self.val_losses.append(val_loss)\n```\n\nEvery 10 epochs, the method prints the current epoch number, the average training loss, and the validation loss if available.\npython\n\n```\nif epoch % 10 == 0:\n    print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}, Val Loss: {val_loss:.5f}')\n\n    self.early_stopping(val_loss)\n```\n\nThe method then checks for early stopping by calling the `early_stopping` method with the current validation loss. If early stopping is triggered, indicated by the `early_stop` attribute of `early_stopping` being `True`, it prints a message and breaks out of the training loop.\nlua\n\n```\nif self.early_stopping.early_stop:\n    print(\"Early stopping\")\n    break\n```\n\nIf validation data is not provided, it simply prints the current epoch number and the average training loss for each epoch.\npython\n\n```\nelse:\n    print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}')\n```\n\n**Compute Loss and Validate Methods** The `compute_loss` method calculates the mean squared error loss and the `validate` method evaluates the model on validation data.\npython\n\n```\ndef compute_loss(self, y_pred, y_true):\n    return np.mean((y_pred - y_true) ** 2)\n```\n\nThe `compute_loss` method in the `GRUTrainer` class calculates the mean squared error loss between the predicted output (`y_pred`) and the true label (`y_true`). This method takes two arguments: `y_pred`, the predictions made by the model, and `y_true`, the actual labels. It computes the element-wise difference between `y_pred` and `y_true`, squares these differences, and then takes the mean of these squared differences to get the mean squared error loss.\npython\n\n```\ndef validate(self, X_val, y_val):\n    val_losses = []\n    for x, y_true in zip(X_val, y_val):\n        y_pred, _ = self.model.forward(x)\n        loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))\n        val_losses.append(loss)\n    return np.mean(val_losses)\n```\n\nThe `validate` method evaluates the model on a separate validation dataset to check its performance. This method takes two arguments: `X_val`, the validation inputs, and `y_val`, the validation labels.\nFirst, it initializes an empty list `val_losses` to store the loss for each validation sample. It then iterates over each sample in the validation dataset, extracting the input `x` and the corresponding true label `y_true`.\nFor each sample, it performs the forward pass by calling the `forward` method of the GRU model, which returns the predicted output (`y_pred`). It then computes the loss between the predicted output and the true label, reshaped to match the prediction's shape, by calling the `compute_loss` method. This loss is appended to the `val_losses` list.\nAfter processing all validation samples, it calculates the mean of the collected losses (`val_losses`) and returns this mean value as the validation loss. This provides an overall measure of the model's performance on the validation dataset.\n### 4.4: TimeSeriesDataset Class\nThe `TimeSeriesDataset` class is designed to handle time series data, specifically for loading, preprocessing, and creating datasets suitable for training and testing a model. It takes two optional parameters: `look_back`, which determines the number of previous time steps to use as input features (default is 1), and `train_size`, which specifies the proportion of data to be used for training (default is 0.67).\npython\n\n```\nclass TimeSeriesDataset:\n    def __init__(self, look_back=1, train_size=0.67):\n        self.look_back = look_back\n        self.train_size = train_size\n\n    def load_data(self):\n        df = sns.load_dataset('flights')\n        df['Date']  = pd.to_datetime(df['year'].astype(str) + '-' + df['month'].astype(str) + '-01')\n        df = df.set_index('Date')\n        df = df.drop(['year', 'month'], axis=1)\n        df = df.values.astype(float)\n        df = self.MinMaxScaler(df)\n        train_size = int(len(df) * self.train_size)\n        train, test = df[0:train_size,:], df[train_size:len(df),:]\n        return train, test\n\n    def MinMaxScaler(self, data):\n        numerator = data - np.min(data, 0)\n        denominator = np.max(data, 0) - np.min(data, 0)\n        return numerator / (denominator + 1e-7)\n\n    def create_dataset(self, dataset):\n        dataX, dataY = []\n        for i in range(len(dataset)-self.look_back):\n            a = dataset[i:(i + self.look_back), 0]\n            dataX.append(a)\n            dataY.append(dataset[i + self.look_back, 0])\n        return np.array(dataX), np.array(dataY)\n\n    def get_train_test(self):\n        train, test = self.load_data()\n        trainX, trainY = self.create_dataset(train)\n        testX, testY = self.create_dataset(test)\n        return trainX, trainY, testX, testY\n```\n\nThe `load_data` method loads a dataset of flight data using Seaborn's `load_dataset` function. It converts the 'year' and 'month' columns into a datetime format and sets this `datetime` as the index, it then drops the original 'year' and 'month' columns, leaving only the 'passengers' column. The method converts this column to a float type and scales the values between 0 and 1 using the `MinMaxScaler` method. It then splits the scaled data into training and testing sets based on the `train_size` attribute. The training set consists of the first `train_size` proportion of the data, and the testing set consists of the remaining data. The method returns these two sets.\npython\n\n```\ndef load_data(self):\n    df = sns.load_dataset('flights')\n    df['Date']  = pd.to_datetime(df['year'].astype(str) + '-' + df['month'].astype(str) + '-01')\n    df = df.set_index('Date')\n    df = df.drop(['year', 'month'], axis=1)\n    df = df.values.astype(float)\n    df = self.MinMaxScaler(df)\n    train_size = int(len(df) * self.train_size)\n    train, test = df[0:train_size,:], df[train_size:len(df),:]\n    return train, test\n```\n\nThe `MinMaxScaler` method takes in the data and normalizes it. It subtracts the minimum value of each feature from the data and then divides it by the range (maximum value minus minimum value) of each feature. This ensures that the data values are scaled between 0 and 1.\npython\n\n```\ndef MinMaxScaler(self, data):\n    numerator = data - np.min(data, 0)\n    denominator = np.max(data, 0) - np.min(data, 0)\n    return numerator / (denominator + 1e-7)\n```\n\nThe `create_dataset` method creates input-output pairs for the model. It takes the dataset and initializes two empty lists, `dataX` and `dataY`. For each time step in the dataset (up to `len(dataset) - self.look_back`), it creates a sequence of `self.look_back` previous time steps as the input (`dataX`) and the next time step as the output (`dataY`). These pairs are appended to the respective lists. The method returns these lists as numpy arrays.\npython\n\n```\ndef create_dataset(self, dataset):\n    dataX, dataY = []\n    for i in range(len(dataset)-self.look_back):\n        a = dataset[i:(i + self.look_back), 0]\n        dataX.append(a)\n        dataY.append(dataset[i + self.look_back, 0])\n    return np.array(dataX), np.array(dataY)\n```\n\nThe `get_train_test` method combines the previous methods to produce the final training and testing datasets. It first calls `load_data` to obtain the training and testing sets. Then, it calls `create_dataset` on each set to generate the input-output pairs for both training and testing. Finally, it returns the training inputs (`trainX`), training outputs (`trainY`), testing inputs (`testX`), and testing outputs (`testY`). This method provides a complete pipeline for preparing the data for training and testing a model.\npython\n\n```\ndef get_train_test(self):\n    train, test = self.load_data()\n    trainX, trainY = self.create_dataset(train)\n    testX, testY = self.create_dataset(test)\n    return trainX, trainY, testX, testY\n```\n\n### 4.5: Training the GRU Model\nFirst, we instantiate the `TimeSeriesDataset` class with a `train_size` of 0.7, meaning 70% of the data will be used for training and a `look_back` of 1, meaning each input feature will consist of one previous time step. We call the `get_train_test` method of the dataset instance to obtain the training and testing datasets. The method returns four arrays: `trainX`, `trainY`, `testX`, and `testY`.\npython\n\n```\n# Instantiate the dataset\ndataset = TimeSeriesDataset(train_size=0.7, look_back=1)\ntrainX, trainY, testX, testY = dataset.get_train_test()\n```\n\nNext, we reshape `trainX` and `testX` to be in the format [samples, time steps, features]. This is done using NumPy's `reshape` function. For `trainX`, the new shape is (number of samples, number of time steps, 1), where the last dimension represents the single feature (passenger count). The same reshaping is applied to `testX`.\npython\n\n```\n# Reshape input to be [samples, time steps, features]\ntrainX = np.reshape(trainX, (trainX.shape[0], trainX.shape[1], 1))\ntestX = np.reshape(testX, (testX.shape[0], testX.shape[1], 1))\n```\n\nWe then instantiate the `GRU` class with an `input_size` of 1 (since each input has one feature), a `hidden_size` of 256 (the number of GRU units in the hidden layer), and an `output_size` of 1 (since we want to predict one value, the passenger count for the next time step).\npython\n\n```\ngru = GRU(input_size=1, hidden_size=256, output_size=1)\n```\n\nNext, we instantiate the `GRUTrainer` class, passing in the GRU model instance. We set the `learning_rate` to 0.001, `patience` to 50 (the number of epochs to wait for an improvement in validation loss before stopping early), `verbose` to True (so training information will be printed), and `delta` to 0.001 (the minimum change in validation loss to qualify as an improvement).\npython\n\n```\ntrainer = GRUTrainer(gru, learning_rate=1e-3, patience=50, verbose=True, delta=0.001)\n```\n\nFinally, we call the `train` method of the `GRUTrainer` instance to train the GRU model. We pass in the reshaped training and testing datasets (`trainX`, `trainY`, `testX`, `testY`), set the number of training epochs to 1000, the batch size to 32, and the gradient clipping value to 1.0. The training process will iteratively update the model's parameters to minimize the loss of the training data while monitoring the validation loss for early stopping.\npython\n\n```\ntrainer.train(trainX, trainY, testX, testY, epochs=1000, batch_size=32)\n```\n\n## 5: Conclusion\nIn this article, we explored Gated Recurrent Units (GRUs) and how they work. We started by comparing GRUs to LSTMs and Vanilla RNNs, then broke down the math behind key components like the update gate, reset gate, and candidate activation. We also provided a step-by-step guide to implementing a GRU from scratch in Python, with clear explanations of the code. Understanding and building GRUs helps us handle sequential data effectively, making them useful for tasks like time series prediction and natural language processing.\n## References\n  1. Cho, Kyunghyun, et al. \"Learning phrase representations using RNN encoder-decoder for statistical machine translation.\" [_arXiv preprint arXiv:1406.1078_](https://arxiv.org/abs/1406.1078) (2014).\n  2. Towards Data Science. \"The Math Behind Recurrent Neural Networks.\" _Towards Data Science_ (2020). [Link to Article](https://towardsdatascience.com/the-math-behind-recurrent-neural-networks-2de4e0098ab8)\n  3. Towards Data Science. \"The Math Behind LSTM.\" _Towards Data Science_ (2020). [Link to Article](https://towardsdatascience.com/the-math-behind-lstm-9069b835289d)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-8f73ac5a40c91a7b503f.jpeg)\nWritten By\n## [Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[See all from Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Mathematics](https://towardsdatascience.com/tag/mathematics/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F&title=The%20Math%20Behind%20Gated%20Recurrent%20Units)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F&title=The%20Math%20Behind%20Gated%20Recurrent%20Units)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F&text=The%20Math%20Behind%20Gated%20Recurrent%20Units)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/015.jpg)](https://towardsdatascience.com/running-sql-concurrently-across-three-remote-duckdb-servers-with-quack/)\n## [Running SQL Concurrently Across Three Remote DuckDB Servers with Quack](https://towardsdatascience.com/running-sql-concurrently-across-three-remote-duckdb-servers-with-quack/)\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\nA small experiment in remote SQL execution\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nAugust 16, 202617 min read\n  * [![](../images/016.jpg)](https://towardsdatascience.com/object-oriented-programming-a-practical-introduction-part-2-f376bc25f839/)\n## [Object-Oriented Programming: A Practical Introduction (Part 2)](https://towardsdatascience.com/object-oriented-programming-a-practical-introduction-part-2-f376bc25f839/)\n[Programming](https://towardsdatascience.com/category/programming/)\nLet's dig in to the more formal details of OOP.\n[Mark Douthwaite](https://towardsdatascience.com/author/mark-l-douthwaite/)\nOctober 9, 202012 min read\n  * [![](../images/017.jpeg)](https://towardsdatascience.com/how-to-use-machine-learning-to-inform-design-decisions-and-make-predictions-838106acf639/)\n## [How to Use Machine Learning to Inform Design Decisions and Make Predictions](https://towardsdatascience.com/how-to-use-machine-learning-to-inform-design-decisions-and-make-predictions-838106acf639/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAn Introductory Guide and Use Case for Applied Data Science\n[Kurt Klingensmith](https://towardsdatascience.com/author/kurt.klingensmith/)\nAugust 8, 202413 min read\n  * [![](../images/018.jpg)](https://towardsdatascience.com/exploring-the-ml-tooling-landscape-part-1-of-3-53f8c39e6e4e/)\n## [Exploring the ML Tooling Landscape (Part 1 of 3)](https://towardsdatascience.com/exploring-the-ml-tooling-landscape-part-1-of-3-53f8c39e6e4e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nML Maturity in Industry\n[Tom Martin](https://towardsdatascience.com/author/tpgmartin/)\nMay 3, 202210 min read\n  * [![](../images/019.gif)](https://towardsdatascience.com/terrificeye-an-edge-computing-system-for-traffic-analytics-from-videos-216cde072b74/)\n## [TerrificEye: An Edge Computing System For Traffic Analytics From Videos](https://towardsdatascience.com/terrificeye-an-edge-computing-system-for-traffic-analytics-from-videos-216cde072b74/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nHow I built a real-time, standalone traffic monitoring and analytics system\n[J. Rafid Siddiqui, PhD](https://towardsdatascience.com/author/azad-wolf/)\nMarch 6, 20227 min read\n  * [![](../images/020.png)](https://towardsdatascience.com/deep-learning-with-cifar-10-image-classification-64ab92110d79/)\n## [Deep Learning with CIFAR-10 Image Classification](https://towardsdatascience.com/deep-learning-with-cifar-10-image-classification-64ab92110d79/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nNeural Networks are the programmable patterns that helps to solve complex problems and bring the best achievable output. Deep Learning as...\n[Aarya Brahmane](https://towardsdatascience.com/author/aaryabrahmane/)\nOctober 26, 202010 min read\n  * [![](../images/021.jpeg)](https://towardsdatascience.com/learning-wolfram-resources-to-supercharge-your-machine-learning-projects-37d7472d5c02/)\n## [Learning Wolfram: Resources to Supercharge Your Machine Learning Projects](https://towardsdatascience.com/learning-wolfram-resources-to-supercharge-your-machine-learning-projects-37d7472d5c02/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nImprove Your Workflows and Productivity With Wolfram\n[Arnoud Buzing](https://towardsdatascience.com/author/arnoudbuzing/)\nSeptember 21, 20203 min read\n  * [![](../images/022.jpeg)](https://towardsdatascience.com/to-index-or-not-to-index-8be32ad45cae/)\n## [To Index or Not to Index](https://towardsdatascience.com/to-index-or-not-to-index-8be32ad45cae/)\n[Programming](https://towardsdatascience.com/category/programming/)\nLeverage SQL indexing to speed up your queries. Learn when to index, when not to, and how indexing works under the hood.\n[Christopher Karg](https://towardsdatascience.com/author/christopher_karg/)\nNovember 8, 202419 min read\n  * [![](../images/body-4e9bca8a9f9ba81206b0.jpg)](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/)\n## [RAG Workflow and Loop Engineering: The Dispatcher That Decides When to Loop and When to Stop](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #13] - Putting the patterns together, and why this is what “agentic RAG” should look like\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 14, 202627 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "metadata": {
    "title": "The Math Behind Gated Recurrent Units | Towards Data Science",
    "description": "Dive into advanced deep learning with gated recurrent units (GRUs), understand their mathematics, and implement them from scratch.",
    "keywords": null,
    "author": "Cristian Leo",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "The Math Behind Gated Recurrent Units",
    "og:description": "Dive into advanced deep learning with gated recurrent units (GRUs), understand their mathematics, and implement them from scratch.",
    "og:url": "https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/0TES6Qq35dFI2G04H.jpg",
    "og:image:width": "1792",
    "og:image:height": "1024",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Cristian Leo",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "29 min read",
    "article:published_time": "2024-06-04T06:15:02.000Z",
    "article:modified_time": "2024-06-04T06:15:02.000Z",
    "article:author": "https://towardsdatascience.com/author/cristianleo120/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_000.txt",
      "chars": 178
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_001.txt",
      "chars": 939
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_002.txt",
      "chars": 203
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_003.txt",
      "chars": 726
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_004.txt",
      "chars": 64
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_005.txt",
      "chars": 98
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_006.txt",
      "chars": 108
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_007.txt",
      "chars": 94
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_008.txt",
      "chars": 30
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_009.txt",
      "chars": 78
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_010.txt",
      "chars": 11
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_011.txt",
      "chars": 34
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_012.txt",
      "chars": 17
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_013.txt",
      "chars": 2231
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_014.txt",
      "chars": 204
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_015.txt",
      "chars": 80
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_016.txt",
      "chars": 90
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_017.txt",
      "chars": 246
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_018.txt",
      "chars": 260
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_019.txt",
      "chars": 165
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_020.txt",
      "chars": 18
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_021.txt",
      "chars": 75
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_022.txt",
      "chars": 194
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_023.txt",
      "chars": 424
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_024.txt",
      "chars": 304
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_025.txt",
      "chars": 1580
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_026.txt",
      "chars": 50
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_027.txt",
      "chars": 141
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_028.txt",
      "chars": 167
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_029.txt",
      "chars": 151
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_030.txt",
      "chars": 61
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_031.txt",
      "chars": 80
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_032.txt",
      "chars": 124
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_033.txt",
      "chars": 151
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_034.txt",
      "chars": 73
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_035.txt",
      "chars": 76
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_036.txt",
      "chars": 83
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_037.txt",
      "chars": 263
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_038.txt",
      "chars": 1319
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_039.txt",
      "chars": 427
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_040.txt",
      "chars": 165
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_041.txt",
      "chars": 274
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_042.txt",
      "chars": 194
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_043.txt",
      "chars": 139
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_044.txt",
      "chars": 184
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_045.txt",
      "chars": 56
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_046.txt",
      "chars": 86
    },
    {
      "language": "txt",
      "file": "attachments/3f95a2e3c48e/code/block_047.txt",
      "chars": 72
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:08:01Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:08:01Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:08:01Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:08:01Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/cristianleo120/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/the-math-behind-recurrent-neural-networks-2de4e0098ab8"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/the-math-behind-lstm-9069b835289d"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/mathematics/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/python/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/running-sql-concurrently-across-three-remote-duckdb-servers-with-quack/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/data-engineering/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/thomas_reid/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/object-oriented-programming-a-practical-introduction-part-2-f376bc25f839/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/computer-science"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/1406.1078"
    },
    {
      "type": "external",
      "href": "https://github.com/mwaskom/seaborn/blob/master/LICENSE.md"
    },
    {
      "type": "external",
      "href": "https://github.com/cristianleoo/models-from-scratch-python/blob/main/Gated%20Recurrent%20Unit/demo.ipynb"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=The+Math+Behind+Gated+Recurrent+Units&u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=The+Math+Behind+Gated+Recurrent+Units&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=The+Math+Behind+Gated+Recurrent+Units&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh"
    },
    {
      "type": "external",
      "href": "https://insightmediagroup.io/creator-network"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:33:18Z",
  "source_url": "https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# The Math Behind Gated Recurrent Units\nDive into advanced deep learning with gated recurrent units (GRUs), understand their mathematics, and implement them from scratch.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nJune 4, 202429 min read\n![Image generated by DALL-E](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/0TES6Qq35dFI2G04H.jpg)\nImage generated by DALL-E\nGated Recurrent Units (GRUs) are a powerful type of recurrent neural network (RNN) designed to handle sequential data efficiently. In this article, we'll explore what GRUs are, and their math, and we will build an implementation from scratch in Python.\n* * *\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) and [Computer Science](https://roadmap.sh/computer-science) roadmaps.\n## Index\n[**1: Understanding the Basics**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#6a1a) ∘ [1.1: What are GRUs?](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#919d) ∘ [1.2: Comparison with LSTMs and Vanilla RNNs](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#ffb2) [**2: The Architecture of GRUs**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#9a12) ∘ [2.1: GRU Cell Structure](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#f047) [**3: The Mathematics of GRUs**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#c218) ∘ [3.1: Update Gate](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#798d) ∘ [3.2: Reset Gate](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#760d) ∘ [3.3: Candidate Activation](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#adc2) ∘ [3.4: Step-by-Step Example](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#6c28) [**4: Building GRU From Scratch in Python**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#49b9) ∘ [4.1: Imports and Custom Classes](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#3222) ∘ [4.2: GRU Class](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#4bd1) ∘ [4.3: GRUTrainer Class](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#71bd) ∘ [4.4: TimeSeriesDataset Class](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#3426) ∘ [4.5: Training the GRU Model](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#d983) [**5: Conclusion**](https://towardsdatascience.com/the-math-behind-gated-recurrent-units-854d88aded65/#790b)\n* * *\n## 1: Understanding the Basics\n### 1.1: What are GRUs?\nGRUs, or Gated Recurrent Units, are a type of recurrent neural network (RNN) introduced by [Kyunghyun Cho and his colleagues in 2014](https://arxiv.org/abs/1406.1078). Think of GRUs as a smarter version of traditional RNNs, designed to handle sequences of data more effectively.\nImagine you're trying to learn a song by listening to it repeatedly. A basic RNN might forget the beginning of the song by the time it gets to the end. GRUs solve this problem by using gates that control what information is remembered and what is forgotten.\nGRUs simplify the structure of Long Short-Term Memory (LSTM) networks by merging the input and forget gates into a single update gate and adding a reset gate. This makes them faster to train and easier to work with, while still keeping the ability to remember important information for a long time.\n> **Update Gate** : This gate decides how much of the past information should be carried forward to the future.\n> **Reset Gate** : This gate determines how much of the past information to forget.\nThese gates help GRUs maintain a balance between remembering important details and forgetting unimportant ones, similar to how you might focus on remembering the melody of a song while ignoring the background noise.\nGRUs are great for tasks where data comes in sequences, like predicting the stock market, understanding language, or even generating music. They can learn patterns in data by keeping track of past information and using it to make better predictions. This makes them incredibly useful for any application where understanding the context from previous data points is crucial.\n### 1.2: Comparison with LSTMs and Vanilla RNNs\nTo understand where GRUs fit in, let's compare them with LSTMs and Vanilla RNNs.\n**Vanilla RNNs** Think of Vanilla RNNs as the basic version of recurrent neural networks. They work by passing information from one time step to the next, like a relay race where each runner passes the baton to the next. However, they have a big flaw: _they tend to forget things over long sequences_. This is due to the vanishing gradient problem, which makes it hard for them to learn long-term dependencies in data.\nHere's my previous article about Recurrent Neural Networks, where we build them from scratch:\n> [**The Math Behind Recurrent Neural Networks**](https://towardsdatascience.com/the-math-behind-recurrent-neural-networks-2de4e0098ab8)\n**LSTMs** Long Short-Term Memory Networks were designed to fix this problem. They use a more complex structure with three types of gates: _input, forget, and output gates_. These gates act like a sophisticated filing system, deciding what information to keep, what to update, and what to discard. This allows LSTMs to remember important information for long periods, making them great for tasks where context over many time steps is crucial, like understanding paragraphs of text or recognizing patterns in long time series.\nHere's my previous article about LSTM:\n> [**The Math Behind LSTM**](https://towardsdatascience.com/the-math-behind-lstm-9069b835289d)\n**GRUs** Gated Recurrent Units are a streamlined version of LSTMs. They simplify things by combining the input and forget gates into a single update gate, and they also have a reset gate. This makes GRUs less computationally intensive and faster to train than LSTMs, while still being able to handle long-term dependencies effectively.\n## 2: The Architecture of GRUs\n![Gated Recurrent Unit Diagram - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1asP3fK79CC2NAwGGKEVfDQ.png)\nGated Recurrent Unit Diagram - Image by Author\n### 2.1: GRU Cell Structure\nNow, let's take a look inside a single GRU cell. A GRU cell can be thought of as a small control unit in a factory that decides how much of the old information to keep and how much new information to add. It does this through two main components: the update gate and the reset gate. Let's explore these two concepts in more detail.\n**Update Gate:** This gate acts like a smart filter. It determines how much of the past information should be carried forward. If the update gate decides a piece of information is important, it will keep it; otherwise, it will discard it.\n**Reset Gate:** This gate decides how much of the past information to forget. When the reset gate is activated, it allows the cell to ignore parts of the past data, which helps in focusing on new, relevant information.\nEach GRU cell takes in the current input and the previous hidden state (like a memory of what happened before). The update gate is calculated using the current input and the previous hidden state. This helps the cell decide how much of the past information to keep.\nThe reset gate is calculated similarly, using the current input and the previous hidden state. This gate helps decide how much of the past information to forget.\nWith the reset gate applied, a candidate for the new hidden state is created by combining the current input and the previous state. This candidate represents the potential new information that could be added to the hidden state.\nThe update gate then decides the final hidden state by blending the previous hidden state and the candidate activation. This blend ensures that important past information is retained while incorporating relevant new information.\n## 3: The Mathematics of GRUs\nIn this section, we will go over the math behind GRU. I suggest you have the image at the top of the article displaying the GRU architecture, making it easier to see how the data flows and get transformed.\n### 3.1: Update Gate\nAs we said before, the update gate determines how much of the past information should be carried forward to influence the current state. This gate helps the GRU retain important information over long sequences, making it effective for handling sequential data.\nThe update gate _z _t_ ​ is calculated using the current input _x _t_ ​ and the previous hidden state _h _t_ −1​. The formula for the update gate is:\n![Update Gate Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/10T5jRFBoGYdk1gXGK-LJ-A.png)\nUpdate Gate Formula - Image by Author\nHere:\n  * **𝑧_𝑡 i** s the update gate at time step t.__\n  * **𝜎** is the sigmoid activation function, which squashes the output to be between 0 and 1.\n  * **𝑊_𝑧​** is the weight matrix for the update gate.\n  * **ℎ_𝑡−1** is the hidden state from the previous time step.\n  * **𝑥_𝑡​** is the current input.\n  * **[_h** _**t**_**−1​, _x** _**t**_**​]** represents the concatenation of the previous hidden state and the current input.\n\n\nLet's focus for a moment on [_h _t_ −1​, _x _t_ ​], which represents the concatenation of the previous hidden state _h _t_ −1​ and the current input _x _t_ ​. In the concatenation, two vectors are joined end-to-end to form a single larger vector. Suppose:\n  * _h _t_ −1​ is a vector of size _n_.\n  * _x _t_ ​ is a vector of size _m_.\n\n\nThen the concatenated vector [_h _t_ −1​, _x _t_ ​] is a new vector of size _n_ +_m_.\nMathematically, if:\n![Previous hidden state vector \\(left\\) and current input x vector \\(right\\) - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1Wzl6aTgSLYzz5WH893S9Rg.png)\nPrevious hidden state vector (left) and current input x vector (right) - Image by Author\nThen the concatenated vector [_ht_ −1​, _xt_ ​] is:\n![Concatenated Vector - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/15NPnHiAv2RhymQC_x__FGg.png)\nConcatenated Vector - Image by Author\nThis new vector [_h _t_ −1​, _x _t_ ​] combines all the elements of _h _t_ −1​ followed by all the elements of _x _t_ ​, forming a single column vector of size _n_ +_m_. The concatenation ensures that both the past hidden state and the current input contribute to the calculation.\nLet's now look at the role of the sigmoid function, _σ_ , which is defined as:\n![Sigmoid Function Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/1tJQ365_qOWLu7AJ5J-GnYQ.png)\nSigmoid Function Formula - Image by Author\nThis function ensures that the values of the update gate _z _t_ ​ are between 0 and 1, which allows the gate to act as a filter. When _z _t_ ​ is close to 1, it means the GRU should retain a lot of the past information. When _z _t_ ​ is close to 0, it means the GRU should focus more on the new information.\nThink of the update gate as a valve that controls the flow of information from the past to the present. If the valve is wide open (i.e., _z _t_ ​ is close to 1), a lot of the past information is allowed through. If the valve is nearly closed (i.e., _z _t_ ​ is close to 0), very little past information is carried forward.\nThis mechanism allows the GRU to strike a balance between remembering important past information and incorporating new data, which is crucial for tasks like language processing, where the context provided by previous words can be essential for understanding the current word.\n### 3.2: Reset Gate\nThe reset gate in a GRU helps the model decide how much of the past information to forget. It's another crucial component that allows the GRU to manage dependencies over sequences effectively.\nThe reset gate _r _t_ ​ is calculated using the current input _x _t_ ​ and the previous hidden state _h _t_ −1​. The formula for the reset gate is:\n![Reset Gate Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1h4a79yU4QLqtCc-pOHjJ-g.png)\nReset Gate Formula - Image by Author\nHere:\n  * _**r_t** _​ is the reset gate at time step _t_.\n  * _**σ**_ is the sigmoid activation function, which squashes the output between 0 and 1.\n  * _**W_r** _​ is the weight matrix for the reset gate.\n  * **_h** _**t**_**−1** ​ is the hidden state from the previous time step.\n  * **_x** _**t**_**​** is the current input.\n  * **[_h** _**t**_**−1​, _x** _**t**_**​]** is the concatenation of the previous hidden state and the current input, as described in the previous section.\n\n\nThink of the reset gate as a switch that determines how much of the past should be ignored when considering new information. If the reset gate value _r _t_ ​ is close to 0, it means the past information should be largely forgotten. If _r _t_ ​ is close to 1, the past information is mostly retained.\nIn practical terms, the reset gate allows the GRU to selectively reset its memory, which can be particularly useful when the model needs to process a new sequence of data or when the context changes significantly.\n### 3.3: Candidate Activation\nThe candidate activation in a GRU represents the new information that could potentially update the hidden state. This step integrates the reset gate's decision on how much past information to consider with the current input.\nThe candidate activation _h _t_ ​ is computed as follows:\n![Candidate Activation Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1ZbDMC_C-JljzoeGDMZfYqQ.png)\nCandidate Activation Formula - Image by Author\nHere:\n  * _**h_t** _​ is the candidate activation at time step _t_.\n  * **⁡tanh** is the hyperbolic tangent activation function, which outputs values between -1 and 1.\n  * _**W**_ is the weight matrix for the candidate activation.\n  * **_r** _**t**_**​** is the reset gate value at time step _t_ (_refer to section 3.2)_.\n  * **_h** _**t**_**−1** ​ is the hidden state from the previous time step.\n  * _**x_t** _​ is the current input.\n  * **[**_**rt**_**​∗** _**ht**_**−1​,**_**xt**_**​]** represents the concatenation of the reset-modified previous hidden state and the current input.\n\n\nIn this case, we are replacing the sigmoid function with tanh. The ⁡tanh function is defined as:\n![tanh Function Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1iYoEjn6Wi23pP1UylkxVjQ.png)\ntanh Function Formula - Image by Author\n​This function outputs values between -1 and 1, allowing the GRU to introduce non-linearity into the model and control the range of the candidate activation.\nThink of the candidate activation as a proposal for the new hidden state, combining the new input with the relevant past information. The reset gate _r _t_ ​ decides how much of the previous hidden state _h _t_ −1​ to include in this proposal.\nIf _r _t_ ​ is close to 0, the reset gate effectively zeros out the past hidden state, meaning the candidate activation relies heavily on the new input _x _t_ ​. If _r _t_ ​ is close to 1, more of the past hidden state is considered, blending it with the new input to form _h _t_ ​.\nThe candidate activation _h _t_ ​ then contributes to the final hidden state _h _t_ ​ for the current time step. The update gate _z _t_ ​ (refer to section 3.1) determines how much of this candidate activation is mixed with the previous hidden state:\n![Updated Hidden State Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1lNgueKe1bKgS1knKsQRW2A.png)\nUpdated Hidden State Formula - Image by Author\nWhere:\n  * **_z** _**t**_**​** is the update gate value at time step _t_ , calculated using the sigmoid function. It decides how much of the previous hidden state _h _t_ −1​ and the candidate activation _h _t_ ​ should contribute to the new hidden state.\n  * **_h** _**t**_**−1​** is the hidden state from the previous time step. It contains information about what the network has learned from the past inputs.\n  * **_h** _**t**_**​(tilde)** is the candidate activation for the current time step, calculated using the reset gate and the current input. It represents the new information that could potentially update the hidden state.\n  * **(1−_z** _**t**_**​)∗_h** _**t**_**−1** ​ controls how much of the previous hidden state is retained. When _z _t_ ​ is close to 0, the previous hidden state _h _t_ −1​ has a larger influence.\n  * **_z** _**t**_**​∗_h** _**t**_ ​(tilde) determines how much of the candidate activation _h _t_ ​ is added to the new hidden state. When _z _t_ ​ is close to 1, the candidate activation _h _t_ ​ has a larger influence.\n\n\n​By adjusting _z _t_ ​, the GRU can smoothly transition between preserving past information and incorporating new information. This balance helps the GRU retain important historical context while adapting to new inputs, making it effective for handling sequential data with varying dependencies.\n### 3.4: Step-by-Step Example\nTo understand how GRUs work in practice, let's walk through a simple example step-by-step. We'll use a small set of made-up numbers to illustrate the process.\nLet's assume:\n  * An input vector at time _t_ , _x _t_ ​= [0.5]\n  * A previous hidden state, _h _t_ −1​ = [0.3]\n  * Weights for the update gate, reset gate, and candidate activation are all 1 for simplicity.\n  * Biases are 0 for simplicity.\n\n\n**Step 1: Calculate the Update Gate** First, we calculate the update gate _z _t_ ​:\n![Update Gate Calculation - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/176EqAHRgeBhbMjoczqkGYQ.png)\nUpdate Gate Calculation - Image by Author\nUsing the sigmoid function:\n![Sigmoid Function applied to Update Gate - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/12SfYCGQF0aigOmkwzkt9zg.png)\nSigmoid Function applied to Update Gate - Image by Author\n**Step 2: Calculate the Reset Gate** Next, we calculate the reset gate _r _t_ ​:\n![Reset Gate Calculation - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/10RK3UvEwsv_tyz8JPvESbg.png)\nReset Gate Calculation - Image by Author\nUsing the sigmoid function again:\n![Sigmoid Function applied to Reset Gate - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1IqdrIBdv9WzUQGEdRKKN7A.png)\nSigmoid Function applied to Reset Gate - Image by Author\n**Step 3: Calculate the Candidate Activation** Now, we calculate the candidate activation _h _t_ ​:\n![Candidate Activation Calculation - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1MWUiswwc-xMyR2QaH2lorQ.png)\nCandidate Activation Calculation - Image by Author\nUsing the tanh function:\n![tanh Function applied to Candidate Activation - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1RAc0wufFfYmOVYVV9KHFZA.png)\ntanh Function applied to Candidate Activation - Image by Author\n**Step 4: Calculate the Final Hidden State** Finally, we calculate the new hidden state _h _t_ ​:\n![Final Hidden State Calculation - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1iG58crpdZjJZ0M722reTGg.png)\nFinal Hidden State Calculation - Image by Author\n## 4: Building GRU From Scratch in Python\nThis section will guide you through implementing a Gated Recurrent Unit (GRU) network from scratch using Python. We will train the model over a flight passenger dataset. We will import the dataset from the `seaborn` library, which is [free for commercial use](https://github.com/mwaskom/seaborn/blob/master/LICENSE.md).\nYou can find all the code we'll cover here:\n> [**models-from-scratch-python/Gated Recurrent Unit/demo.ipynb at main ·...**](https://github.com/cristianleoo/models-from-scratch-python/blob/main/Gated%20Recurrent%20Unit/demo.ipynb)\n### 4.1: Imports and Custom Classes\nFirst, we need to import the necessary libraries and custom classes that will help us build and train our GRU model. We'll use `numpy` for numerical operations, `pandas` for dataset manipulation, and `seaborn` for plotting. Additionally, we'll use custom classes for weight initialization and early stopping.\npython\n\n```\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\n# Custom classes (built from scratch)\nfrom src.model import WeightInitializer\nfrom src.trainer import EarlyStopping\n```\n\nThese custom classes (`WeightInitializer` and `EarlyStopping`) were created in a previous article. The `WeightInitializer` allows us to choose between several weight initialization techniques, and `EarlyStopping` stops the training process when there is no significant improvement in performance, preventing overfitting.\nYou can find the code for `WeightInitializer` and `EarlyStopping` in the article about LSTM:\n> [**The Math Behind LSTM**](https://towardsdatascience.com/the-math-behind-lstm-9069b835289d)\n### 4.2: GRU Class\nHere, we define the `GRU` class, which includes initialization, forward propagation, and backward propagation methods.\n**Initialization** The `__init__` method initializes the GRU with a specified _input size_ , _hidden size_ , and _output size_. It also initializes weights and biases using a custom weight initializer.\npython\n\n```\nclass GRU:\n    def __init__(self, input_size, hidden_size, output_size, init_method='xavier'):\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.output_size = output_size\n        self.weight_initializer = WeightInitializer(method=init_method)\n\n        # Initialize weights\n        self.wz = self.weight_initializer.initialize((hidden_size, hidden_size + input_size))\n        self.wr = self.weight_initializer.initialize((hidden_size, hidden_size + input_size))\n        self.wh = self.weight_initializer.initialize((hidden_size, hidden_size + input_size))\n\n        # Initialize biases\n        self.bz = np.zeros((hidden_size, 1))\n        self.br = np.zeros((hidden_size, 1))\n        self.bh = np.zeros((hidden_size, 1))\n\n        # Initialize output layer weights and biases\n        self.why = self.weight_initializer.initialize((output_size, hidden_size))\n        self.by = np.zeros((output_size, 1))\n```\n\nFirst, the initialization stores the sizes of the input, hidden, and output layers as class attributes. It also creates an instance of a WeightInitializer, using the specified method to initialize the weights, in this case, Xavier initialization.\nNext, it initializes the weights for the update gate (`wz`), reset gate (`wr`), and candidate hidden state (`wh`). Each of these weights is a matrix that combines the hidden state and input dimensions.\nAfter that, it initializes the biases for the update gate (`bz`), reset gate (`br`), and candidate hidden state (`bh`). These biases are vectors filled with zeros, one for each hidden unit.\nFinally, it initializes the weights and biases for the output layer. The weights (`why`) are a matrix connecting the hidden state to the output, and the biases (`by`) are a vector filled with zeros, one for each output unit.\n**Activation Functions** Next, we define static methods for the sigmoid function and its derivative, and the derivative of the tanh function.\nless\n\n```\n    @staticmethod\n    def sigmoid(z):\n        return 1 / (1 + np.exp(-z))\n\n    @staticmethod\n    def dsigmoid(y):\n        return y * (1 - y)\n\n    @staticmethod\n    def dtanh(y):\n        return 1 - y * y\n```\n\n**Forward Propagation** The `forward` method processes the input through the GRU cell for each time step, calculating the reset and update gates, candidate activation, and the new hidden state.\npython\n\n```\n    def forward(self, x):\n        caches = []\n        h_prev = np.zeros((self.hidden_size, 1))\n        h = h_prev\n\n        for t in range(x.shape[0]):\n            x_t = x[t].reshape(-1, 1)\n            combined = np.vstack((h_prev, x_t))\n\n            z = self.sigmoid(np.dot(self.wz, combined) + self.bz)\n            r = self.sigmoid(np.dot(self.wr, combined) + self.br)\n            combined_r = np.vstack((r * h_prev, x_t))\n            h_ = np.tanh(np.dot(self.wh, combined_r) + self.bh)\n\n            h = (1 - z) * h_prev + z * h_\n\n            cache = (h_prev, z, r, h_, x_t, combined, combined_r, h)\n            caches.append(cache)\n\n            h_prev = h\n\n        y = np.dot(self.why, h) + self.by\n        return y, caches\n```\n\nFirst, it initializes an empty list called `caches` to store intermediate values needed for backpropagation later. It also initializes the previous hidden state `h_prev` as a column vector of zeros with a size equal to the hidden layer size.\npython\n\n```\ncaches = []\nh_prev = np.zeros((self.hidden_size, 1))\nh = h_prev\n```\n\nThen, for each time step in the input sequence, it extracts the current input `x_t` and reshapes it into a column vector. It concatenates this input with the previous hidden state to form a combined input.\npython\n\n```\nfor t in range(x.shape[0]):\n    x_t = x[t].reshape(-1, 1)\n    combined = np.vstack((h_prev, x_t))\n```\n\nNext, it calculates the update gate `z` by applying the sigmoid function to the dot product of the update gate weights and the combined input, plus the update gate bias. Similarly, it calculates the reset gate `r` by applying the sigmoid function to the dot product of the reset gate weights and the combined input, plus the reset gate bias.\npython\n\n```\nz = self.sigmoid(np.dot(self.wz, combined) + self.bz)\nr = self.sigmoid(np.dot(self.wr, combined) + self.br)\n```\n\nThe reset gate `r` is then used to modify the previous hidden state, which is concatenated with the current input to form `combined_r`. This combined input is then used to calculate the candidate hidden state `h_` by applying the tanh function to the dot product of the candidate hidden state weights and `combined_r`, plus the candidate hidden state bias.\nini\n\n```\ncombined_r = np.vstack((r * h_prev, x_t))\nh_ = np.tanh(np.dot(self.wh, combined_r) + self.bh)\n```\n\nThe new hidden state `h` is computed as a combination of the previous hidden state and the candidate hidden state, weighted by the update gate `z`.\nini\n\n```\nh = (1 - z) * h_prev + z * h_\n```\n\nAll these intermediate values (`h_prev`, `z`, `r`, `h_`, `x_t`, `combined`, `combined_r`, and `h`) are stored in a tuple called `cache`, which is appended to the `caches` list.\ncpp\n\n```\ncache = (h_prev, z, r, h_, x_t, combined, combined_r, h)\ncaches.append(cache)\n```\n\nThe loop updates the previous hidden state `h_prev` to the current hidden state `h` for the next time step.\npython\n\n```\nh_prev = h\n```\n\nAfter processing all time steps, the method calculates the final output `y` by applying the output layer weights and biases to the last hidden state.\npython\n\n```\ny = np.dot(self.why, h) + self.by\n```\n\nFinally, the method returns the output `y` and the list of caches `caches`.\npython\n\n```\nreturn y, caches\n```\n\n**Backward Propagation** The `backward` method calculates gradients using the caches from the forward pass and performs gradient clipping to prevent exploding gradients.\npython\n\n```\n    def backward(self, dy, caches, clip_value=1.0):\n        \"\"\"\n        Backward pass through the GRU network.\n\n        Parameters:\n        - dy: np.ndarray, gradient of the loss with respect to the output\n        - caches: list, caches from the forward pass\n        - clip_value: float, value to clip gradients to (default: 1.0)\n\n        Returns:\n        - tuple, gradients of the loss with respect to the parameters\n        \"\"\"\n        # Initialize gradients\n        dWz, dWr, dWh = [np.zeros_like(w) for w in (self.wz, self.wr, self.wh)]\n        dbz, dbr, dbh = [np.zeros_like(b) for b in (self.bz, self.br, self.bh)]\n        dWhy = np.zeros_like(self.why)\n        dby = np.zeros_like(self.by)\n\n        # Ensure dy is reshaped to match output size\n        dy = dy.reshape(self.output_size, -1)\n        dh_next = np.zeros((self.hidden_size, 1))  # shape must match hidden_size\n\n        # Backpropagate through time\n        for cache in reversed(caches):\n            h_prev, z, r, h_, x_t, combined, combined_r, h = cache\n\n            # Add gradient from next step to current output gradient\n            dh = np.dot(self.why.T, dy) + dh_next\n            dh_ = dh * z * self.dtanh(h_)\n            dz = dh * (h_ - h_prev) * self.dsigmoid(z)\n            dr = np.dot(self.wh[:, :self.hidden_size].T, dh_) * h_prev * self.dsigmoid(r)\n\n            # Update gradients with respect to weights and biases\n            dWz += np.dot(dz, combined.T)\n            dWr += np.dot(dr, combined.T)\n            dWh += np.dot(dh_, combined_r.T)\n\n            dbz += dz.sum(axis=1, keepdims=True)\n            dbr += dr.sum(axis=1, keepdims=True)\n            dbh += dh_.sum(axis=1, keepdims=True)\n\n            # Compute the gradient for the previous time step\n            dh_prev = dh * (1 - z) + np.dot(self.wz[:, :self.hidden_size].T, dz) + np.dot(self.wr[:, :self.hidden_size].T, dr)\n\n            dh_next = dh_prev\n\n        # Update output layer weights and biases\n        dWhy += np.dot(dy, h.T)\n        dby += dy\n\n        gradients = (dWz, dWr, dWh, dbz, dbr, dbh, dWhy, dby)\n\n        # Gradient clipping\n        for i in range(len(gradients)):\n            np.clip(gradients[i], -clip_value, clip_value, out=gradients[i])\n\n        return gradients\n```\n\nFirst, the method initializes gradients for the _update gate_ weights (`dWz`), reset gate weights (`dWr`), candidate hidden state weights (`dWh`), and their respective biases (`dbz`, `dbr`, `dbh`). It also initializes gradients for the output layer weights (`dWhy`) and biases (`dby`). All these gradients are initialized to arrays of zeros with the same shape as the corresponding weights and biases.\npython\n\n```\ndWz, dWr, dWh = [np.zeros_like(w) for w in (self.wz, self.wr, self.wh)]\ndbz, dbr, dbh = [np.zeros_like(b) for b in (self.bz, self.br, self.bh)]\ndWhy = np.zeros_like(self.why)\ndby = np.zeros_like(self.by)\n```\n\nNext, the method reshapes the gradient of the loss with respect to the output (`dy`) to ensure it matches the output size. It also initializes `dh_next`, the gradient of the loss with respect to the next hidden state, to a column vector of zeros with a size equal to the hidden layer size.\npython\n\n```\ndy = dy.reshape(self.output_size, -1)\ndh_next = np.zeros((self.hidden_size, 1))\n```\n\nThe method then iterates backward through the stored caches from the forward pass. For each cache, which contains intermediate values from a specific time step, it extracts the cached values: `h_prev`, `z`, `r`, `h_`, `x_t`, `combined`, `combined_r`, and `h`.\npython\n\n```\nfor cache in reversed(caches):\n    h_prev, z, r, h_, x_t, combined, combined_r, h = cache\n```\n\nIt calculates the gradient of the loss with respect to the hidden state (`dh`) by adding the gradient from the next time step (`dh_next`) to the dot product of the output layer weights transpose and `dy`. It then computes the gradients of the candidate hidden state (`dh_`), update gate (`dz`), and reset gate (`dr`).\npython\n\n```\n# Add gradient from next step to current output gradient\ndh = np.dot(self.why.T, dy) + dh_next\ndh_ = dh * z * self.dtanh(h_)\ndz = dh * (h_ - h_prev) * self.dsigmoid(z)\ndr = np.dot(self.wh[:, :self.hidden_size].T, dh_) * h_prev * self.dsigmoid(r)\n```\n\nIn this step, we update the gradients of the weights and biases. Specifically, it computes the gradients with respect to the combined input (`dWz`, `dWr`) and the combined reset input (`dWh`). It also updates the gradients of the biases (`dbz`, `dbr`, `dbh`).\npython\n\n```\n# Update gradients with respect to weights and biases\ndWz += np.dot(dz, combined.T)\ndWr += np.dot(dr, combined.T)\ndWh += np.dot(dh_, combined_r.T)\n\ndbz += dz.sum(axis=1, keepdims=True)\ndbr += dr.sum(axis=1, keepdims=True)\ndbh += dh_.sum(axis=1, keepdims=True)\n```\n\nAfter that, it calculates the gradient of the loss with respect to the previous hidden state (`dh_prev`) to propagate the gradient back to the previous time step. The gradient `dh_prev` is a combination of the gradient of the loss with respect to the current hidden state and the contributions from the update gate and reset gate weights.\npython\n\n```\n# Compute the gradient for the previous time step\ndh_prev = dh * (1 - z) + np.dot(self.wz[:, :self.hidden_size].T, dz) + np.dot(self.wr[:, :self.hidden_size].T, dr)\n```\n\nThis process continues for each time step, updating `dh_next` with `dh_prev` for the next iteration.\npython\n\n```\ndh_next = dh_prev\n```\n\nOnce the backpropagation through time is complete, the method updates the gradients of the output layer weights (`dWhy`) and biases (`dby`) using the gradient of the loss with respect to the output.\npython\n\n```\n# Update output layer weights and biases\ndWhy += np.dot(dy, h.T)\ndby += dy\n```\n\nFinally, it clips the gradients to a specified value (`clip_value`) to prevent exploding gradients, ensuring that all gradient values are within the range [-clip_value, clip_value]. The method returns the computed gradients for all parameters as a tuple.\npython\n\n```\ngradients = (dWz, dWr, dWh, dbz, dbr, dbh, dWhy, dby)\n\n# Gradient clipping\nfor i in range(len(gradients)):\n    np.clip(gradients[i], -clip_value, clip_value, out=gradients[i])\n\nreturn gradients\n```\n\n**Updating Parameters** The `update_params` method applies the gradients to update the model parameters.\npython\n\n```\n    def update_params(self, grads, learning_rate):\n        dWz, dWr, dWh, dbz, dbr, dbh, dWhy, dby = grads\n\n        self.wz -= learning_rate * dWz\n        self.wr -= learning_rate * dWr\n        self.wh -= learning_rate * dWh\n\n        self.bz -= learning_rate * dbz\n        self.br -= learning_rate * dbr\n        self.bh -= learning_rate * dbh\n\n        self.why -= learning_rate * dWhy\n        self.by -= learning_rate * dby\n```\n\nFirst, the method unpacks the `grads` tuple into individual gradient variables: `dWz`, `dWr`, `dWh`, `dbz`, `dbr`, `dbh`, `dWhy`, and `dby`. These represent the gradients of the loss with respect to the update gate weights, reset gate weights, candidate hidden state weights, their corresponding biases, and the output layer weights and biases, respectively.\nThen, for each parameter, the method performs a gradient descent step. Specifically, it subtracts the product of the gradient and the learning rate from the current value of the parameter. This operation is performed for the update gate weights (`wz`), reset gate weights (`wr`), candidate hidden state weights (`wh`), their biases (`bz`, `br`, `bh`), and the output layer weights (`why`) and biases (`by`).\nTherefore, this method adjusts the parameters of the GRU network in the direction that reduces the loss. The learning rate controls how big each adjustment step is, ensuring that the parameters are updated incrementally based on the gradients calculated during backpropagation. This process is repeated for each parameter, resulting in an updated set of weights and biases for the network.\n### 4.3: GRUTrainer Class\nThe `GRUTrainer` class handles the training process, including forward and backward passes, and updates the model parameters.\n**Initialization** The `__init__` method initializes the trainer with the GRU model, learning rate, and early stopping criteria.\npython\n\n```\nclass GRUTrainer:\n    def __init__(self, model, learning_rate=0.01, patience=7, verbose=True, delta=0):\n        self.model = model\n        self.learning_rate = learning_rate\n        self.train_losses = []\n        self.val_losses = []\n        self.early_stopping = EarlyStopping(patience, verbose, delta)\n```\n\nThis class takes four parameters: `model`, which is the GRU model to be trained; `learning_rate`, which specifies the step size for updating the model's parameters during training (default is 0.01); `patience`, which defines how many epochs to wait for an improvement in validation loss before stopping the training early (default is 7); and `verbose`, a boolean that controls whether to print training information (default is True). It also accepts `delta`, which is the minimum change in validation loss to qualify as an improvement (default is 0).\nInside the `__init__` method, it sets these parameters as attributes of the `GRUTrainer` instance. It initializes an empty list `train_losses` to keep track of the training loss for each epoch and another empty list `val_losses` to store the validation loss for each epoch.\nFinally, it initializes an instance of `EarlyStopping` with the given `patience`, `verbose`, and `delta` parameters. This instance will be used to monitor the validation loss and determine when to stop training if the loss does not improve within the specified _patience_ period.\n**Training Method** The `train` method trains the GRU model on the training data and optionally validates it on a separate validation set.\npython\n\n```\n    def train(self, X_train, y_train, X_val=None, y_val=None, epochs=10, batch_size=1, clip_value=1.0):\n        for epoch in range(epochs):\n            epoch_losses = []\n            for i in range(0, len(X_train), batch_size):\n                batch_X = X_train[i:i + batch_size]\n                batch_y = y_train[i:i + batch_size]\n                losses = []\n\n                for x, y_true in zip(batch_X, batch_y):\n                    y_pred, caches = self.model.forward(x)\n                    loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))\n                    losses.append(loss)\n\n                    dy = y_pred - y_true.reshape(-1, 1)\n                    grads = self.model.backward(dy, caches, clip_value=clip_value)\n                    self.model.update_params(grads, self.learning_rate)\n\n                batch_loss = np.mean(losses)\n                epoch_losses.append(batch_loss)\n\n            avg_epoch_loss = np.mean(epoch_losses)\n            self.train_losses.append(avg_epoch_loss)\n\n            if X_val is not None and y_val is not None:\n                val_loss = self.validate(X_val, y_val)\n                self.val_losses.append(val_loss)\n\n                if epoch % 10 == 0:\n                    print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}, Val Loss: {val_loss:.5f}')\n\n                self.early_stopping(val_loss)\n                if self.early_stopping.early_stop:\n                    print(\"Early stopping\")\n                    break\n            else:\n                print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}')\n```\n\nFirst, the method iterates through the number of epochs specified. For each epoch, it initializes an empty list `epoch_losses` to store the loss for each batch.\npython\n\n```\nfor epoch in range(epochs):\n    epoch_losses = []\n```\n\nWithin each epoch, it iterates over the training data in mini-batches of size `batch_size`. For each batch, it extracts the current batch of inputs (`batch_X`) and their corresponding labels (`batch_y`). It initializes an empty list `losses` to store the loss for each sample in the batch.\npython\n\n```\nfor i in range(0, len(X_train), batch_size):\n    batch_X = X_train[i:i + batch_size]\n    batch_y = y_train[i:i + batch_size]\n    losses = []\n```\n\nFor each sample in the current batch, it performs the forward pass by calling the `forward` method of the GRU model, which returns the predicted output (`y_pred`) and the caches containing intermediate values needed for backpropagation. It then computes the loss between the predicted output and the true label (`y_true`), reshaped to match the prediction's shape, and appends this loss to the `losses` list.\npython\n\n```\nfor x, y_true in zip(batch_X, batch_y):\n    y_pred, caches = self.model.forward(x)\n    loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))\n    losses.append(loss)\n```\n\nNext, it computes the gradient of the loss with respect to the output (`dy`) and performs backpropagation by calling the `backward` method of the GRU model, which returns the gradients for the model's parameters. It then updates the model's parameters using these gradients by calling the `update_params` method with the computed gradients and the learning rate.\npython\n\n```\ndy = y_pred - y_true.reshape(-1, 1)\ngrads = self.model.backward(dy, caches, clip_value=clip_value)\nself.model.update_params(grads, self.learning_rate)\n```\n\nAfter processing all samples in the current batch, it calculates the mean loss for the batch (`batch_loss`) and appends it to the `epoch_losses` list.\ngo\n\n```\nbatch_loss = np.mean(losses)\nepoch_losses.append(batch_loss)\n```\n\nOnce all batches are processed for the current epoch, it calculates the average loss for the epoch (`avg_epoch_loss`) and appends it to the `train_losses` list.\npython\n\n```\navg_epoch_loss = np.mean(epoch_losses)\nself.train_losses.append(avg_epoch_loss)\n```\n\nIf validation data is provided (`X_val`, `y_val`), the method validates the model by calling the `validate` method, which returns the validation loss (`val_loss`). This validation loss is appended to the `val_losses` list.\npython\n\n```\nif X_val is not None and y_val is not None:\n    val_loss = self.validate(X_val, y_val)\n    self.val_losses.append(val_loss)\n```\n\nEvery 10 epochs, the method prints the current epoch number, the average training loss, and the validation loss if available.\npython\n\n```\nif epoch % 10 == 0:\n    print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}, Val Loss: {val_loss:.5f}')\n\n    self.early_stopping(val_loss)\n```\n\nThe method then checks for early stopping by calling the `early_stopping` method with the current validation loss. If early stopping is triggered, indicated by the `early_stop` attribute of `early_stopping` being `True`, it prints a message and breaks out of the training loop.\nlua\n\n```\nif self.early_stopping.early_stop:\n    print(\"Early stopping\")\n    break\n```\n\nIf validation data is not provided, it simply prints the current epoch number and the average training loss for each epoch.\npython\n\n```\nelse:\n    print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.5f}')\n```\n\n**Compute Loss and Validate Methods** The `compute_loss` method calculates the mean squared error loss and the `validate` method evaluates the model on validation data.\npython\n\n```\ndef compute_loss(self, y_pred, y_true):\n    return np.mean((y_pred - y_true) ** 2)\n```\n\nThe `compute_loss` method in the `GRUTrainer` class calculates the mean squared error loss between the predicted output (`y_pred`) and the true label (`y_true`). This method takes two arguments: `y_pred`, the predictions made by the model, and `y_true`, the actual labels. It computes the element-wise difference between `y_pred` and `y_true`, squares these differences, and then takes the mean of these squared differences to get the mean squared error loss.\npython\n\n```\ndef validate(self, X_val, y_val):\n    val_losses = []\n    for x, y_true in zip(X_val, y_val):\n        y_pred, _ = self.model.forward(x)\n        loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))\n        val_losses.append(loss)\n    return np.mean(val_losses)\n```\n\nThe `validate` method evaluates the model on a separate validation dataset to check its performance. This method takes two arguments: `X_val`, the validation inputs, and `y_val`, the validation labels.\nFirst, it initializes an empty list `val_losses` to store the loss for each validation sample. It then iterates over each sample in the validation dataset, extracting the input `x` and the corresponding true label `y_true`.\nFor each sample, it performs the forward pass by calling the `forward` method of the GRU model, which returns the predicted output (`y_pred`). It then computes the loss between the predicted output and the true label, reshaped to match the prediction's shape, by calling the `compute_loss` method. This loss is appended to the `val_losses` list.\nAfter processing all validation samples, it calculates the mean of the collected losses (`val_losses`) and returns this mean value as the validation loss. This provides an overall measure of the model's performance on the validation dataset.\n### 4.4: TimeSeriesDataset Class\nThe `TimeSeriesDataset` class is designed to handle time series data, specifically for loading, preprocessing, and creating datasets suitable for training and testing a model. It takes two optional parameters: `look_back`, which determines the number of previous time steps to use as input features (default is 1), and `train_size`, which specifies the proportion of data to be used for training (default is 0.67).\npython\n\n```\nclass TimeSeriesDataset:\n    def __init__(self, look_back=1, train_size=0.67):\n        self.look_back = look_back\n        self.train_size = train_size\n\n    def load_data(self):\n        df = sns.load_dataset('flights')\n        df['Date']  = pd.to_datetime(df['year'].astype(str) + '-' + df['month'].astype(str) + '-01')\n        df = df.set_index('Date')\n        df = df.drop(['year', 'month'], axis=1)\n        df = df.values.astype(float)\n        df = self.MinMaxScaler(df)\n        train_size = int(len(df) * self.train_size)\n        train, test = df[0:train_size,:], df[train_size:len(df),:]\n        return train, test\n\n    def MinMaxScaler(self, data):\n        numerator = data - np.min(data, 0)\n        denominator = np.max(data, 0) - np.min(data, 0)\n        return numerator / (denominator + 1e-7)\n\n    def create_dataset(self, dataset):\n        dataX, dataY = []\n        for i in range(len(dataset)-self.look_back):\n            a = dataset[i:(i + self.look_back), 0]\n            dataX.append(a)\n            dataY.append(dataset[i + self.look_back, 0])\n        return np.array(dataX), np.array(dataY)\n\n    def get_train_test(self):\n        train, test = self.load_data()\n        trainX, trainY = self.create_dataset(train)\n        testX, testY = self.create_dataset(test)\n        return trainX, trainY, testX, testY\n```\n\nThe `load_data` method loads a dataset of flight data using Seaborn's `load_dataset` function. It converts the 'year' and 'month' columns into a datetime format and sets this `datetime` as the index, it then drops the original 'year' and 'month' columns, leaving only the 'passengers' column. The method converts this column to a float type and scales the values between 0 and 1 using the `MinMaxScaler` method. It then splits the scaled data into training and testing sets based on the `train_size` attribute. The training set consists of the first `train_size` proportion of the data, and the testing set consists of the remaining data. The method returns these two sets.\npython\n\n```\ndef load_data(self):\n    df = sns.load_dataset('flights')\n    df['Date']  = pd.to_datetime(df['year'].astype(str) + '-' + df['month'].astype(str) + '-01')\n    df = df.set_index('Date')\n    df = df.drop(['year', 'month'], axis=1)\n    df = df.values.astype(float)\n    df = self.MinMaxScaler(df)\n    train_size = int(len(df) * self.train_size)\n    train, test = df[0:train_size,:], df[train_size:len(df),:]\n    return train, test\n```\n\nThe `MinMaxScaler` method takes in the data and normalizes it. It subtracts the minimum value of each feature from the data and then divides it by the range (maximum value minus minimum value) of each feature. This ensures that the data values are scaled between 0 and 1.\npython\n\n```\ndef MinMaxScaler(self, data):\n    numerator = data - np.min(data, 0)\n    denominator = np.max(data, 0) - np.min(data, 0)\n    return numerator / (denominator + 1e-7)\n```\n\nThe `create_dataset` method creates input-output pairs for the model. It takes the dataset and initializes two empty lists, `dataX` and `dataY`. For each time step in the dataset (up to `len(dataset) - self.look_back`), it creates a sequence of `self.look_back` previous time steps as the input (`dataX`) and the next time step as the output (`dataY`). These pairs are appended to the respective lists. The method returns these lists as numpy arrays.\npython\n\n```\ndef create_dataset(self, dataset):\n    dataX, dataY = []\n    for i in range(len(dataset)-self.look_back):\n        a = dataset[i:(i + self.look_back), 0]\n        dataX.append(a)\n        dataY.append(dataset[i + self.look_back, 0])\n    return np.array(dataX), np.array(dataY)\n```\n\nThe `get_train_test` method combines the previous methods to produce the final training and testing datasets. It first calls `load_data` to obtain the training and testing sets. Then, it calls `create_dataset` on each set to generate the input-output pairs for both training and testing. Finally, it returns the training inputs (`trainX`), training outputs (`trainY`), testing inputs (`testX`), and testing outputs (`testY`). This method provides a complete pipeline for preparing the data for training and testing a model.\npython\n\n```\ndef get_train_test(self):\n    train, test = self.load_data()\n    trainX, trainY = self.create_dataset(train)\n    testX, testY = self.create_dataset(test)\n    return trainX, trainY, testX, testY\n```\n\n### 4.5: Training the GRU Model\nFirst, we instantiate the `TimeSeriesDataset` class with a `train_size` of 0.7, meaning 70% of the data will be used for training and a `look_back` of 1, meaning each input feature will consist of one previous time step. We call the `get_train_test` method of the dataset instance to obtain the training and testing datasets. The method returns four arrays: `trainX`, `trainY`, `testX`, and `testY`.\npython\n\n```\n# Instantiate the dataset\ndataset = TimeSeriesDataset(train_size=0.7, look_back=1)\ntrainX, trainY, testX, testY = dataset.get_train_test()\n```\n\nNext, we reshape `trainX` and `testX` to be in the format [samples, time steps, features]. This is done using NumPy's `reshape` function. For `trainX`, the new shape is (number of samples, number of time steps, 1), where the last dimension represents the single feature (passenger count). The same reshaping is applied to `testX`.\npython\n\n```\n# Reshape input to be [samples, time steps, features]\ntrainX = np.reshape(trainX, (trainX.shape[0], trainX.shape[1], 1))\ntestX = np.reshape(testX, (testX.shape[0], testX.shape[1], 1))\n```\n\nWe then instantiate the `GRU` class with an `input_size` of 1 (since each input has one feature), a `hidden_size` of 256 (the number of GRU units in the hidden layer), and an `output_size` of 1 (since we want to predict one value, the passenger count for the next time step).\npython\n\n```\ngru = GRU(input_size=1, hidden_size=256, output_size=1)\n```\n\nNext, we instantiate the `GRUTrainer` class, passing in the GRU model instance. We set the `learning_rate` to 0.001, `patience` to 50 (the number of epochs to wait for an improvement in validation loss before stopping early), `verbose` to True (so training information will be printed), and `delta` to 0.001 (the minimum change in validation loss to qualify as an improvement).\npython\n\n```\ntrainer = GRUTrainer(gru, learning_rate=1e-3, patience=50, verbose=True, delta=0.001)\n```\n\nFinally, we call the `train` method of the `GRUTrainer` instance to train the GRU model. We pass in the reshaped training and testing datasets (`trainX`, `trainY`, `testX`, `testY`), set the number of training epochs to 1000, the batch size to 32, and the gradient clipping value to 1.0. The training process will iteratively update the model's parameters to minimize the loss of the training data while monitoring the validation loss for early stopping.\npython\n\n```\ntrainer.train(trainX, trainY, testX, testY, epochs=1000, batch_size=32)\n```\n\n## 5: Conclusion\nIn this article, we explored Gated Recurrent Units (GRUs) and how they work. We started by comparing GRUs to LSTMs and Vanilla RNNs, then broke down the math behind key components like the update gate, reset gate, and candidate activation. We also provided a step-by-step guide to implementing a GRU from scratch in Python, with clear explanations of the code. Understanding and building GRUs helps us handle sequential data effectively, making them useful for tasks like time series prediction and natural language processing.\n## References\n  1. Cho, Kyunghyun, et al. \"Learning phrase representations using RNN encoder-decoder for statistical machine translation.\" [_arXiv preprint arXiv:1406.1078_](https://arxiv.org/abs/1406.1078) (2014).\n  2. Towards Data Science. \"The Math Behind Recurrent Neural Networks.\" _Towards Data Science_ (2020). [Link to Article](https://towardsdatascience.com/the-math-behind-recurrent-neural-networks-2de4e0098ab8)\n  3. Towards Data Science. \"The Math Behind LSTM.\" _Towards Data Science_ (2020). [Link to Article](https://towardsdatascience.com/the-math-behind-lstm-9069b835289d)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657505387_wlyc0o.jpeg)\nWritten By\n## [Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[See all from Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Mathematics](https://towardsdatascience.com/tag/mathematics/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F&title=The%20Math%20Behind%20Gated%20Recurrent%20Units)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F&title=The%20Math%20Behind%20Gated%20Recurrent%20Units)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-gated-recurrent-units-854d88aded65%2F&text=The%20Math%20Behind%20Gated%20Recurrent%20Units)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/exec-b645139d-72cc-456b-9ad9-a810bca8e5e0.jpg)](https://towardsdatascience.com/running-sql-concurrently-across-three-remote-duckdb-servers-with-quack/)\n## [Running SQL Concurrently Across Three Remote DuckDB Servers with Quack](https://towardsdatascience.com/running-sql-concurrently-across-three-remote-duckdb-servers-with-quack/)\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\nA small experiment in remote SQL execution\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nAugust 16, 202617 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0AhPhmEE5tB77pCjU.jpg)](https://towardsdatascience.com/object-oriented-programming-a-practical-introduction-part-2-f376bc25f839/)\n## [Object-Oriented Programming: A Practical Introduction (Part 2)](https://towardsdatascience.com/object-oriented-programming-a-practical-introduction-part-2-f376bc25f839/)\n[Programming](https://towardsdatascience.com/category/programming/)\nLet's dig in to the more formal details of OOP.\n[Mark Douthwaite](https://towardsdatascience.com/author/mark-l-douthwaite/)\nOctober 9, 202012 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1hIi13-FwQo3iCMU-PQFPUg-scaled.jpeg)](https://towardsdatascience.com/how-to-use-machine-learning-to-inform-design-decisions-and-make-predictions-838106acf639/)\n## [How to Use Machine Learning to Inform Design Decisions and Make Predictions](https://towardsdatascience.com/how-to-use-machine-learning-to-inform-design-decisions-and-make-predictions-838106acf639/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAn Introductory Guide and Use Case for Applied Data Science\n[Kurt Klingensmith](https://towardsdatascience.com/author/kurt.klingensmith/)\nAugust 8, 202413 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/05/0MzyvSmtqj8SZxpsm-scaled.jpg)](https://towardsdatascience.com/exploring-the-ml-tooling-landscape-part-1-of-3-53f8c39e6e4e/)\n## [Exploring the ML Tooling Landscape (Part 1 of 3)](https://towardsdatascience.com/exploring-the-ml-tooling-landscape-part-1-of-3-53f8c39e6e4e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nML Maturity in Industry\n[Tom Martin](https://towardsdatascience.com/author/tpgmartin/)\nMay 3, 202210 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/1etLHbeM000St-E3y3DVK9g.gif)](https://towardsdatascience.com/terrificeye-an-edge-computing-system-for-traffic-analytics-from-videos-216cde072b74/)\n## [TerrificEye: An Edge Computing System For Traffic Analytics From Videos](https://towardsdatascience.com/terrificeye-an-edge-computing-system-for-traffic-analytics-from-videos-216cde072b74/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nHow I built a real-time, standalone traffic monitoring and analytics system\n[J. Rafid Siddiqui, PhD](https://towardsdatascience.com/author/azad-wolf/)\nMarch 6, 20227 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/145D2GKyphL5G7fcCJL1BnA.png)](https://towardsdatascience.com/deep-learning-with-cifar-10-image-classification-64ab92110d79/)\n## [Deep Learning with CIFAR-10 Image Classification](https://towardsdatascience.com/deep-learning-with-cifar-10-image-classification-64ab92110d79/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nNeural Networks are the programmable patterns that helps to solve complex problems and bring the best achievable output. Deep Learning as...\n[Aarya Brahmane](https://towardsdatascience.com/author/aaryabrahmane/)\nOctober 26, 202010 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/1wuEUDAkSJw0qIIAOK4qoiw-scaled.jpeg)](https://towardsdatascience.com/learning-wolfram-resources-to-supercharge-your-machine-learning-projects-37d7472d5c02/)\n## [Learning Wolfram: Resources to Supercharge Your Machine Learning Projects](https://towardsdatascience.com/learning-wolfram-resources-to-supercharge-your-machine-learning-projects-37d7472d5c02/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nImprove Your Workflows and Productivity With Wolfram\n[Arnoud Buzing](https://towardsdatascience.com/author/arnoudbuzing/)\nSeptember 21, 20203 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1r_TJZMdbJdCdhCY_yONmGQ-scaled.jpeg)](https://towardsdatascience.com/to-index-or-not-to-index-8be32ad45cae/)\n## [To Index or Not to Index](https://towardsdatascience.com/to-index-or-not-to-index-8be32ad45cae/)\n[Programming](https://towardsdatascience.com/category/programming/)\nLeverage SQL indexing to speed up your queries. Learn when to index, when not to, and how indexing works under the hood.\n[Christopher Karg](https://towardsdatascience.com/author/christopher_karg/)\nNovember 8, 202419 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/industrial_buttons_32529354_v3_card.jpg)](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/)\n## [RAG Workflow and Loop Engineering: The Dispatcher That Decides When to Loop and When to Stop](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #13] - Putting the patterns together, and why this is what “agentic RAG” should look like\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 14, 202627 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "body_base_path": "attachments/3f95a2e3c48e/pages/",
  "body_markdown_path": "attachments/3f95a2e3c48e/pages/archived-body.md",
  "local_html": "attachments/3f95a2e3c48e/pages/archived-body.html",
  "body_source": "prior_capture"
}