{ "source_file": "0930", "source_platform": "Medium", "author": "Sascha Kirch", "title_listed": "Structured State Space Models Visually Explained", "keywords": "", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/", "publish_date": "2024-08-22", "corpus_stage": "", "notes": "", "id": "24596f2cc41a", "collected_at": "2026-10-02T00:48:09", "title": "Structured State Space Models Visually Explained | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Structured State Space Models Visually Explained\nPart 2 - Towards Mamba State Space Models for Images, Videos and Time Series\n[Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\nAugust 22, 202418 min read\n### [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://towardsdatascience.com/tagged/mamba-image-video-signal)\n![Image by Sascha Kirch.](../images/000.png)\nImage by [Sascha Kirch](https://medium.com/@SaschaKirch).\n> This is part 2 of my new multi-part series [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://medium.com/@SaschaKirch/list/mamba-state-space-models-for-images-videos-and-timeseries-861ae0ad08fb).\nState space models, known to many disciplines of engineering for decades, are now making their debut in deep learning. On our journey towards Mamba selective state space models and their recent achievements in research, understanding the state space model is crucial. And, as is often the case in engineering, it is the details that make theoretical concepts applicable in practice. In addition to state space models, we must also discuss how to apply them to sequence data, how to handle long-range dependencies, and how to efficiently train them by exploiting certain matrix structures.\nStructured state space models build the theoretical foundation for Mamba. However, their connection to system theory and advanced algebra might be one of the obstacles to adopt this new framework. So, let's break it down, ensure we understand the key concepts and visualize them to shed some light onto this new-old theory.\n> Even if you don't end up using State Space models, understanding some of the tricks like why we need to speed up matrix multiplication and how we can exploit certain structures of matrices for that will certainly level up your skills as an engineer or developer.\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n* * *\n## Outline\n 1. [State Space Models in a Nutshell](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#51c1) 2.1 [Continuous Time Representation of the State Space Model](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#d5c1) 2.2 [Discretization of the State Space Model](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#ec5f) 2.3 [The Recurrent and Convolutional Representations of the SSM](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#8e7d)\n 2. [Enabling Long-Range Relationships](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#644c) 3.1 [State Size Matters](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#28b2) 3.2 [The HiPPO Framework](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#4384)\n 3. [Impose Structure on State Matrices for Efficiency](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#15b4) 4.1 [What Are Structured Matrices and Why Do They Matter?](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#fc04) 4.2 [Putting It All Together - S4: Structured State Space Sequence Models](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#4432)\n 4. [Wrapping Up](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#52fe) 5.1 [Summary](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#e69c) 5.2 [Limitations of Structured SSMs](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#0284) 5.3 [Continue Reading Part 3](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#192b)\n\n\n* * *\n## 1. Where We Left Off in Part 1\n> [**Towards Mamba State Space Models for Images, Videos and Time Series**](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a)\nIn part 1 we reviewed the strengths and weaknesses of recurrent neuronal networks (RNNs) and Transformers to motivate why we need a new model architecture.\nWe said RNNs are fast at inference time but slow during training and the Transformer is fast to train but slowly inferred. What we want is to find a model that is fast for both, training and inference, while beeing competititve with the Transformer's performance.\n* * *\n## 2. State Space Models in a Nutshell\nMamba builds on the idea of using a state space model for deep learning by learning its various matrices. So, let's have a brief primer on state space models before exploring what the \"structured\" part is all about. A state space model can be defined in continuous time representation to deal with continuous signals or can be discretized to handle discrete sequences of data.\nIt is the discretized state space model we are mostly interested in, since like recurrent neuronal networks (RNNs) and Transformers, a discrete SSM processes sequences of data., e.g. text tokens or samples of an analog time signal.\n### 2.1 Continuous Time Representation of the State Space Model\nA continuous-time state space model describes the relationship between an input signal propagating through a stateful system and the resulting output signal.\n![Figure 1: High-level functionality of a state space model. Image by Sascha Kirch.](../images/001.gif)\nFigure 1: High-level functionality of a state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nIt is an idea borrowed from system theory. The output depends on the input and on the current state of the system, while the current state depends on the previous state and the input.\nThis relationship can be formulated efficiently by two simple equations, where _A_ , _B_ , _C_ and _D_ are matrices (which, as we'll see later, will be learned), _x(t)_ is the input signal, _y(t)_ the output signal and _h(t)_ and _h'(t)_ are the current and updated state respectively.\n![Equation. 1: State and output equations of the continuous time SSM.](../images/002.png)\nEquation. 1: State and output equations of the continuous time SSM.\nMatrix _D_ transforms the input _x(t)_ and maps it to the output _y(t)_ , which is typically included in the output equation of a classical SSM. However, when applying an SSM to deep learning, we remove this transformation and model it as a simple skip connection, thereby simplifying the SSM.\nWe can represent those equations in a block diagram that looks something like this:\n![Figure 2: Block diagram of a continuous time state space model. Image by Sascha Kirch.](../images/003.gif)\nFigure 2: Block diagram of a continuous time state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThis is the continuous-time representation of an SSM. There are two difficulties though:\n 1. finding an analytic solution for the state _h(t)_ of the model is challenging and\n 2. since we work on computers we usually work with discrete signals, not continuous ones.\n\n\n### 2.2 Discretization of the State Space Model\nSo, instead of mapping a function _x(t)_ to a function _y(t)_ , we are actually mapping a sequence _x[k]_ to a sequence _y[k]_. This implies we need a discretized state space model that can handle discrete signals.\n![Figure 3: High-level functionality of a discrete state space model. Image by Sascha Kirch.](../images/004.gif)\nFigure 3: High-level functionality of a discrete state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nTo do so, we discretize our matrices _A_ and _B_. This results in the discretized representation of our state space model, now including a discretized state _h[k]_.\n![Equation 2: State and output equations of the discretized SSM.](../images/005.png)\nEquation 2: State and output equations of the discretized SSM.\nTo discretize our state space model, we use yet another trick frequently used in system theory: We use a Zero-Order Hold (ZOH) module to turn our discrete signal _x[k]_ into a continuous time signal _x(t)_ which is then fed into the continuous SSM. The SSM will generate a continuous output _y(t)_ , which we then sample to obtain the discrete output sequence _y[k]_ :\n![Figure 4: Block diagram of a discretized state space model. Image by Sascha Kirch.](../images/006.png)\nFigure 4: Block diagram of a discretized state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nTo discretize the state space model, we must discretize matrices _A_ and _B_. Initially, most of the first deep learning based state space models like [LSSL](https://arxiv.org/abs/2110.13985) and [S4](https://arxiv.org/abs/2111.00396) used the bilinear discretization, while later ones like the [DSS](https://arxiv.org/abs/2203.14343) and Mamba use the ZOH discretization. For now, all we care about is that we need to discretize and that there might be more than one way to do so.\n![Equation 3: Zero Order Hold \\(ZOH\\) discretization vs. bilinear discretization.](../images/007.png)\nEquation 3: Zero Order Hold (ZOH) discretization vs. bilinear discretization.\n_A_ and _B_ are the SSM's continuous-time matrices, _I_ is the identity matrix and Δ is the step size representing the resolution of the input, which in the deep learning setting is learned.\n> Note that the discretization allows us to use the time-continuous formulation of the SSM and apply it to discrete signals used in a computer. This implies that we become resolution invariant since we learn the underlying time-signal. Check out the [S4ND](https://arxiv.org/abs/2210.06583) paper for some further reading on that.\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n### 2.3 The Recurrent and Convolutional Representations of the SSM\nHave you noticed the recurrency in the state equation because _h[k]_ depends on _h[k-1]_? In fact, the state space model is quite like an RNN in that manner:\n![Figure 5: Recurrent representation of a discretized state space model. Image by Sascha Kirch.](../images/008.gif)\nFigure 5: Recurrent representation of a discretized state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThe matrices _A_ , _B_ and _C_ are time-invariant. That means that each input and each state is transformed with the same parameters, regardless of the time step. Linear time invariant systems, or short LTI-systems, have a very neat property: they can be represented as a convolution!\n![Equation 4: Convolution kernel of the discretized state space model.](../images/009.png)\nEquation 4: Convolution kernel of the discretized state space model.\nThe kernel is constructed from the matrix _C_ and the discretized matrices _A_ and _B_. Note that the kernel has the same length _L_ as the input sequence. The kernel is convolved with the input _x_ to obtain the output _y_. To end up with the same sequence length at the output as at the input, the input is usually padded.\n![Figure 6: Example of the convolutional representation of a discretized state space model. Image by Sascha Kirch.](../images/010.gif)\nFigure 6: Example of the convolutional representation of a discretized state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThat's awesome because we know that convolutions are parallelizable and that GPUs and other HW-accelerators are optimized to calculate them fast.\nTo conclude this section, leveraging the properties of a **L** inear **T** ime **I** nvariant System, we ended up with 3 different ways to represent the SSM:\n![Figure 7: Different representations of state space models. Image by Sascha Kirch.](../images/011.png)\nFigure 7: Different representations of state space models. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nBoth the recurrent and the convolutional representations of the SSM are equal, hence we can decide whether we use one or the other.\n**During training** we would choose the convolutional representation because we have access to the entire sequence _x[k]_ and labels _y[k]_ ahead of time through our training data.\n**During inference** we would choose the recurrent representation because usually we only have access to samples from the past (e.g. autoregressive text generation). **** Note that the inference scales linearly with the sequence length since our next output only depends on the previous state and the current input.\nWe now have a model with efficient training and efficient inference. Awesome! 😎\n> It was the [LSSL (Linear State Space Layer) paper](https://arxiv.org/abs/2110.13985), that showed that we can train an SSM and that highlighted the connection between the different representations.\n> [**Get an email whenever Sascha Kirch publishes 🚀**](https://medium.com/@SaschaKirch/subscribe)[ _Get an email whenever Sascha Kirch publishes 🚀 Looking to learn more about deep learning or simply stay up to dat_e...medium.com](https://medium.com/@SaschaKirch/subscribe)\n## 3. Enabling Long-Range Relationships\nMatrix _A_ is responsible to create the next state from the previous state. The state of the system is a result of multiple updates considering past states and new inputs. It could be said that the state contains the information of the entire history of inputs. It is not hard to imagine that for very long sequences it might be a difficult task to remember all inputs accurately. So, how can we handle long-range dependencies?\n### 3.1 State-Size Matters\nLet's first talk about the state, its compression and the efficiency of inference.\n![Figure 8: Efficiency vs. performance of the RNN, SSM, Transformer and the desired model. Image by Sascha Kirch.](../images/012.png)\nFigure 8: Efficiency vs. performance of the RNN, SSM, Transformer and the desired model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nRNNs and Transformers can both theoretically deal with a never-ending sequence of data (sometimes people say: these models are unbound). **The RNN** does this by compressing the entire history into a single low-dimensional state. This is very efficient for inference, because now the next state only depends on the previous state and the current input. The performance of the RNN depends on how well the model learns to do this compression. Additionally, for long sequences and especially due to vanishing gradients, this is not an easy task to achieve, even for more advanced RNNs using gating mechanisms like GRUs and LSTMs. **The Transformer** on the other hand does not compress the history at all. It considers the entire history (and future if not masked like in autoregressive generation) through its attention mechanism. This allows the Transformer being trained in parallel, but at the same time memory and compute requirements explode with increasing sequence lengths during inference.\n**❓ So, again: how can we handle long-range dependencies with a highly compressed state, without the difficulties during training while achiving high performance?**\nOne Solution: we initialize our matrices using the HiPPO (**Hi** gh-order **P** olynomial **P** rojection **O** perators) framework.\n### 3.2 The HiPPO Framework\nThe main idea is to parametrize the matrices _A_ and _B_ from the state equation in such a way that we can reconstruct the history (i.e. the input signal from time _t=0_ to _t=now_) at any given point. Specifically, if the state is represented as some coefficients of certain orthogonal polynomials (e.g. the [Legendre Polynomial](https://en.wikipedia.org/wiki/Legendre_polynomials)), we can simply evaluate the polynomial to reconstruct the input signal up until the current time.\nFor example: We have an input function _x(t)_. At any given time step _ti_ , we want to find a reconstruction _g(ti)_ that approximates _x(t for t np.ndarray:\n A = np.zeros((N,N))\n for n in range(N):\n for k in range(N):\n if n > k:\n A[n,k] = (2*n+1)**0.5*(2*k+1)**0.5\n elif n == k:\n A[n,k] = (n+1)\n return A\n```\n\nThe LSSL mentioned earlier did show that one can train an SSM if initialized with the HiPPO matrix and it did show that its performance even surpasses those of the Transformer in the shown experiments. However, its expensive computation and memory requirements made the LSSL infeasible as a general sequence model. This hasn't been solved until the introduction of the S4 (Structured State Space Sequen _ce)_ model, which we will address in the next chapter.\n## 4. Impose Structure on State Matrices for Efficiency\nIt is finally time to talk about what the structure in the structured state space model is all about. We'll first try to gain some very basic intuition what structure is and why it is important, and then we will take a closer look onto the S4 model.\n### 4.1 What Are Structured Matrices and Why Do They Matter?\nLet's consider a standard square matrix of _NxN_ elements. We will focus on two aspects:\n 1. how much memory do we need to represent this matrix,\n 2. how fast can we multiply 2 matrices, where fast can be a result of several things, like being able to parallelize the computation, exploiting redundancies, and having less values to deal with in the first place.\n\n\n**How Much memory do we need?** Let's start with the memory requirements to store a matrix. We can have different kinds of matrices. Sometimes they are already in a certain form and sometimes we can bring them into a certain form leveraging rules from linear algebra. Let's have a look on some common matrix types:\n![Figure 10: Number of values to store for several common matrix types. Image by Sascha Kirch.](../images/016.png)\nFigure 10: Number of values to store for several common matrix types. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nWe see that in the worst case for a dense matrix, we need to store _N²_ values in memory. Not only that, but we must also load _N²_ values from the CPU to the GPU, where we then apply certain matrix operations.\nNote that there is a difference on how much values we need to store in our memory on CPU side (meaning in RAM or on disk) and how much memory we occupy in the GPU for computations. To make this clear, let's look at the low-rank factorization. Some matrices can be factorized into 2 low-rank matrices of _NxR_ , where _R_ is the rank of these matrices. We now only need to store _NR_ values for each of the two matrices, hence _2NR_ values. That also means we only need to load _2NR_ values instead of _N²_ from CPU to GPU. But if we want to make calculations with the original matrix, we first need to reconstruct the original matrix (some say materialize them) inside the GPU, meaning we still need _N²_ values in the cache of the GPU.\nOther representations like the diagonal or the sparse matrix on the other hand decrease both: the values stored on disk and the number of values needed to perform matrix operations on the GPU.\n**How many computations are required?** Now let's consider a standard matrix multiplication _C=AB_ , where _A_ and _B_ are either dense or diagonal matrices and _C_ is the result of the matrix multiplication:\n![Figure 11: Number of operations for matrix multiplication of different matrix types. Image by Sascha Kirch.](../images/body-14dba7d280cc67429abb.png)\nFigure 11: Number of operations for matrix multiplication of different matrix types. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nFor the **case where both are dense matrices** , using the standard algorithm we all know for matrix multiplication, we need _N³_ multiplications and _(N-1)N²_ additions.\nIn **case** _**A**_**is a dense matrix and** _**B**_**is a diagonal matrix** , this reduces to _N²_ multiplications and _0_ additions.\nIn **case both** _**A**_**and** _**B**_**are diagonal** , we only require _N_ multiplications which are the element-wise multiplications of the entries of the diagonal. While this is desirable in terms of hardware resources, note that a diagonal matrix might be less expressive as a dense matrix, because it has less values. Like everything, it is a trade-off.\n**Parallelization of Computations** If you take a closer look onto the dense matrix multiplication, you might notice that calculating a single row of the output _C_ , involves all values of _B_ , but only a single row of _A_. Naively, you could speed up the computation by calculating each row of _C_ in parallel on different cores of a GPU. This means each core would only need _N²_ multiplications and _(N-1)N_ additions instead of _N³_ multiplications and _(N-1)N²_ additions as in the single core case because we now have a vector-matrix product, not a matrix-matrix product.\n**Long Story Short** To conclude, and using the example of a square matrix of _NxN_ , by trying to exploit certain properties of matrices, we want to bring them into a structured form such that we need to store and load less than _N²_ values and that we require less than _N³_ multiplications and _(N-1)N²_ additions for matrix multiplications.\nOf course, there are many other matrix structures that we can exploit all with their own pros and cons. Throughout this series on Mamba state space models we will encounter many of those, and all serve the same purpose: **requiring less resources while keeping the best performance possible.**\n### 4.2 Putting It All Together - S4: Structured State Space Sequence Models\nS4 aims to make SSMs more efficient by enforcing a certain structure upon the involved matrices. By doing so, it can apply well studied rules of algebra to reduce the complexity of the involved computations and save memory, especially for the repeated computation of the kernel _K_ of the convolutional view.\n![Figure 12: S4 Structured State Sequence Model. Image by Sascha Kirch.](../images/body-ecf251c14df1c19c7929.png)\nFigure 12: S4 Structured State Sequence Model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nCompared to LSSL, S4 is has been shown to be 30x faster with 400x less memory usage while it also gained a boost in performance.\n> The S4 paper is involves a lot of advanced linear algebra I will mostly skip in this blog post. If you want to dive deeper into the math and also the code implementation I recommend you to check out [The Annotated S4](https://srush.github.io/annotated-s4/#part-2-implementing-s4).\nRecall that calculating the convolution kernel _K_ for the convolutional representation of the SSM, repeated matrix multiplications of the discretized square matrix _A_ are required. Remember that the kernel has the same length _L_ as the input sequence _x_ and that the power of the matrix _A_ is increased with each entry of the kernel.\n![Equation 6: Convolutional kernel of the discretized state space model.](../images/009.png)\nEquation 6: Convolutional kernel of the discretized state space model.\nTo be clear, a sequence of length _L=101_ would have matrices to the power of _100_! And _99_ and _98_ and... you get the point. Of course, we could reuse previous results, but still this is a lot to compute.\nThe main goal of S4 is to reduce the number of resources needed to calculate the kernel _K_ , by imposing a certain structure on matrix _A_.\nIdeally, matrix _A_ would be a diagonal matrix because as we saw earlier in Figure 11, multiplying two diagonal matrices result in another diagonal matrix requiring only _N_ multiplications.\nSince a HiPPO matrix cannot be diagonalized in a numerically stable manner (as proofed in the S4 paper), the next best thing is to decompose it into a normal plus low-rank (NPLR) form that can then further be simplified into a diagonal plus low-rank (DPLR) form by conjugation. Note that we want to hold on to the HiPPO matrix and do not change it to another diagonalizable matrix, because we need it for dealing with long-range dependencies.\n![Figure 13: Comparing different representations of matrix A and there resulting complexity to compute the convolution kernel K. Image by Sascha Kirch.](../images/body-6e21bf26d11b02783c0e.png)\nFigure 13: Comparing different representations of matrix A and there resulting complexity to compute the convolution kernel K. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nHaving matrix _A_ in DPLR form allows us to apply many complicated tricks. In summary, the kernel is calculated in the frequency domain and having the DPLR form allows rearranging the matrices into a Cauchy matrix, for which many hardware optimized algorithms exist.\nFurther, using the Woodburry identity allows transforming the calculation of multiple powers of _A_ into a single inverse. Finally, the inverse FFT (Fast Fourier Transform) is applied to obtain the final convolutional filter _K_.\nSo, the _SSM(A,B,C)_ previously a function of _A, B,_ and _C_ , is now a function _SSM(__**Λ-**__PP*,B,C)_ , where _**Λ,**__P, B_ and _C_ are learnable parameters of the S4.\n> If you are interested in the original implementation, here is the [dplr()](https://github.com/state-spaces/s4/blob/e757cef57d89e448c413de7325ed5601aceaac13/models/s4/s4.py#L453) function and the [nplr()](https://github.com/state-spaces/s4/blob/e757cef57d89e448c413de7325ed5601aceaac13/models/s4/s4.py#L374) function of the [S4 repository](https://github.com/state-spaces/s4).\n## 5. Wrapping Up\n### 5.1 Summary\n * State Space Models (SSMs) are sequence models.\n * SSMs can be described as recurrent model or as convolution model\n * They are inferred fast in their recurrent representation that scales linearly because they only depend on the current input and the previous state.\n * They are trained fast because of their convolutional representation that can be parallelized.\n * Since they are an LTI system, each input at any time step is treated equally, meaning transformed by the same matrix.\n * Structured state space sequence models aka. S4 use a certain structure on their matrices to make SSMs faster allowing them to be less restrictive with compute and memory requirements.\n\n\n### Thank You!\n> _🤝_ I'd like to t**hank you,** **t** he readers of my stories, for your constant support, clapping for my stories and, following me to never miss any of my latest articles. It is you who motivates me to continue writing, dive deep into complex topics, and become a better engineer/researcher by writing about them. You are awesome!\n### 5.2 Limitations of Structured SSMs\nThere is one big big issue though: SSMs cannot reason about different inputs at different time steps, since the matrices _A_ , _B_ and _C_ are time invariant meaning the same for all inputs.\n### 5.3 Continue Reading Part 3\n> [**Here Comes Mamba: The Selective State Space Model**](https://towardsdatascience.com/here-comes-mamba-the-selective-state-space-model-435e5d17a451)\nIn Part 3 we will discover how Mamba solves the issue of the structured state space model not differentiating between samples in the input sequence by adding selectivity to the model architecture. **❓ But what do we gain and at which cost?**\n## 6. Further Readings & Resources\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n### Blog Posts\n * [A Visual Guide to Mamba and State Space Models](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state)\n * [The Annotated S4](https://srush.github.io/annotated-s4/#part-2-implementing-s4)\n * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://hazyresearch.stanford.edu/blog/2020-12-05-hippo)\n\n\n### Papers:\n * [S4: Efficiently Modeling Long Sequences with Structured State Spaces](https://arxiv.org/abs/2111.00396) by A. Gu et.al., 31 Oct. 2021\n * [S5: Simplified State Space Layers for Sequence Modeling](https://arxiv.org/abs/2208.04933) by J. Smith et. al., 9 Aug. 2022\n * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://arxiv.org/abs/2008.07669) by A. Gu et.al., 17 Aug. 2020\n * [LSSL: Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://arxiv.org/abs/2110.13985) by A. Gu et. al., 26 Oct. 2021\n * [DSS: Diagonal State Spaces are as Effective as Structured State Spaces](https://arxiv.org/abs/2203.14343) by A. Gupta et. al., 27 Mar. 2022\n * [S4D: On the Parameterization and Initialization of Diagonal State Space Models](https://arxiv.org/abs/2206.11893) by A. Gu et. al., 23 Jun. 2022\n * [S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces](https://arxiv.org/abs/2210.06583) by E. Nguyen et. al., 12 Oct. 2020\n\n\n### Code\n * [S4 Code Repository](https://github.com/state-spaces/s4)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-2648fe230469ec3ba2b1.png)\nWritten By\n## [Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[See all from Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Linear Algebra](https://towardsdatascience.com/tag/linear-algebra/)[Mamba](https://towardsdatascience.com/tag/mamba/)[Mamba Image Video Signal](https://towardsdatascience.com/tag/mamba-image-video-signal/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fstructured-state-space-models-visually-explained-86cfe2757386%2F&title=Structured%20State%20Space%20Models%20Visually%20Explained)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fstructured-state-space-models-visually-explained-86cfe2757386%2F&title=Structured%20State%20Space%20Models%20Visually%20Explained)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fstructured-state-space-models-visually-explained-86cfe2757386%2F&text=Structured%20State%20Space%20Models%20Visually%20Explained)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/017.png)](https://towardsdatascience.com/using-pytorchvideo-for-efficient-video-understanding-24d3cd99bc3c/)\n## [Using PyTorchVideo for efficient video understanding](https://towardsdatascience.com/using-pytorchvideo-for-efficient-video-understanding-24d3cd99bc3c/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLearn how to easily visualize and evaluate activity classification models from the PyTorchVideo library The...\n[Eric Hofesmann](https://towardsdatascience.com/author/eric.hofesmann/)\nMay 19, 20218 min read\n * [![](../images/018.png)](https://towardsdatascience.com/writing-custom-cuda-kernels-with-triton-abf6b6ad1168/)\n## [Writing custom CUDA kernels with Triton](https://towardsdatascience.com/writing-custom-cuda-kernels-with-triton-abf6b6ad1168/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTriton is a language compiler to write highly optimized CUDA kernels. Learn the basics of GPU programming and how to use Triton.\n[Kushajveer Singh](https://towardsdatascience.com/author/kushaj/)\nSeptember 12, 20218 min read\n * [![](../images/019.jpeg)](https://towardsdatascience.com/generating-haiku-with-deep-learning-dbf5d18b4246/)\n## [Generating Haiku with Deep Learning](https://towardsdatascience.com/generating-haiku-with-deep-learning-dbf5d18b4246/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nUsing neural networks to generate haiku while enforcing the 5-7-5 syllable structure.\n[Jeremy Neiman](https://towardsdatascience.com/author/docmarionum1/)\nDecember 29, 20189 min read\n * [![](../images/020.jpg)](https://towardsdatascience.com/mobilenetv3-paper-walkthrough-the-tiny-giant-getting-even-smarter/)\n## [MobileNetV3 Paper Walkthrough: The Tiny Giant Getting Even Smarter](https://towardsdatascience.com/mobilenetv3-paper-walkthrough-the-tiny-giant-getting-even-smarter/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nMobileNetV3 with PyTorch — now featuring SE blocks and hard activation functions\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nNovember 2, 202527 min read\n * [![](../images/021.jpg)](https://towardsdatascience.com/neural-networks-are-like-human-bodies-852871fcbdb8/)\n## [Neural Networks are like human bodies](https://towardsdatascience.com/neural-networks-are-like-human-bodies-852871fcbdb8/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nOptimizers are like personal trainers\n[Pascal Janetzky](https://towardsdatascience.com/author/pascaljanetzky/)\nOctober 25, 20216 min read\n * [![](../images/022.jpg)](https://towardsdatascience.com/prefill-is-compute-bound-decode-is-memory-bound-why-your-gpu-shouldnt-do-both/)\n## [Prefill Is Compute-Bound. Decode Is Memory-Bound. Why Your GPU Shouldn’t Do Both.](https://towardsdatascience.com/prefill-is-compute-bound-decode-is-memory-bound-why-your-gpu-shouldnt-do-both/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nInside disaggregated LLM inference — the architecture shift behind 2-4x cost reduction that most ML teams haven't adopted yet.\n[Gokul Chandra Purnachandra Reddy](https://towardsdatascience.com/author/gokulchandra.purnachandrareddy/)\nApril 15, 202614 min read\n * [![](../images/023.jpg)](https://towardsdatascience.com/reinforcement-learning-101-q-learning-27add4c8536d/)\n## [Reinforcement Learning 101: Q-Learning](https://towardsdatascience.com/reinforcement-learning-101-q-learning-27add4c8536d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDecoding the Math behind Q-Learning, Action-Value Functions, Bellman Equations, and building them from scratch in Python.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nFebruary 28, 202434 min read\n * [![](../images/024.png)](https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa/)\n## [Paired t-test to evaluate Machine Learning classifiers using Python](https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLearn how to compare classifiers using hypothesis testing\n[Marcello Politi](https://towardsdatascience.com/author/marcellopoliti/)\nJuly 6, 20225 min read\n * [![](../images/025.jpeg)](https://towardsdatascience.com/active-and-upcoming-reinforcement-learning-competitions-8ed3c4fb14ab/)\n## [7+ Reinforcement learning competitions to check out in 2022](https://towardsdatascience.com/active-and-upcoming-reinforcement-learning-competitions-8ed3c4fb14ab/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA list of ongoing and annual competitions suitable for getting hands-on practice with reinforcement learning (while winning a prize or...\n[Joy Zhang](https://towardsdatascience.com/author/joooyz/)\nOctober 28, 20214 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1\n", "metadata": { "title": "Structured State Space Models Visually Explained | Towards Data Science", "description": "Part 2 - Towards Mamba State Space Models for Images, Videos and Time Series", "keywords": null, "author": "Sascha Kirch", "og:locale": "en_US", "og:type": "article", "og:title": "Structured State Space Models Visually Explained", "og:description": "Part 2 - Towards Mamba State Space Models for Images, Videos and Time Series", "og:url": "https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1j4Vw22q-llHYhh6YH2IH5w.png", "og:image:width": "4208", "og:image:height": "1131", "og:image:type": "image/png", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Sascha Kirch", "twitter:label2": "Est. reading time", "twitter:data2": "18 min read", "article:published_time": "2024-08-22T04:32:33.000Z", "article:modified_time": "2024-08-22T04:32:33.000Z", "article:author": "https://towardsdatascience.com/author/saschakirch/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "", "retried": true }, "code_blocks": [ { "language": "txt", "file": "attachments/24596f2cc41a/code/block_000.txt", "chars": 254 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:00:38Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:00:38Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:00:38Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:00:38Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [], "retry_at": "2026-10-03T08:03:39", "repair_version": 2, "repaired_at": "2026-10-20T17:30:13Z", "source_url": "https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Structured State Space Models Visually Explained\nPart 2 - Towards Mamba State Space Models for Images, Videos and Time Series\n[Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\nAugust 22, 202418 min read\n### [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://towardsdatascience.com/tagged/mamba-image-video-signal)\n![Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1j4Vw22q-llHYhh6YH2IH5w.png)\nImage by [Sascha Kirch](https://medium.com/@SaschaKirch).\n> This is part 2 of my new multi-part series [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://medium.com/@SaschaKirch/list/mamba-state-space-models-for-images-videos-and-timeseries-861ae0ad08fb).\nState space models, known to many disciplines of engineering for decades, are now making their debut in deep learning. On our journey towards Mamba selective state space models and their recent achievements in research, understanding the state space model is crucial. And, as is often the case in engineering, it is the details that make theoretical concepts applicable in practice. In addition to state space models, we must also discuss how to apply them to sequence data, how to handle long-range dependencies, and how to efficiently train them by exploiting certain matrix structures.\nStructured state space models build the theoretical foundation for Mamba. However, their connection to system theory and advanced algebra might be one of the obstacles to adopt this new framework. So, let's break it down, ensure we understand the key concepts and visualize them to shed some light onto this new-old theory.\n> Even if you don't end up using State Space models, understanding some of the tricks like why we need to speed up matrix multiplication and how we can exploit certain structures of matrices for that will certainly level up your skills as an engineer or developer.\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n* * *\n## Outline\n 1. [State Space Models in a Nutshell](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#51c1) 2.1 [Continuous Time Representation of the State Space Model](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#d5c1) 2.2 [Discretization of the State Space Model](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#ec5f) 2.3 [The Recurrent and Convolutional Representations of the SSM](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#8e7d)\n 2. [Enabling Long-Range Relationships](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#644c) 3.1 [State Size Matters](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#28b2) 3.2 [The HiPPO Framework](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#4384)\n 3. [Impose Structure on State Matrices for Efficiency](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#15b4) 4.1 [What Are Structured Matrices and Why Do They Matter?](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#fc04) 4.2 [Putting It All Together - S4: Structured State Space Sequence Models](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#4432)\n 4. [Wrapping Up](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#52fe) 5.1 [Summary](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#e69c) 5.2 [Limitations of Structured SSMs](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#0284) 5.3 [Continue Reading Part 3](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386/#192b)\n\n\n* * *\n## 1. Where We Left Off in Part 1\n> [**Towards Mamba State Space Models for Images, Videos and Time Series**](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a)\nIn part 1 we reviewed the strengths and weaknesses of recurrent neuronal networks (RNNs) and Transformers to motivate why we need a new model architecture.\nWe said RNNs are fast at inference time but slow during training and the Transformer is fast to train but slowly inferred. What we want is to find a model that is fast for both, training and inference, while beeing competititve with the Transformer's performance.\n* * *\n## 2. State Space Models in a Nutshell\nMamba builds on the idea of using a state space model for deep learning by learning its various matrices. So, let's have a brief primer on state space models before exploring what the \"structured\" part is all about. A state space model can be defined in continuous time representation to deal with continuous signals or can be discretized to handle discrete sequences of data.\nIt is the discretized state space model we are mostly interested in, since like recurrent neuronal networks (RNNs) and Transformers, a discrete SSM processes sequences of data., e.g. text tokens or samples of an analog time signal.\n### 2.1 Continuous Time Representation of the State Space Model\nA continuous-time state space model describes the relationship between an input signal propagating through a stateful system and the resulting output signal.\n![Figure 1: High-level functionality of a state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1z5ujpGhemnL2zH_It07NiA.gif)\nFigure 1: High-level functionality of a state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nIt is an idea borrowed from system theory. The output depends on the input and on the current state of the system, while the current state depends on the previous state and the input.\nThis relationship can be formulated efficiently by two simple equations, where _A_ , _B_ , _C_ and _D_ are matrices (which, as we'll see later, will be learned), _x(t)_ is the input signal, _y(t)_ the output signal and _h(t)_ and _h'(t)_ are the current and updated state respectively.\n![Equation. 1: State and output equations of the continuous time SSM.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/17gP7D-UCDLsUdSSCzyCgGQ.png)\nEquation. 1: State and output equations of the continuous time SSM.\nMatrix _D_ transforms the input _x(t)_ and maps it to the output _y(t)_ , which is typically included in the output equation of a classical SSM. However, when applying an SSM to deep learning, we remove this transformation and model it as a simple skip connection, thereby simplifying the SSM.\nWe can represent those equations in a block diagram that looks something like this:\n![Figure 2: Block diagram of a continuous time state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1MyATIKTDfprFHhQXjR1UPA.gif)\nFigure 2: Block diagram of a continuous time state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThis is the continuous-time representation of an SSM. There are two difficulties though:\n 1. finding an analytic solution for the state _h(t)_ of the model is challenging and\n 2. since we work on computers we usually work with discrete signals, not continuous ones.\n\n\n### 2.2 Discretization of the State Space Model\nSo, instead of mapping a function _x(t)_ to a function _y(t)_ , we are actually mapping a sequence _x[k]_ to a sequence _y[k]_. This implies we need a discretized state space model that can handle discrete signals.\n![Figure 3: High-level functionality of a discrete state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1qJiIzKB4zNQ8rn0lsmIlYA.gif)\nFigure 3: High-level functionality of a discrete state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nTo do so, we discretize our matrices _A_ and _B_. This results in the discretized representation of our state space model, now including a discretized state _h[k]_.\n![Equation 2: State and output equations of the discretized SSM.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/19vi-dmQLo0gtw6uXv2wuCg.png)\nEquation 2: State and output equations of the discretized SSM.\nTo discretize our state space model, we use yet another trick frequently used in system theory: We use a Zero-Order Hold (ZOH) module to turn our discrete signal _x[k]_ into a continuous time signal _x(t)_ which is then fed into the continuous SSM. The SSM will generate a continuous output _y(t)_ , which we then sample to obtain the discrete output sequence _y[k]_ :\n![Figure 4: Block diagram of a discretized state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1b-XO91g3YCkfym8NnNuDSA.png)\nFigure 4: Block diagram of a discretized state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nTo discretize the state space model, we must discretize matrices _A_ and _B_. Initially, most of the first deep learning based state space models like [LSSL](https://arxiv.org/abs/2110.13985) and [S4](https://arxiv.org/abs/2111.00396) used the bilinear discretization, while later ones like the [DSS](https://arxiv.org/abs/2203.14343) and Mamba use the ZOH discretization. For now, all we care about is that we need to discretize and that there might be more than one way to do so.\n![Equation 3: Zero Order Hold \\(ZOH\\) discretization vs. bilinear discretization.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1tfAgho2rwI_sfZLppTgZGw.png)\nEquation 3: Zero Order Hold (ZOH) discretization vs. bilinear discretization.\n_A_ and _B_ are the SSM's continuous-time matrices, _I_ is the identity matrix and Δ is the step size representing the resolution of the input, which in the deep learning setting is learned.\n> Note that the discretization allows us to use the time-continuous formulation of the SSM and apply it to discrete signals used in a computer. This implies that we become resolution invariant since we learn the underlying time-signal. Check out the [S4ND](https://arxiv.org/abs/2210.06583) paper for some further reading on that.\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n### 2.3 The Recurrent and Convolutional Representations of the SSM\nHave you noticed the recurrency in the state equation because _h[k]_ depends on _h[k-1]_? In fact, the state space model is quite like an RNN in that manner:\n![Figure 5: Recurrent representation of a discretized state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1TLUr5yk1jagHxetx1iZIRA.gif)\nFigure 5: Recurrent representation of a discretized state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThe matrices _A_ , _B_ and _C_ are time-invariant. That means that each input and each state is transformed with the same parameters, regardless of the time step. Linear time invariant systems, or short LTI-systems, have a very neat property: they can be represented as a convolution!\n![Equation 4: Convolution kernel of the discretized state space model.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1zjPkO08ggy6UFXjCswbdOw.png)\nEquation 4: Convolution kernel of the discretized state space model.\nThe kernel is constructed from the matrix _C_ and the discretized matrices _A_ and _B_. Note that the kernel has the same length _L_ as the input sequence. The kernel is convolved with the input _x_ to obtain the output _y_. To end up with the same sequence length at the output as at the input, the input is usually padded.\n![Figure 6: Example of the convolutional representation of a discretized state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/12Pj0CIM4wIX3n5Yhw0zxzw.gif)\nFigure 6: Example of the convolutional representation of a discretized state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThat's awesome because we know that convolutions are parallelizable and that GPUs and other HW-accelerators are optimized to calculate them fast.\nTo conclude this section, leveraging the properties of a **L** inear **T** ime **I** nvariant System, we ended up with 3 different ways to represent the SSM:\n![Figure 7: Different representations of state space models. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1g2fAoI-WYZ-sLPI7KseziA.png)\nFigure 7: Different representations of state space models. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nBoth the recurrent and the convolutional representations of the SSM are equal, hence we can decide whether we use one or the other.\n**During training** we would choose the convolutional representation because we have access to the entire sequence _x[k]_ and labels _y[k]_ ahead of time through our training data.\n**During inference** we would choose the recurrent representation because usually we only have access to samples from the past (e.g. autoregressive text generation). **** Note that the inference scales linearly with the sequence length since our next output only depends on the previous state and the current input.\nWe now have a model with efficient training and efficient inference. Awesome! 😎\n> It was the [LSSL (Linear State Space Layer) paper](https://arxiv.org/abs/2110.13985), that showed that we can train an SSM and that highlighted the connection between the different representations.\n> [**Get an email whenever Sascha Kirch publishes 🚀**](https://medium.com/@SaschaKirch/subscribe)[ _Get an email whenever Sascha Kirch publishes 🚀 Looking to learn more about deep learning or simply stay up to dat_e...medium.com](https://medium.com/@SaschaKirch/subscribe)\n## 3. Enabling Long-Range Relationships\nMatrix _A_ is responsible to create the next state from the previous state. The state of the system is a result of multiple updates considering past states and new inputs. It could be said that the state contains the information of the entire history of inputs. It is not hard to imagine that for very long sequences it might be a difficult task to remember all inputs accurately. So, how can we handle long-range dependencies?\n### 3.1 State-Size Matters\nLet's first talk about the state, its compression and the efficiency of inference.\n![Figure 8: Efficiency vs. performance of the RNN, SSM, Transformer and the desired model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1TT8x6JLkOvKF6ULfakCZ8A.png)\nFigure 8: Efficiency vs. performance of the RNN, SSM, Transformer and the desired model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nRNNs and Transformers can both theoretically deal with a never-ending sequence of data (sometimes people say: these models are unbound). **The RNN** does this by compressing the entire history into a single low-dimensional state. This is very efficient for inference, because now the next state only depends on the previous state and the current input. The performance of the RNN depends on how well the model learns to do this compression. Additionally, for long sequences and especially due to vanishing gradients, this is not an easy task to achieve, even for more advanced RNNs using gating mechanisms like GRUs and LSTMs. **The Transformer** on the other hand does not compress the history at all. It considers the entire history (and future if not masked like in autoregressive generation) through its attention mechanism. This allows the Transformer being trained in parallel, but at the same time memory and compute requirements explode with increasing sequence lengths during inference.\n**❓ So, again: how can we handle long-range dependencies with a highly compressed state, without the difficulties during training while achiving high performance?**\nOne Solution: we initialize our matrices using the HiPPO (**Hi** gh-order **P** olynomial **P** rojection **O** perators) framework.\n### 3.2 The HiPPO Framework\nThe main idea is to parametrize the matrices _A_ and _B_ from the state equation in such a way that we can reconstruct the history (i.e. the input signal from time _t=0_ to _t=now_) at any given point. Specifically, if the state is represented as some coefficients of certain orthogonal polynomials (e.g. the [Legendre Polynomial](https://en.wikipedia.org/wiki/Legendre_polynomials)), we can simply evaluate the polynomial to reconstruct the input signal up until the current time.\nFor example: We have an input function _x(t)_. At any given time step _ti_ , we want to find a reconstruction _g(ti)_ that approximates _x(t for t np.ndarray:\n A = np.zeros((N,N))\n for n in range(N):\n for k in range(N):\n if n > k:\n A[n,k] = (2*n+1)**0.5*(2*k+1)**0.5\n elif n == k:\n A[n,k] = (n+1)\n return A\n```\n\nThe LSSL mentioned earlier did show that one can train an SSM if initialized with the HiPPO matrix and it did show that its performance even surpasses those of the Transformer in the shown experiments. However, its expensive computation and memory requirements made the LSSL infeasible as a general sequence model. This hasn't been solved until the introduction of the S4 (Structured State Space Sequen _ce)_ model, which we will address in the next chapter.\n## 4. Impose Structure on State Matrices for Efficiency\nIt is finally time to talk about what the structure in the structured state space model is all about. We'll first try to gain some very basic intuition what structure is and why it is important, and then we will take a closer look onto the S4 model.\n### 4.1 What Are Structured Matrices and Why Do They Matter?\nLet's consider a standard square matrix of _NxN_ elements. We will focus on two aspects:\n 1. how much memory do we need to represent this matrix,\n 2. how fast can we multiply 2 matrices, where fast can be a result of several things, like being able to parallelize the computation, exploiting redundancies, and having less values to deal with in the first place.\n\n\n**How Much memory do we need?** Let's start with the memory requirements to store a matrix. We can have different kinds of matrices. Sometimes they are already in a certain form and sometimes we can bring them into a certain form leveraging rules from linear algebra. Let's have a look on some common matrix types:\n![Figure 10: Number of values to store for several common matrix types. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1kOVf7OSvPBfbe0SCezJlxw.png)\nFigure 10: Number of values to store for several common matrix types. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nWe see that in the worst case for a dense matrix, we need to store _N²_ values in memory. Not only that, but we must also load _N²_ values from the CPU to the GPU, where we then apply certain matrix operations.\nNote that there is a difference on how much values we need to store in our memory on CPU side (meaning in RAM or on disk) and how much memory we occupy in the GPU for computations. To make this clear, let's look at the low-rank factorization. Some matrices can be factorized into 2 low-rank matrices of _NxR_ , where _R_ is the rank of these matrices. We now only need to store _NR_ values for each of the two matrices, hence _2NR_ values. That also means we only need to load _2NR_ values instead of _N²_ from CPU to GPU. But if we want to make calculations with the original matrix, we first need to reconstruct the original matrix (some say materialize them) inside the GPU, meaning we still need _N²_ values in the cache of the GPU.\nOther representations like the diagonal or the sparse matrix on the other hand decrease both: the values stored on disk and the number of values needed to perform matrix operations on the GPU.\n**How many computations are required?** Now let's consider a standard matrix multiplication _C=AB_ , where _A_ and _B_ are either dense or diagonal matrices and _C_ is the result of the matrix multiplication:\n![Figure 11: Number of operations for matrix multiplication of different matrix types. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1uRqa5AMZj0Pjm4aTG4c2ag.png)\nFigure 11: Number of operations for matrix multiplication of different matrix types. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nFor the **case where both are dense matrices** , using the standard algorithm we all know for matrix multiplication, we need _N³_ multiplications and _(N-1)N²_ additions.\nIn **case** _**A**_**is a dense matrix and** _**B**_**is a diagonal matrix** , this reduces to _N²_ multiplications and _0_ additions.\nIn **case both** _**A**_**and** _**B**_**are diagonal** , we only require _N_ multiplications which are the element-wise multiplications of the entries of the diagonal. While this is desirable in terms of hardware resources, note that a diagonal matrix might be less expressive as a dense matrix, because it has less values. Like everything, it is a trade-off.\n**Parallelization of Computations** If you take a closer look onto the dense matrix multiplication, you might notice that calculating a single row of the output _C_ , involves all values of _B_ , but only a single row of _A_. Naively, you could speed up the computation by calculating each row of _C_ in parallel on different cores of a GPU. This means each core would only need _N²_ multiplications and _(N-1)N_ additions instead of _N³_ multiplications and _(N-1)N²_ additions as in the single core case because we now have a vector-matrix product, not a matrix-matrix product.\n**Long Story Short** To conclude, and using the example of a square matrix of _NxN_ , by trying to exploit certain properties of matrices, we want to bring them into a structured form such that we need to store and load less than _N²_ values and that we require less than _N³_ multiplications and _(N-1)N²_ additions for matrix multiplications.\nOf course, there are many other matrix structures that we can exploit all with their own pros and cons. Throughout this series on Mamba state space models we will encounter many of those, and all serve the same purpose: **requiring less resources while keeping the best performance possible.**\n### 4.2 Putting It All Together - S4: Structured State Space Sequence Models\nS4 aims to make SSMs more efficient by enforcing a certain structure upon the involved matrices. By doing so, it can apply well studied rules of algebra to reduce the complexity of the involved computations and save memory, especially for the repeated computation of the kernel _K_ of the convolutional view.\n![Figure 12: S4 Structured State Sequence Model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1Mc0_z_80Bk9ZObgn3kIQug.png)\nFigure 12: S4 Structured State Sequence Model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nCompared to LSSL, S4 is has been shown to be 30x faster with 400x less memory usage while it also gained a boost in performance.\n> The S4 paper is involves a lot of advanced linear algebra I will mostly skip in this blog post. If you want to dive deeper into the math and also the code implementation I recommend you to check out [The Annotated S4](https://srush.github.io/annotated-s4/#part-2-implementing-s4).\nRecall that calculating the convolution kernel _K_ for the convolutional representation of the SSM, repeated matrix multiplications of the discretized square matrix _A_ are required. Remember that the kernel has the same length _L_ as the input sequence _x_ and that the power of the matrix _A_ is increased with each entry of the kernel.\n![Equation 6: Convolutional kernel of the discretized state space model.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1zjPkO08ggy6UFXjCswbdOw.png)\nEquation 6: Convolutional kernel of the discretized state space model.\nTo be clear, a sequence of length _L=101_ would have matrices to the power of _100_! And _99_ and _98_ and... you get the point. Of course, we could reuse previous results, but still this is a lot to compute.\nThe main goal of S4 is to reduce the number of resources needed to calculate the kernel _K_ , by imposing a certain structure on matrix _A_.\nIdeally, matrix _A_ would be a diagonal matrix because as we saw earlier in Figure 11, multiplying two diagonal matrices result in another diagonal matrix requiring only _N_ multiplications.\nSince a HiPPO matrix cannot be diagonalized in a numerically stable manner (as proofed in the S4 paper), the next best thing is to decompose it into a normal plus low-rank (NPLR) form that can then further be simplified into a diagonal plus low-rank (DPLR) form by conjugation. Note that we want to hold on to the HiPPO matrix and do not change it to another diagonalizable matrix, because we need it for dealing with long-range dependencies.\n![Figure 13: Comparing different representations of matrix A and there resulting complexity to compute the convolution kernel K. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1TWP1YfIsPTi_zw1Z0_iq5Q.png)\nFigure 13: Comparing different representations of matrix A and there resulting complexity to compute the convolution kernel K. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nHaving matrix _A_ in DPLR form allows us to apply many complicated tricks. In summary, the kernel is calculated in the frequency domain and having the DPLR form allows rearranging the matrices into a Cauchy matrix, for which many hardware optimized algorithms exist.\nFurther, using the Woodburry identity allows transforming the calculation of multiple powers of _A_ into a single inverse. Finally, the inverse FFT (Fast Fourier Transform) is applied to obtain the final convolutional filter _K_.\nSo, the _SSM(A,B,C)_ previously a function of _A, B,_ and _C_ , is now a function _SSM(__**Λ-**__PP*,B,C)_ , where _**Λ,**__P, B_ and _C_ are learnable parameters of the S4.\n> If you are interested in the original implementation, here is the [dplr()](https://github.com/state-spaces/s4/blob/e757cef57d89e448c413de7325ed5601aceaac13/models/s4/s4.py#L453) function and the [nplr()](https://github.com/state-spaces/s4/blob/e757cef57d89e448c413de7325ed5601aceaac13/models/s4/s4.py#L374) function of the [S4 repository](https://github.com/state-spaces/s4).\n## 5. Wrapping Up\n### 5.1 Summary\n * State Space Models (SSMs) are sequence models.\n * SSMs can be described as recurrent model or as convolution model\n * They are inferred fast in their recurrent representation that scales linearly because they only depend on the current input and the previous state.\n * They are trained fast because of their convolutional representation that can be parallelized.\n * Since they are an LTI system, each input at any time step is treated equally, meaning transformed by the same matrix.\n * Structured state space sequence models aka. S4 use a certain structure on their matrices to make SSMs faster allowing them to be less restrictive with compute and memory requirements.\n\n\n### Thank You!\n> _🤝_ I'd like to t**hank you,** **t** he readers of my stories, for your constant support, clapping for my stories and, following me to never miss any of my latest articles. It is you who motivates me to continue writing, dive deep into complex topics, and become a better engineer/researcher by writing about them. You are awesome!\n### 5.2 Limitations of Structured SSMs\nThere is one big big issue though: SSMs cannot reason about different inputs at different time steps, since the matrices _A_ , _B_ and _C_ are time invariant meaning the same for all inputs.\n### 5.3 Continue Reading Part 3\n> [**Here Comes Mamba: The Selective State Space Model**](https://towardsdatascience.com/here-comes-mamba-the-selective-state-space-model-435e5d17a451)\nIn Part 3 we will discover how Mamba solves the issue of the structured state space model not differentiating between samples in the input sequence by adding selectivity to the model architecture. **❓ But what do we gain and at which cost?**\n## 6. Further Readings & Resources\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n### Blog Posts\n * [A Visual Guide to Mamba and State Space Models](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state)\n * [The Annotated S4](https://srush.github.io/annotated-s4/#part-2-implementing-s4)\n * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://hazyresearch.stanford.edu/blog/2020-12-05-hippo)\n\n\n### Papers:\n * [S4: Efficiently Modeling Long Sequences with Structured State Spaces](https://arxiv.org/abs/2111.00396) by A. Gu et.al., 31 Oct. 2021\n * [S5: Simplified State Space Layers for Sequence Modeling](https://arxiv.org/abs/2208.04933) by J. Smith et. al., 9 Aug. 2022\n * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://arxiv.org/abs/2008.07669) by A. Gu et.al., 17 Aug. 2020\n * [LSSL: Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://arxiv.org/abs/2110.13985) by A. Gu et. al., 26 Oct. 2021\n * [DSS: Diagonal State Spaces are as Effective as Structured State Spaces](https://arxiv.org/abs/2203.14343) by A. Gupta et. al., 27 Mar. 2022\n * [S4D: On the Parameterization and Initialization of Diagonal State Space Models](https://arxiv.org/abs/2206.11893) by A. Gu et. al., 23 Jun. 2022\n * [S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces](https://arxiv.org/abs/2210.06583) by E. Nguyen et. al., 12 Oct. 2020\n\n\n### Code\n * [S4 Code Repository](https://github.com/state-spaces/s4)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657088944_hlxvw4.png)\nWritten By\n## [Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[See all from Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Linear Algebra](https://towardsdatascience.com/tag/linear-algebra/)[Mamba](https://towardsdatascience.com/tag/mamba/)[Mamba Image Video Signal](https://towardsdatascience.com/tag/mamba-image-video-signal/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fstructured-state-space-models-visually-explained-86cfe2757386%2F&title=Structured%20State%20Space%20Models%20Visually%20Explained)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fstructured-state-space-models-visually-explained-86cfe2757386%2F&title=Structured%20State%20Space%20Models%20Visually%20Explained)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fstructured-state-space-models-visually-explained-86cfe2757386%2F&text=Structured%20State%20Space%20Models%20Visually%20Explained)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/1e3hsEHljdJ1Y18E9gKFJqw.png)](https://towardsdatascience.com/using-pytorchvideo-for-efficient-video-understanding-24d3cd99bc3c/)\n## [Using PyTorchVideo for efficient video understanding](https://towardsdatascience.com/using-pytorchvideo-for-efficient-video-understanding-24d3cd99bc3c/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLearn how to easily visualize and evaluate activity classification models from the PyTorchVideo library The...\n[Eric Hofesmann](https://towardsdatascience.com/author/eric.hofesmann/)\nMay 19, 20218 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/0ltGtN0KSKTL9rNBS.png)](https://towardsdatascience.com/writing-custom-cuda-kernels-with-triton-abf6b6ad1168/)\n## [Writing custom CUDA kernels with Triton](https://towardsdatascience.com/writing-custom-cuda-kernels-with-triton-abf6b6ad1168/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTriton is a language compiler to write highly optimized CUDA kernels. Learn the basics of GPU programming and how to use Triton.\n[Kushajveer Singh](https://towardsdatascience.com/author/kushaj/)\nSeptember 12, 20218 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/12/1KI0b00-QwNWcK3rrIW-d_w.jpeg)](https://towardsdatascience.com/generating-haiku-with-deep-learning-dbf5d18b4246/)\n## [Generating Haiku with Deep Learning](https://towardsdatascience.com/generating-haiku-with-deep-learning-dbf5d18b4246/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nUsing neural networks to generate haiku while enforcing the 5-7-5 syllable structure.\n[Jeremy Neiman](https://towardsdatascience.com/author/docmarionum1/)\nDecember 29, 20189 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/11/Cover-Image-MobileNetV3.jpg)](https://towardsdatascience.com/mobilenetv3-paper-walkthrough-the-tiny-giant-getting-even-smarter/)\n## [MobileNetV3 Paper Walkthrough: The Tiny Giant Getting Even Smarter](https://towardsdatascience.com/mobilenetv3-paper-walkthrough-the-tiny-giant-getting-even-smarter/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nMobileNetV3 with PyTorch — now featuring SE blocks and hard activation functions\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nNovember 2, 202527 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/0KrD-CazeOMUfS6JB-scaled.jpg)](https://towardsdatascience.com/neural-networks-are-like-human-bodies-852871fcbdb8/)\n## [Neural Networks are like human bodies](https://towardsdatascience.com/neural-networks-are-like-human-bodies-852871fcbdb8/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nOptimizers are like personal trainers\n[Pascal Janetzky](https://towardsdatascience.com/author/pascaljanetzky/)\nOctober 25, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/TDS_Banner.jpg)](https://towardsdatascience.com/prefill-is-compute-bound-decode-is-memory-bound-why-your-gpu-shouldnt-do-both/)\n## [Prefill Is Compute-Bound. Decode Is Memory-Bound. Why Your GPU Shouldn’t Do Both.](https://towardsdatascience.com/prefill-is-compute-bound-decode-is-memory-bound-why-your-gpu-shouldnt-do-both/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nInside disaggregated LLM inference — the architecture shift behind 2-4x cost reduction that most ML teams haven't adopted yet.\n[Gokul Chandra Purnachandra Reddy](https://towardsdatascience.com/author/gokulchandra.purnachandrareddy/)\nApril 15, 202614 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/0ekbczAH7WsCyijEF.jpg)](https://towardsdatascience.com/reinforcement-learning-101-q-learning-27add4c8536d/)\n## [Reinforcement Learning 101: Q-Learning](https://towardsdatascience.com/reinforcement-learning-101-q-learning-27add4c8536d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDecoding the Math behind Q-Learning, Action-Value Functions, Bellman Equations, and building them from scratch in Python.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nFebruary 28, 202434 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/1Sy6P9RA3aTyka0WKvH434A.png)](https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa/)\n## [Paired t-test to evaluate Machine Learning classifiers using Python](https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLearn how to compare classifiers using hypothesis testing\n[Marcello Politi](https://towardsdatascience.com/author/marcellopoliti/)\nJuly 6, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1ItZnim8bgUg2kd0NfqRKJQ-scaled.jpeg)](https://towardsdatascience.com/active-and-upcoming-reinforcement-learning-competitions-8ed3c4fb14ab/)\n## [7+ Reinforcement learning competitions to check out in 2022](https://towardsdatascience.com/active-and-upcoming-reinforcement-learning-competitions-8ed3c4fb14ab/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA list of ongoing and annual competitions suitable for getting hands-on practice with reinforcement learning (while winning a prize or...\n[Joy Zhang](https://towardsdatascience.com/author/joooyz/)\nOctober 28, 20214 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1\n", "body_base_path": "attachments/24596f2cc41a/pages/", "body_markdown_path": "attachments/24596f2cc41a/pages/archived-body.md", "local_html": "attachments/24596f2cc41a/pages/archived-body.html", "body_source": "prior_capture" }