{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Sascha Kirch",
  "title_listed": "Towards Mamba State Space Models for Images, Videos and Time Series",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/",
  "publish_date": "2024-08-14",
  "corpus_stage": "",
  "notes": "",
  "id": "66cae8cbf72f",
  "collected_at": "2026-10-02T00:48:09",
  "title": "Towards Mamba State Space Models for Images, Videos and Time Series | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Towards Mamba State Space Models for Images, Videos and Time Series\nPart 1\n[Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\nAugust 14, 202417 min read\n### [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://towardsdatascience.com/tagged/mamba-image-video-signal)\n# Towards Mamba State Space Models for Images, Videos and Time Series\n![Image by Sascha Kirch](../images/000.png)\nImage by [Sascha Kirch](https://medium.com/@SaschaKirch)\n> This is part 1 of my new multi-part series [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://medium.com/@SaschaKirch/list/mamba-state-space-models-for-images-videos-and-timeseries-861ae0ad08fb).\nIs Mamba all you need? Certainly, people have thought that for a long time of the Transformer architecture introduced by A. Vaswani et. al. in [Attention is all you need](https://arxiv.org/abs/1706.03762) back in 2017. And without any doubt, the transformer has revolutionized the field of deep learning over and over again. Its general-purpose architecture can easily be adapted for various data modalities such as text, images, videos and time series and it seems that the more compute resources and data you throw at the Transformer, the more performant it becomes.\nHowever, the Transformer's attention mechanism has a major drawback: it is of complexity _O(N²)_ , meaning it scales quadratically with the sequence length. This implies the larger the input sequence, the more compute resources you need, making large sequences often unfeasible to work with.\n❓ The question is: can we do better? Is there a way we can reduce the _O(N²)_ complexity while still being performant? And what would this new architecture look like?\n🏊 ‍♂️ Jump in and join me in this multi-part deep dive series on Mamba state space models where we will discover why Mamba is an interesting new general-purpose approach for many applications and data modalities.\n* * *\n## Outline\n  1. [What is this Series About?](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#9613)\n  2. [Why Do We Need a New Model?](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#c058)\n  3. [Structured State Space Models](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#0687)\n  4. [Here Comes Mamba](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#0e15)\n  5. [Vision Mamba - Mamba Applied on Images](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#8560)\n  6. [VideoMamba - Understanding Videos](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#fda1)\n  7. [MambaMixer - Adding Selectivity to Channels](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#efef)\n  8. [Mamba-2 - Closing the Gap Between Attention and SSMs](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#1708)\n  9. [Conclusion](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#8fa1)\n  10. [Further Readings & Resources](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#cbb9)\n\n\n* * *\n## **1. What is this series about?**\nObviously, it is all about Mamba, the model to be said to replace the mighty Transformer! But how does it work? What are the details not mentioned in the papers? And how do you deal with all different kinds of data modalities? To these and more questions you will find answers in this series of articles.\nExplicitly, in **part 1** , we will kick off this series by understanding the strengths and weaknesses of RNNs and Transformers to motivate the need for a new architecture. Further, I'll provide a primer on all the upcoming parts of this series introducing many recent research papers that build upon the success of Mamba.\nIn **part 2** we will look into structured state space models (SSMs) that provided the foundation for Mamba. We'll discover that while they solve the issue of the _O(N²)_ complexity, other obstacles emerge like how to deal with long-range dependencies and how to increase their performance. We continue in **part 3** with the introduction of Mamba and see how it forms a new general-purpose architecture that is trained fast, inferred fast and has a similar, and in some applications even superior, performance as the mighty Transformer.\n**Further down the road** , we will dive into recent research papers like [Vision Mamba](https://arxiv.org/abs/2401.09417), [VideoMamba](https://arxiv.org/abs/2403.06977), [MambaMixer](https://arxiv.org/abs/2403.19888v1) and [Mamba-2](https://arxiv.org/abs/2405.21060) to see how Mamba is extended to various data modalities such as images, videos and time signals and how it is constantly improved.\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n* * *\n## 2. Why Do We Need a New Model?\nTo understand the need for a new model architecture, let's have a quick look on the two most prominent sequence models we had until recently: The Transformer and the Recurent Neuronal Network (RNN):\n![Image 1: Properties of Transformers, RNNs and the new model architecture we desire to find. Image by Sascha Kirch.](../images/001.png)\nImage 1: Properties of Transformers, RNNs and the new model architecture we desire to find. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nUltimately, our goal is to have a model that trains fast, infers fast, while having a small memory footprint and does have access to all the relevant data of other samples of the sequence (Not more and not less).\nLet's compare the RNN and the Transformer on a very high level:\n![Image 2: Comparison of the high-level abstraction of how RNNs work vs. how Transformers work. Image by Sascha Kirch.](../images/002.png)\nImage 2: Comparison of the high-level abstraction of how RNNs work vs. how Transformers work. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\n### 2.1 A Closer Look onto the RNN\nIn an RNN, the input _x_ is propagated through the hidden state _h_ to obtain the output _y_. Note that the output _y[t]_ depends only on the hidden state _h[t]_ and the current input _x[t]_ and further note the current hidden state _h[t]_ depends on the previous hidden state _h[t-1]_ and the current input _x[t]_. This has the following implications:\n  1. Past input samples can only be accessed through the hidden state that compresses the information of past data points and that must be updated for each new sample using the new input.\n  2. We can only access samples from the past.\n  3. Regardless of the length of the input, the number of computations per output stay constant and the total number of computations scale linearly with the length of the input sequence.\n\n\nRNNs are fast at inference due to **3** , but cannot be parallelized during training due to **1** and **2**. Especially the high compression of the state might limit the performance of the RNN compared to the Transformer. Due to the compression, valuable information might be lost especially when considering long sequences and the fact that the states need to be trained with a gradient based algorithm which might lead to difficulties like vanishing gradients.\n### 2.2 What About the Transformer?\nIn the Transformer, each sample in the input sequence directly attends to itself and to every other sample of the sequence. In a sequence of _N_ samples, each individual sample calculates a score on how much it attends to the other samples, which is the reason for the _O(N²)_ complexity. This means that (for the vanilla attention mechanism):\n  1. There is no compression at all between different samples and the state has a size of _NxN_.\n  2. each sample can attend to previous and to future samples at once.\n  3. If we do autoregressive sampling (e.g. next word prediction), we need to recompute all attention maps again to consider the new sample.\n\n\nEspecially **2** is the reason we can parallelize the Transformer well during training and it is **1** that its prediction performance is so good. You can tackle almost every problem with this brute force approach of letting everything attend to everything and let the Transformer figure out what it should look at to improve its performance. Since there is no compression, it is not hard to imagine that we also waste a lot of resources computing things that are irrelevant.\n**❓ The question is how do we get the best of both worlds? Fast training, fast sampling and efficiency in terms of memory.**\n🤔Might Mamba be a candidate to achieve exactly this?\n## 3. Structured State Space Models\n> A primer for part 2 of this series\nLet's start with the basics. Mamba builds on top of the idea of using state space models for deep learning. A state space model, as well known in system theory, describes the relationship between an input signal propagating through a stateful system and the resulting output signal.\n![Image 3: Block diagram of a continuous-time state space model. Image by Sascha Kirch.](../images/003.gif)\nImage 3: Block diagram of a continuous-time state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nSince on a computer we work with discrete data, we need to discretize the state space model. Having the discrete state space model, we can uncover the two different representations: the recurrent representation and the convolutional representation. These two representations are equal, since the standard SSM is a linear and time-invariant (LTI) system.\n![Image 4: Different representations of state space models. Image by Sascha Kirch.](../images/004.png)\nImage 4: Different representations of state space models. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nWe can leverage this duality and can use the recurrent representation for inference and the convolutional representation for training. This means the inference time scales linearly with the sequence length while we can train the model efficiently on modern hardware accelerators.\n> _👉_ W**hat you'll find in part 2:**\n> We will dive deeper into the formulation of the state space model and the state equations. We'll continue with the discretization and uncover the different representations of the SSM. We then change gears and discover how we resolve long range dependencies using the HiPPO framework and finally, we reach the S4 model, the structured state space sequence model, a milestone on the way towards Mamba. We will also discuss why we even care about structure in the involved matrices.\n> [**Structured State Space Models Visually Explained**](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386)\n* * *\n## 4. Here Comes Mamba\n> A primer for part 3 of this series\nWith the S4 (Structured State Space Sequence Model) we ended up having a trainable state space model whose inference time scales linearly with the length of the input sequence and that can be trained efficiently leveraging the convolutional representation. Not only that, it also showed superior performance compared to the mighty Transformer (at least in the presented experiments).\nThe fact that S4 is a linear system, which we leveraged to switch between representations, is also its biggest weakness. Each sample of the input is treated equally because the matrices are time invariant. This is a big limitation because we might like to treat important inputs differently or probably ignore irrelevant inputs.\n![Image 5: Detailed view on the matrices involved in the selective state space model. Image by Sascha Kirch.](../images/005.png)\nImage 5: Detailed view on the matrices involved in the selective state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nMamba does exactly that: it makes the involved matrices time-variant to introduce the selectivity we want. As a consequence, we now have a time-varying system and we cannot use the convolutional representation anymore, hence we lose the possibility for fast and efficient training.\nMamba tackles this issue by further introducing a selective parallel scan, a technique combining kernel fusion, parallel associative scans and gradient recomputation to speed up the training of the model.\nIt further presents a general architecture, the Mamba Block, that just like the Transformer block can be stacked on top of each other build deep networks.\n![Image 6: Detailed block diagram of the Mamba-Block. Image by Sascha Kirch.](../images/006.png)\nImage 6: Detailed block diagram of the Mamba-Block. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\n> _👉_ W**hat you will find in part 3:**\n> We will talk about the selection mechanism of Mamba and see how this impacts the dimension of the involved matrices. We will further dive into the selective parallel scan and uncover what is behind the kernel fusion, the parallel associative scan and the recomputation of the gradients. Further, we will learn about the Mamba Block and I'll show you where the actually trained weights of the model hide and which dimensions they have (something that took me quite a while to understand to be honest)\n> [**Here Comes Mamba: The Selective State Space Model**](https://towardsdatascience.com/here-comes-mamba-the-selective-state-space-model-435e5d17a451)\n> [**Get an email whenever Sascha Kirch publishes 🚀**](https://medium.com/@SaschaKirch/subscribe)[ _Get an email whenever Sascha Kirch publishes 🚀 Looking to learn more about deep learning or simply stay up to dat_e...medium.com](https://medium.com/@SaschaKirch/subscribe)\n* * *\n## 5. Vision Mamba - Mamba Applied on Images\n> A primer for part 4 of this series\nHaving a new general architecture for sequence modeling, the Mamba Block, history starts to repeat itself. Just like the Transformer was applied to images by the introduction of the [Vision Transformer (ViT)](https://arxiv.org/abs/2010.11929), Mamba is applied to images resulting in Vision Mamba (Vim).\n![Image 7: Comparing the ViT-Vision Transformer and the ViM-Vision Mamba models. Images taken from A. Dosovitskiy et. al. and L. Zhu et. al., annotated by Sascha Kirch.](../images/007.png)\nImage 7: Comparing the ViT-Vision Transformer and the ViM-Vision Mamba models. Images taken from [A. Dosovitskiy et. al.](https://arxiv.org/abs/2010.11929) and [L. Zhu et. al.](https://arxiv.org/abs/2401.09417), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nVision Mamba aims to define a new generic vision backbone using Mamba state space models.\nTo represent an image as a sequence, because Mamba is still a sequence-to-sequence model after all, it is divided into patches, unrolled and then linearly projected and combined with positional encodings before fed into a stack of Mamba Encoder Blocks. The linearly scaling law of Mamba is especially interesting for high resolution images. Because patching the image as introduced in the ViT was a direct consequence of the _O(N²)_ complexity of the attention block in the Transformer. With Mamba having an _O(N)_ complexity, either smaller patches or higher resolutions can be processed for a fraction of the compute resources. In fact, without any attention mechanism involved, Vision Mamba was 2.8x faster than [DeiT](https://arxiv.org/abs/2012.12877) (Data efficient Vision Transformer) and saves up to 86.8% GPU memory on images of resolution 1024x1024.\nUnlike most time signals, where each new sample has a causal relationship with its past samples, an image does not has this property. In images we do have long range dependencies between various pixel positions before and after the current pixel. To deal with this non-causal representation, Vision Mamba introduces a bidirectional Encoder Block, that includes two SSMs: one that is inferred with the normal sequence and one that is inferred on the inverted sequence.\n![Image 8: Detailed block diagram of the Mamba-Block used in the ViM - Vision Mamba Encoder model. Image by Sascha Kirch.](../images/008.png)\nImage 8: Detailed block diagram of the Mamba-Block used in the ViM - Vision Mamba Encoder model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\n> _👉_ W**hat you will find in part 4:**\n> We will take a closer look on how to transform an image into a sequence. We will talk about the main motivation of the patchification and check how Mamba can handle smaller patches. We'll further review the insertion of the positional embeddings and the CLS token. Finally, we will talk about the bidirectional MambaBlock, what its intensions are and how it is implemented. We close part 4 with interpreting the experiments to see how it compares to previous methods.\n> [**Vision Mamba: Like a Vision Transformer but Better**](https://towardsdatascience.com/vision-mamba-like-a-vision-transformer-but-better-3b2660c35848)\n* * *\n## 6. VideoMamba - Understanding Videos\n> A primer for part 5 of this series\nMamba had to catch up very fast with the current developments of deep learning. Having more compute resources, data and foundation models as ever before, deep learning tackles data modalities previously thought to be too complex to handle. So, it was just a question off time how long it takes for Mamba to tackle videos (and this time was less than 2 Months after the ViM was published on arxiv).\nVideoMamba is the extension of Vision Mamba but applied on videos. It classifies videos instead of images.\n![Image 9: Architecture of VideoMamba. Image by K. Li et al., annotated by Sascha Kirch.](../images/009.png)\nImage 9: Architecture of VideoMamba. Image by [K. Li et al.](https://arxiv.org/abs/2403.06977), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nVideos add an extra temporal dimension to the input data. Previous methods used 3D convolutions or Transformers to tackle videos, but both are very expensive computationally and in terms of memory usage. Not only does video introduces long range dependencies through time, but video is also very redundant, since between frames, pixels often do not change or do only change slightly (imagine a scene containing a blue sky). One trick to deal with these redundancies was key frame extraction to reduce the number of frames that needed to be processed. Again, Mamba with its linear complexity of _O(N)_ comes in handy.\nThe architecture of VideoMamba is very similar to Vision Mamba. It patchifies the input, does a linear projection and adds positional encodings before feeding it into the Mamba Block. Now take a closer look onto the positional embeddings. The index of the spatial positional embedding for the patches is reset with every frame and each frame gets an extra temporal positional embedding that is constant for all patches within a frame.\nRemember that for Vision Mamba the bidirectional block was introduced to deal with the non-causal relationships of pixels within a given frame. As you can imagine, this becomes even more complex when adding an extra temporal dimension. In their experiments they showed it works better to first scan through the spatial dimensions before tackling the temporal dimensions.\n> _👉_ W**hat you will find in part 5:**\n> We start part 5 with a general overview of the model architecture and how it has evolved from Vision Mamba. We then take a closer look onto the positional embeddings and the different scanning directions of the spatiotemporal inputs. We will talk about how VideoMamba avoids overfitting through a self-distillation mechanism and we will find out how short-term actions and long range dependencies are improved.\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n* * *\n## 7. MambaMixer - Adding Selectivity to Channels\n> A primer for part 6 of this series\nOne minor detail we haven't mentioned yet in this article: Mamba and its variants apply the SSM to each channel individualy. This means information cannot flow from one channel to another. Or put differently: there is no selection between channels which limits Mamba's ability to find relationships in multi-dimensional data (e.g. images with their 3 color channels) or multi-variate time signals.\nMambaMixer solves this issue by a neat little trick: it transposes the feature map before feeding it through a second SSM:\n![Image 10: Effect of transposing the input matrix. Image by A. Behrouz et.al., annotated by Sascha Kirch.](../images/010.png)\nImage 10: Effect of transposing the input matrix. Image by [A. Behrouz et.al.](https://arxiv.org/abs/2403.19888), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nIn that way the SSM is first applied over the token axis and is then applied over the channel axis. The individual blocks are called the TokenMixer and the ChannelMixer respectively. Both, the TokenMixer and the ChannelMixer combined build the MambaMixer Block.\n![Image 11: General architecture of a MambaMixer Block. Image by A. Behrouz et.al., annotated by Sascha Kirch.](../images/011.png)\nImage 11: General architecture of a MambaMixer Block. Image by [A. Behrouz et.al.](https://arxiv.org/abs/2403.19888), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nEach block can in theory have multiple SMMs for different scanning directions over the input patches. Multiple MambaMixerBlocks are stacked to build the MambaMixer. To further improve the performance and the training stability, outputs from previous TokenMixer and ChannelMixer blocks are combined via a learned weighted average.\nIn the Paper they build two models using the MambaMixer Block: Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2). As the Name suggest ViM2 is applied on images and TSM2 on multi-variate time series.\n![Image 12: Architecture of the ViM2 \\(Vision MambaMixer\\) and the TSM2 \\(Time Series MambaMixer\\). Image by A. Behrouz et.al., annotated by Sascha Kirch.](../images/012.png)\nImage 12: Architecture of the ViM2 (Vision MambaMixer) and the TSM2 (Time Series MambaMixer). Image by [A. Behrouz et.al.](https://arxiv.org/abs/2403.19888), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThis paper is strongly inspired by the MLP Mixer Paper that aimed to simplify the architecture of the Vision Transformer (ViT). Here the friend-link to my post where I briefly introduced the MLP Mixer:\n> [**MLP Mixer in a Nutshell**](https://towardsdatascience.com/mlp-mixer-in-a-nutshell-eccffb68e4fc)\n> _👉_ W**hat you will find in part 6:**\n> We start part 6 by motivating the issue of not exchanging information between different channels. We then take a closer look onto the general architecture that achieves selectivity for tokens and for channels. After that, we inspect the two proposed implementations using the MambaMixer that resulted in the ViM2 and the TSM2 architecture.\n* * *\n## 8. Mamba-2 - Closing the Gap Between Attention and SSMs\n> A primer for part 7 of this series\nSo far Mamba has been applied very successful to various applications and data modalities. It has been a strong competitor to the Transformer architecture surpassing the Transformer in some of the tasks and benchmarks. Still, SSMs suffer from some drawbacks: They are up against a deep learning infrastructure specifically tailored towards the Transfomer and its attention mechanism. In addition, Mamba's selective parallel scan does not leverage the matrix multiplication units of modern GPUs.\nMamba-2 aims to reformulate the SSM to show the duality with the attention mechanism to leverage this optimized ecosystem and to take advantage of fast matrix multiplication units of modern hardware accelerators.\nSpecifically, if the SSM is formulated in terms of a _TxT_ semiseparable matrix, one can achieve exactly that.\n![Image 13: SSMs as semiseparable matrix. Image by Sascha Kirch.](../images/body-919327f3b4fdc9eb63cc.png)\nImage 13: SSMs as semiseparable matrix. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThe calculation of the output _Y_ becomes a matrix multiplication of the input _X_ and the SSM, which is defined by the matrices _A_ , _B_ and _C_ , that form a semiseparable matrix. Having such a matrix allows block decomposition, meaning, one can divide the matrix into sub-matrices of dimensions _QxQ_.\nThe resulting blocks on the diagonal are triangular matrices. None of those blocks share the same entries, meaning they can all be computed in parallel. The blocks off-diagonal can further be simplified using a low-rank factorization. Note that the same factors apear in multiple blocks, meaning they only need to be calculated once and can then be reused, allowing further speed up.\nBy defining this state space duality (SSD), the authors were able to\n  * show the connection between SSMs and attention,\n  * reformulate the SSM using semiseparable matrix transformations which in turn allows the implementation of a faster algorithm that uses less memory and takes advantage of the GPU's dedicated matrix multiplication cores,\n  * design of the new Mamba-2 Block that also allows to apply modern parallelization techniques such as tensor parallelism or sequence parallelism,\n  * introduce various analog concepts of prominent multi-head patterns in Transformers to SSMs like for example the multi-head SSM (MHS).\n\n\nTo conclude, the SSD framework ends up having a smaller memory footprint as attention and the SSM and is 2x-8x faster as the SSM since it uses matrix multiplication in the core of its formulation.\n> _👉_ W**hat you will find in part 7:**\n> In Part 7 we will revisit why having semiseparable matrices improves speed and memory footprint. We will also see, that the low-rank factorization can be interpreted as a state space model and how we can leverage modern hardware accelerators. We will further learn about the Mamba-2 block, and how it enables prominent parallelization techniques. And of course, you'll learn about the multi-head patterns that can be achieved.\n* * *\n## 9. Conclusion\nMamba state space models are a promising alternative to Transformers and RNNs to build a new generation general purpose architecture that can be applied to any data modality.\nWithin a short amount of time, Mamba-like models have been successfully applied to many problems in deep learning.\nThe close connection to system theory, the advanced algebra involved and the fact that modern deep learning ecosystems are tailored towards the Transformer might be an obstacle for researchers to adopt this new family of models. But thanks to the hard work and dedication of some researchers, Mamba-like models are becoming increasingly faster and more efficient on modern hardware accelerators and its integration into standard deep learning layers makes it increasingly easier to include into your own project.\n### Thank You!\n> _🤝_ I'd like to t**hank you,** **t** he readers of my stories, for your constant support, clapping for my stories and, following me to never miss any of my latest articles. It is you who motivates me to continue writing, dive deep into complex topics, and become a better engineer/researcher by writing about them. You are awesome!\n### Continue Reading Part 2\n> [**Structured State Space Models Visually Explained**](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386)\n* * *\n## Further Readings & Resources\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n### Blog Posts\n  * [A Visual Guide to Mamba and State Space Models](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state)\n  * [The Annotated S4](https://srush.github.io/annotated-s4/#part-2-implementing-s4)\n  * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://hazyresearch.stanford.edu/blog/2020-12-05-hippo)\n  * [Parallel Prefix Sum (Scan) with CUDA](https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda)\n\n\n### Papers:\n  * [S4: Efficiently Modeling Long Sequences with Structured State Spaces](https://arxiv.org/abs/2111.00396) by A. Gu et.al., 31 Oct. 2021\n  * [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752) by A. Gu et.al., 1 Dec. 2023\n  * [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://arxiv.org/abs/2401.09417) by L. Zhu et.al., 17 Jan. 2024\n  * [VideoMamba: State Space Model for Efficient Video Understanding](https://arxiv.org/abs/2403.06977) by K. Li et.al., 11 Mar. 2024\n  * [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://arxiv.org/abs/2403.19888) by A. Behrouz et.al., 29 Mar. 2024\n  * [Mamba-2: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060) by A. Gu et.al., 31 May 2024\n  * [S5: Simplified State Space Layers for Sequence Modeling](https://arxiv.org/abs/2208.04933) by J. Smith et.al., 9 Aug. 2022\n  * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://arxiv.org/abs/2008.07669) by A. Gu et.al., 17 Aug. 2020\n  * [LSSL: Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://arxiv.org/abs/2110.13985) by A. Gu et.al., 26 Oct. 2021\n  * [DSS: Diagonal State Spaces are as Effective as Structured State Spaces](https://arxiv.org/abs/2203.14343) by A. Gupta et.al., 27 Mar. 2022\n  * [S4D: On the Parameterization and Initialization of Diagonal State Space Models](https://arxiv.org/abs/2206.11893) by A. Gu et.al., 23 Jun. 2022\n  * [The Hidden Attention of Mamba Models](https://arxiv.org/abs/2403.01590) by A. Ali et.al., 3 Mar 2024\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-2648fe230469ec3ba2b1.png)\nWritten By\n## [Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[See all from Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Mamba Image Video Signal](https://towardsdatascience.com/tag/mamba-image-video-signal/)[State Space Model](https://towardsdatascience.com/tag/state-space-model/)[Transformers](https://towardsdatascience.com/tag/transformers/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ftowards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a%2F&title=Towards%20Mamba%20State%20Space%20Models%20for%20Images%2C%20Videos%20and%20Time%20Series)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ftowards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a%2F&title=Towards%20Mamba%20State%20Space%20Models%20for%20Images%2C%20Videos%20and%20Time%20Series)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ftowards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a%2F&text=Towards%20Mamba%20State%20Space%20Models%20for%20Images%2C%20Videos%20and%20Time%20Series)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/013.png)](https://towardsdatascience.com/which-is-better-for-your-machine-learning-task-opencv-or-tensorflow-ed16403c5799/)\n## [Which is Better For Your Machine Learning Task, OpenCV or TensorFlow?](https://towardsdatascience.com/which-is-better-for-your-machine-learning-task-opencv-or-tensorflow-ed16403c5799/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nIs one of them really better than the other? An explanation from a ML engineer who uses both frameworks.\n[Nour Islam Mokhtari](https://towardsdatascience.com/author/nourislam/)\nAugust 1, 20215 min read\n  * [![](../images/014.png)](https://towardsdatascience.com/from-patches-to-slides-how-to-train-deep-learning-models-on-gigapixel-images-with-weak-supervision-d2cd2081cfd7/)\n## [From Patches to Slides: How to Train Deep Learning Models on Gigapixel Images With Weak Supervision](https://towardsdatascience.com/from-patches-to-slides-how-to-train-deep-learning-models-on-gigapixel-images-with-weak-supervision-d2cd2081cfd7/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA review of techniques for modeling whole slide histology images and recommendations for different situations.\n[Heather Couture](https://towardsdatascience.com/author/hdcouture/)\nMarch 31, 202112 min read\n  * [![](../images/015.png)](https://towardsdatascience.com/automating-ticket-creation-in-jira-with-the-openai-agents-sdk-a-step-by-step-guide/)\n## [Automating Ticket Creation in Jira With the OpenAI Agents SDK: A Step-by-Step Guide](https://towardsdatascience.com/automating-ticket-creation-in-jira-with-the-openai-agents-sdk-a-step-by-step-guide/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLearn how to create AI Agents using the OpenAI Agents SDK to automate Jira ticket creation from a meeting transcript.\n[Juan Carlos Suarez](https://towardsdatascience.com/author/juancarlos.suarez/)\nJuly 24, 202514 min read\n  * [![](../images/016.jpg)](https://towardsdatascience.com/building-a-biomedical-entity-linker-with-llms-d385cb85c15a/)\n## [Building a Biomedical Entity Linker with LLMs](https://towardsdatascience.com/building-a-biomedical-entity-linker-with-llms-d385cb85c15a/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow can an LLM be applied effectively for biomedical entity linking?\n[Anand Subramanian](https://towardsdatascience.com/author/anand.subu10/)\nMarch 19, 202427 min read\n  * [![](../images/017.png)](https://towardsdatascience.com/from-physics-to-probability-hamiltonian-mechanics-for-generative-modeling-and-mcmc/)\n## [From Physics to Probability: Hamiltonian Mechanics for Generative Modeling and MCMC](https://towardsdatascience.com/from-physics-to-probability-hamiltonian-mechanics-for-generative-modeling-and-mcmc/)\n[Math](https://towardsdatascience.com/category/math/)\nMathematics, Hamiltonian Monte Carlo, generative models, and research frontiers ✨\n[Soran Ghaderi](https://towardsdatascience.com/author/soran-ghaderi/)\nMarch 28, 202515 min read\n  * [![](../images/018.jpg)](https://towardsdatascience.com/courage-to-learn-ml-demystifying-l1-l2-regularization-part-3-ee27cd4b557a/)\n## [Courage to Learn ML: Demystifying L1 & L2 Regularization (part 3)](https://towardsdatascience.com/courage-to-learn-ml-demystifying-l1-l2-regularization-part-3-ee27cd4b557a/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nWhy L0.5, L3, and L4 Regularizations Are Uncommon\n[Amy Ma](https://towardsdatascience.com/author/amyma101/)\nNovember 29, 20236 min read\n  * [![](../images/019.jpg)](https://towardsdatascience.com/galaxy-zoo-classification-with-keras-219184aff581/)\n## [Galaxy Zoo classification with Keras](https://towardsdatascience.com/galaxy-zoo-classification-with-keras-219184aff581/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nThe full code for this is available on my Github at https://github.com/jameslawlor/kaggle_galaxy_zoo.\n[James](https://towardsdatascience.com/author/jameslawlor/)\nJuly 12, 20175 min read\n  * [![](../images/020.png)](https://towardsdatascience.com/3-ways-to-leverage-apple-silicons-gpu-for-deep-learning-2cbb5b268b76/)\n## [3 ways to leverage Apple Silicon's GPU for Deep Learning](https://towardsdatascience.com/3-ways-to-leverage-apple-silicons-gpu-for-deep-learning-2cbb5b268b76/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nApple Silicon has brought impressive performance gains with great power efficiency. Can we use these chips for Deep Learning as well?\n[Jiří Moravčík](https://towardsdatascience.com/author/jiri.moravcik/)\nMarch 1, 20244 min read\n  * [![](../images/021.jpg)](https://towardsdatascience.com/7-tips-to-choose-the-best-optimizer-47bb9c1219e/)\n## [7 tips to choose the best optimizer](https://towardsdatascience.com/7-tips-to-choose-the-best-optimizer-47bb9c1219e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nBased on my experience.\n[Davide Giordano](https://towardsdatascience.com/author/davidegiordano/)\nJuly 25, 20205 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1\n",
  "metadata": {
    "title": "Towards Mamba State Space Models for Images, Videos and Time Series | Towards Data Science",
    "description": "Part 1",
    "keywords": null,
    "author": "Sascha Kirch",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Towards Mamba State Space Models for Images, Videos and Time Series",
    "og:description": "Part 1",
    "og:url": "https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1IGhsnp6ALHeBjFMVnOJ36Q.png",
    "og:image:width": "4190",
    "og:image:height": "1132",
    "og:image:type": "image/png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Sascha Kirch",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "17 min read",
    "article:published_time": "2024-08-14T17:33:44.000Z",
    "article:modified_time": "2024-08-14T17:33:44.000Z",
    "article:author": "https://towardsdatascience.com/author/saschakirch/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": "",
    "retried": true
  },
  "code_blocks": [],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:58:21Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:58:21Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:58:21Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:58:21Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [],
  "retry_at": "2026-10-03T08:03:32",
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:29:28Z",
  "source_url": "https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Towards Mamba State Space Models for Images, Videos and Time Series\nPart 1\n[Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\nAugust 14, 202417 min read\n### [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://towardsdatascience.com/tagged/mamba-image-video-signal)\n# Towards Mamba State Space Models for Images, Videos and Time Series\n![Image by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1IGhsnp6ALHeBjFMVnOJ36Q.png)\nImage by [Sascha Kirch](https://medium.com/@SaschaKirch)\n> This is part 1 of my new multi-part series [🐍 Towards Mamba State Space Models for Images, Videos and Time Series](https://medium.com/@SaschaKirch/list/mamba-state-space-models-for-images-videos-and-timeseries-861ae0ad08fb).\nIs Mamba all you need? Certainly, people have thought that for a long time of the Transformer architecture introduced by A. Vaswani et. al. in [Attention is all you need](https://arxiv.org/abs/1706.03762) back in 2017. And without any doubt, the transformer has revolutionized the field of deep learning over and over again. Its general-purpose architecture can easily be adapted for various data modalities such as text, images, videos and time series and it seems that the more compute resources and data you throw at the Transformer, the more performant it becomes.\nHowever, the Transformer's attention mechanism has a major drawback: it is of complexity _O(N²)_ , meaning it scales quadratically with the sequence length. This implies the larger the input sequence, the more compute resources you need, making large sequences often unfeasible to work with.\n❓ The question is: can we do better? Is there a way we can reduce the _O(N²)_ complexity while still being performant? And what would this new architecture look like?\n🏊 ‍♂️ Jump in and join me in this multi-part deep dive series on Mamba state space models where we will discover why Mamba is an interesting new general-purpose approach for many applications and data modalities.\n* * *\n## Outline\n  1. [What is this Series About?](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#9613)\n  2. [Why Do We Need a New Model?](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#c058)\n  3. [Structured State Space Models](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#0687)\n  4. [Here Comes Mamba](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#0e15)\n  5. [Vision Mamba - Mamba Applied on Images](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#8560)\n  6. [VideoMamba - Understanding Videos](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#fda1)\n  7. [MambaMixer - Adding Selectivity to Channels](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#efef)\n  8. [Mamba-2 - Closing the Gap Between Attention and SSMs](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#1708)\n  9. [Conclusion](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#8fa1)\n  10. [Further Readings & Resources](https://towardsdatascience.com/towards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a/#cbb9)\n\n\n* * *\n## **1. What is this series about?**\nObviously, it is all about Mamba, the model to be said to replace the mighty Transformer! But how does it work? What are the details not mentioned in the papers? And how do you deal with all different kinds of data modalities? To these and more questions you will find answers in this series of articles.\nExplicitly, in **part 1** , we will kick off this series by understanding the strengths and weaknesses of RNNs and Transformers to motivate the need for a new architecture. Further, I'll provide a primer on all the upcoming parts of this series introducing many recent research papers that build upon the success of Mamba.\nIn **part 2** we will look into structured state space models (SSMs) that provided the foundation for Mamba. We'll discover that while they solve the issue of the _O(N²)_ complexity, other obstacles emerge like how to deal with long-range dependencies and how to increase their performance. We continue in **part 3** with the introduction of Mamba and see how it forms a new general-purpose architecture that is trained fast, inferred fast and has a similar, and in some applications even superior, performance as the mighty Transformer.\n**Further down the road** , we will dive into recent research papers like [Vision Mamba](https://arxiv.org/abs/2401.09417), [VideoMamba](https://arxiv.org/abs/2403.06977), [MambaMixer](https://arxiv.org/abs/2403.19888v1) and [Mamba-2](https://arxiv.org/abs/2405.21060) to see how Mamba is extended to various data modalities such as images, videos and time signals and how it is constantly improved.\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n* * *\n## 2. Why Do We Need a New Model?\nTo understand the need for a new model architecture, let's have a quick look on the two most prominent sequence models we had until recently: The Transformer and the Recurent Neuronal Network (RNN):\n![Image 1: Properties of Transformers, RNNs and the new model architecture we desire to find. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1lKlJVniJ3rDvVlwjticfig.png)\nImage 1: Properties of Transformers, RNNs and the new model architecture we desire to find. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nUltimately, our goal is to have a model that trains fast, infers fast, while having a small memory footprint and does have access to all the relevant data of other samples of the sequence (Not more and not less).\nLet's compare the RNN and the Transformer on a very high level:\n![Image 2: Comparison of the high-level abstraction of how RNNs work vs. how Transformers work. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/19EwF2_wmQugISGQkpxjzTw.png)\nImage 2: Comparison of the high-level abstraction of how RNNs work vs. how Transformers work. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\n### 2.1 A Closer Look onto the RNN\nIn an RNN, the input _x_ is propagated through the hidden state _h_ to obtain the output _y_. Note that the output _y[t]_ depends only on the hidden state _h[t]_ and the current input _x[t]_ and further note the current hidden state _h[t]_ depends on the previous hidden state _h[t-1]_ and the current input _x[t]_. This has the following implications:\n  1. Past input samples can only be accessed through the hidden state that compresses the information of past data points and that must be updated for each new sample using the new input.\n  2. We can only access samples from the past.\n  3. Regardless of the length of the input, the number of computations per output stay constant and the total number of computations scale linearly with the length of the input sequence.\n\n\nRNNs are fast at inference due to **3** , but cannot be parallelized during training due to **1** and **2**. Especially the high compression of the state might limit the performance of the RNN compared to the Transformer. Due to the compression, valuable information might be lost especially when considering long sequences and the fact that the states need to be trained with a gradient based algorithm which might lead to difficulties like vanishing gradients.\n### 2.2 What About the Transformer?\nIn the Transformer, each sample in the input sequence directly attends to itself and to every other sample of the sequence. In a sequence of _N_ samples, each individual sample calculates a score on how much it attends to the other samples, which is the reason for the _O(N²)_ complexity. This means that (for the vanilla attention mechanism):\n  1. There is no compression at all between different samples and the state has a size of _NxN_.\n  2. each sample can attend to previous and to future samples at once.\n  3. If we do autoregressive sampling (e.g. next word prediction), we need to recompute all attention maps again to consider the new sample.\n\n\nEspecially **2** is the reason we can parallelize the Transformer well during training and it is **1** that its prediction performance is so good. You can tackle almost every problem with this brute force approach of letting everything attend to everything and let the Transformer figure out what it should look at to improve its performance. Since there is no compression, it is not hard to imagine that we also waste a lot of resources computing things that are irrelevant.\n**❓ The question is how do we get the best of both worlds? Fast training, fast sampling and efficiency in terms of memory.**\n🤔Might Mamba be a candidate to achieve exactly this?\n## 3. Structured State Space Models\n> A primer for part 2 of this series\nLet's start with the basics. Mamba builds on top of the idea of using state space models for deep learning. A state space model, as well known in system theory, describes the relationship between an input signal propagating through a stateful system and the resulting output signal.\n![Image 3: Block diagram of a continuous-time state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1I4-pv36B9OzqBpriACoTTA.gif)\nImage 3: Block diagram of a continuous-time state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nSince on a computer we work with discrete data, we need to discretize the state space model. Having the discrete state space model, we can uncover the two different representations: the recurrent representation and the convolutional representation. These two representations are equal, since the standard SSM is a linear and time-invariant (LTI) system.\n![Image 4: Different representations of state space models. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1g2fAoI-WYZ-sLPI7KseziA.png)\nImage 4: Different representations of state space models. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nWe can leverage this duality and can use the recurrent representation for inference and the convolutional representation for training. This means the inference time scales linearly with the sequence length while we can train the model efficiently on modern hardware accelerators.\n> _👉_ W**hat you'll find in part 2:**\n> We will dive deeper into the formulation of the state space model and the state equations. We'll continue with the discretization and uncover the different representations of the SSM. We then change gears and discover how we resolve long range dependencies using the HiPPO framework and finally, we reach the S4 model, the structured state space sequence model, a milestone on the way towards Mamba. We will also discuss why we even care about structure in the involved matrices.\n> [**Structured State Space Models Visually Explained**](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386)\n* * *\n## 4. Here Comes Mamba\n> A primer for part 3 of this series\nWith the S4 (Structured State Space Sequence Model) we ended up having a trainable state space model whose inference time scales linearly with the length of the input sequence and that can be trained efficiently leveraging the convolutional representation. Not only that, it also showed superior performance compared to the mighty Transformer (at least in the presented experiments).\nThe fact that S4 is a linear system, which we leveraged to switch between representations, is also its biggest weakness. Each sample of the input is treated equally because the matrices are time invariant. This is a big limitation because we might like to treat important inputs differently or probably ignore irrelevant inputs.\n![Image 5: Detailed view on the matrices involved in the selective state space model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/09/1SrSL3cTLPue63ou_gd7mbw.png)\nImage 5: Detailed view on the matrices involved in the selective state space model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nMamba does exactly that: it makes the involved matrices time-variant to introduce the selectivity we want. As a consequence, we now have a time-varying system and we cannot use the convolutional representation anymore, hence we lose the possibility for fast and efficient training.\nMamba tackles this issue by further introducing a selective parallel scan, a technique combining kernel fusion, parallel associative scans and gradient recomputation to speed up the training of the model.\nIt further presents a general architecture, the Mamba Block, that just like the Transformer block can be stacked on top of each other build deep networks.\n![Image 6: Detailed block diagram of the Mamba-Block. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1nojazskKhQrwZZzNYlcWug.png)\nImage 6: Detailed block diagram of the Mamba-Block. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\n> _👉_ W**hat you will find in part 3:**\n> We will talk about the selection mechanism of Mamba and see how this impacts the dimension of the involved matrices. We will further dive into the selective parallel scan and uncover what is behind the kernel fusion, the parallel associative scan and the recomputation of the gradients. Further, we will learn about the Mamba Block and I'll show you where the actually trained weights of the model hide and which dimensions they have (something that took me quite a while to understand to be honest)\n> [**Here Comes Mamba: The Selective State Space Model**](https://towardsdatascience.com/here-comes-mamba-the-selective-state-space-model-435e5d17a451)\n> [**Get an email whenever Sascha Kirch publishes 🚀**](https://medium.com/@SaschaKirch/subscribe)[ _Get an email whenever Sascha Kirch publishes 🚀 Looking to learn more about deep learning or simply stay up to dat_e...medium.com](https://medium.com/@SaschaKirch/subscribe)\n* * *\n## 5. Vision Mamba - Mamba Applied on Images\n> A primer for part 4 of this series\nHaving a new general architecture for sequence modeling, the Mamba Block, history starts to repeat itself. Just like the Transformer was applied to images by the introduction of the [Vision Transformer (ViT)](https://arxiv.org/abs/2010.11929), Mamba is applied to images resulting in Vision Mamba (Vim).\n![Image 7: Comparing the ViT-Vision Transformer and the ViM-Vision Mamba models. Images taken from A. Dosovitskiy et. al. and L. Zhu et. al., annotated by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/09/1veCGntYlzCe93AnwXCHPVQ.png)\nImage 7: Comparing the ViT-Vision Transformer and the ViM-Vision Mamba models. Images taken from [A. Dosovitskiy et. al.](https://arxiv.org/abs/2010.11929) and [L. Zhu et. al.](https://arxiv.org/abs/2401.09417), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nVision Mamba aims to define a new generic vision backbone using Mamba state space models.\nTo represent an image as a sequence, because Mamba is still a sequence-to-sequence model after all, it is divided into patches, unrolled and then linearly projected and combined with positional encodings before fed into a stack of Mamba Encoder Blocks. The linearly scaling law of Mamba is especially interesting for high resolution images. Because patching the image as introduced in the ViT was a direct consequence of the _O(N²)_ complexity of the attention block in the Transformer. With Mamba having an _O(N)_ complexity, either smaller patches or higher resolutions can be processed for a fraction of the compute resources. In fact, without any attention mechanism involved, Vision Mamba was 2.8x faster than [DeiT](https://arxiv.org/abs/2012.12877) (Data efficient Vision Transformer) and saves up to 86.8% GPU memory on images of resolution 1024x1024.\nUnlike most time signals, where each new sample has a causal relationship with its past samples, an image does not has this property. In images we do have long range dependencies between various pixel positions before and after the current pixel. To deal with this non-causal representation, Vision Mamba introduces a bidirectional Encoder Block, that includes two SSMs: one that is inferred with the normal sequence and one that is inferred on the inverted sequence.\n![Image 8: Detailed block diagram of the Mamba-Block used in the ViM - Vision Mamba Encoder model. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1YZh5pb135M5YP4WGXY4d0w.png)\nImage 8: Detailed block diagram of the Mamba-Block used in the ViM - Vision Mamba Encoder model. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\n> _👉_ W**hat you will find in part 4:**\n> We will take a closer look on how to transform an image into a sequence. We will talk about the main motivation of the patchification and check how Mamba can handle smaller patches. We'll further review the insertion of the positional embeddings and the CLS token. Finally, we will talk about the bidirectional MambaBlock, what its intensions are and how it is implemented. We close part 4 with interpreting the experiments to see how it compares to previous methods.\n> [**Vision Mamba: Like a Vision Transformer but Better**](https://towardsdatascience.com/vision-mamba-like-a-vision-transformer-but-better-3b2660c35848)\n* * *\n## 6. VideoMamba - Understanding Videos\n> A primer for part 5 of this series\nMamba had to catch up very fast with the current developments of deep learning. Having more compute resources, data and foundation models as ever before, deep learning tackles data modalities previously thought to be too complex to handle. So, it was just a question off time how long it takes for Mamba to tackle videos (and this time was less than 2 Months after the ViM was published on arxiv).\nVideoMamba is the extension of Vision Mamba but applied on videos. It classifies videos instead of images.\n![Image 9: Architecture of VideoMamba. Image by K. Li et al., annotated by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/12EOvKiHc6M4vzAmdSALgZQ.png)\nImage 9: Architecture of VideoMamba. Image by [K. Li et al.](https://arxiv.org/abs/2403.06977), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nVideos add an extra temporal dimension to the input data. Previous methods used 3D convolutions or Transformers to tackle videos, but both are very expensive computationally and in terms of memory usage. Not only does video introduces long range dependencies through time, but video is also very redundant, since between frames, pixels often do not change or do only change slightly (imagine a scene containing a blue sky). One trick to deal with these redundancies was key frame extraction to reduce the number of frames that needed to be processed. Again, Mamba with its linear complexity of _O(N)_ comes in handy.\nThe architecture of VideoMamba is very similar to Vision Mamba. It patchifies the input, does a linear projection and adds positional encodings before feeding it into the Mamba Block. Now take a closer look onto the positional embeddings. The index of the spatial positional embedding for the patches is reset with every frame and each frame gets an extra temporal positional embedding that is constant for all patches within a frame.\nRemember that for Vision Mamba the bidirectional block was introduced to deal with the non-causal relationships of pixels within a given frame. As you can imagine, this becomes even more complex when adding an extra temporal dimension. In their experiments they showed it works better to first scan through the spatial dimensions before tackling the temporal dimensions.\n> _👉_ W**hat you will find in part 5:**\n> We start part 5 with a general overview of the model architecture and how it has evolved from Vision Mamba. We then take a closer look onto the positional embeddings and the different scanning directions of the spatiotemporal inputs. We will talk about how VideoMamba avoids overfitting through a self-distillation mechanism and we will find out how short-term actions and long range dependencies are improved.\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n* * *\n## 7. MambaMixer - Adding Selectivity to Channels\n> A primer for part 6 of this series\nOne minor detail we haven't mentioned yet in this article: Mamba and its variants apply the SSM to each channel individualy. This means information cannot flow from one channel to another. Or put differently: there is no selection between channels which limits Mamba's ability to find relationships in multi-dimensional data (e.g. images with their 3 color channels) or multi-variate time signals.\nMambaMixer solves this issue by a neat little trick: it transposes the feature map before feeding it through a second SSM:\n![Image 10: Effect of transposing the input matrix. Image by A. Behrouz et.al., annotated by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1UbZxdRvX6W5nZuworSkEtw.png)\nImage 10: Effect of transposing the input matrix. Image by [A. Behrouz et.al.](https://arxiv.org/abs/2403.19888), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nIn that way the SSM is first applied over the token axis and is then applied over the channel axis. The individual blocks are called the TokenMixer and the ChannelMixer respectively. Both, the TokenMixer and the ChannelMixer combined build the MambaMixer Block.\n![Image 11: General architecture of a MambaMixer Block. Image by A. Behrouz et.al., annotated by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1nD___YXAl8dkfzZbzfNOdA.png)\nImage 11: General architecture of a MambaMixer Block. Image by [A. Behrouz et.al.](https://arxiv.org/abs/2403.19888), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nEach block can in theory have multiple SMMs for different scanning directions over the input patches. Multiple MambaMixerBlocks are stacked to build the MambaMixer. To further improve the performance and the training stability, outputs from previous TokenMixer and ChannelMixer blocks are combined via a learned weighted average.\nIn the Paper they build two models using the MambaMixer Block: Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2). As the Name suggest ViM2 is applied on images and TSM2 on multi-variate time series.\n![Image 12: Architecture of the ViM2 \\(Vision MambaMixer\\) and the TSM2 \\(Time Series MambaMixer\\). Image by A. Behrouz et.al., annotated by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/11KObOVhy9q86YV5Ae4caTg.png)\nImage 12: Architecture of the ViM2 (Vision MambaMixer) and the TSM2 (Time Series MambaMixer). Image by [A. Behrouz et.al.](https://arxiv.org/abs/2403.19888), annotated by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThis paper is strongly inspired by the MLP Mixer Paper that aimed to simplify the architecture of the Vision Transformer (ViT). Here the friend-link to my post where I briefly introduced the MLP Mixer:\n> [**MLP Mixer in a Nutshell**](https://towardsdatascience.com/mlp-mixer-in-a-nutshell-eccffb68e4fc)\n> _👉_ W**hat you will find in part 6:**\n> We start part 6 by motivating the issue of not exchanging information between different channels. We then take a closer look onto the general architecture that achieves selectivity for tokens and for channels. After that, we inspect the two proposed implementations using the MambaMixer that resulted in the ViM2 and the TSM2 architecture.\n* * *\n## 8. Mamba-2 - Closing the Gap Between Attention and SSMs\n> A primer for part 7 of this series\nSo far Mamba has been applied very successful to various applications and data modalities. It has been a strong competitor to the Transformer architecture surpassing the Transformer in some of the tasks and benchmarks. Still, SSMs suffer from some drawbacks: They are up against a deep learning infrastructure specifically tailored towards the Transfomer and its attention mechanism. In addition, Mamba's selective parallel scan does not leverage the matrix multiplication units of modern GPUs.\nMamba-2 aims to reformulate the SSM to show the duality with the attention mechanism to leverage this optimized ecosystem and to take advantage of fast matrix multiplication units of modern hardware accelerators.\nSpecifically, if the SSM is formulated in terms of a _TxT_ semiseparable matrix, one can achieve exactly that.\n![Image 13: SSMs as semiseparable matrix. Image by Sascha Kirch.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1YiArNTJm867Qo-neceDQew.png)\nImage 13: SSMs as semiseparable matrix. Image by [Sascha Kirch](https://medium.com/@SaschaKirch).\nThe calculation of the output _Y_ becomes a matrix multiplication of the input _X_ and the SSM, which is defined by the matrices _A_ , _B_ and _C_ , that form a semiseparable matrix. Having such a matrix allows block decomposition, meaning, one can divide the matrix into sub-matrices of dimensions _QxQ_.\nThe resulting blocks on the diagonal are triangular matrices. None of those blocks share the same entries, meaning they can all be computed in parallel. The blocks off-diagonal can further be simplified using a low-rank factorization. Note that the same factors apear in multiple blocks, meaning they only need to be calculated once and can then be reused, allowing further speed up.\nBy defining this state space duality (SSD), the authors were able to\n  * show the connection between SSMs and attention,\n  * reformulate the SSM using semiseparable matrix transformations which in turn allows the implementation of a faster algorithm that uses less memory and takes advantage of the GPU's dedicated matrix multiplication cores,\n  * design of the new Mamba-2 Block that also allows to apply modern parallelization techniques such as tensor parallelism or sequence parallelism,\n  * introduce various analog concepts of prominent multi-head patterns in Transformers to SSMs like for example the multi-head SSM (MHS).\n\n\nTo conclude, the SSD framework ends up having a smaller memory footprint as attention and the SSM and is 2x-8x faster as the SSM since it uses matrix multiplication in the core of its formulation.\n> _👉_ W**hat you will find in part 7:**\n> In Part 7 we will revisit why having semiseparable matrices improves speed and memory footprint. We will also see, that the low-rank factorization can be interpreted as a state space model and how we can leverage modern hardware accelerators. We will further learn about the Mamba-2 block, and how it enables prominent parallelization techniques. And of course, you'll learn about the multi-head patterns that can be achieved.\n* * *\n## 9. Conclusion\nMamba state space models are a promising alternative to Transformers and RNNs to build a new generation general purpose architecture that can be applied to any data modality.\nWithin a short amount of time, Mamba-like models have been successfully applied to many problems in deep learning.\nThe close connection to system theory, the advanced algebra involved and the fact that modern deep learning ecosystems are tailored towards the Transformer might be an obstacle for researchers to adopt this new family of models. But thanks to the hard work and dedication of some researchers, Mamba-like models are becoming increasingly faster and more efficient on modern hardware accelerators and its integration into standard deep learning layers makes it increasingly easier to include into your own project.\n### Thank You!\n> _🤝_ I'd like to t**hank you,** **t** he readers of my stories, for your constant support, clapping for my stories and, following me to never miss any of my latest articles. It is you who motivates me to continue writing, dive deep into complex topics, and become a better engineer/researcher by writing about them. You are awesome!\n### Continue Reading Part 2\n> [**Structured State Space Models Visually Explained**](https://towardsdatascience.com/structured-state-space-models-visually-explained-86cfe2757386)\n* * *\n## Further Readings & Resources\n> [**Mamba State Space Models for Images, Videos and Time Series**](https://medium.com/@SaschaKirch/list/861ae0ad08fb)\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n### Blog Posts\n  * [A Visual Guide to Mamba and State Space Models](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state)\n  * [The Annotated S4](https://srush.github.io/annotated-s4/#part-2-implementing-s4)\n  * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://hazyresearch.stanford.edu/blog/2020-12-05-hippo)\n  * [Parallel Prefix Sum (Scan) with CUDA](https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda)\n\n\n### Papers:\n  * [S4: Efficiently Modeling Long Sequences with Structured State Spaces](https://arxiv.org/abs/2111.00396) by A. Gu et.al., 31 Oct. 2021\n  * [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752) by A. Gu et.al., 1 Dec. 2023\n  * [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://arxiv.org/abs/2401.09417) by L. Zhu et.al., 17 Jan. 2024\n  * [VideoMamba: State Space Model for Efficient Video Understanding](https://arxiv.org/abs/2403.06977) by K. Li et.al., 11 Mar. 2024\n  * [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://arxiv.org/abs/2403.19888) by A. Behrouz et.al., 29 Mar. 2024\n  * [Mamba-2: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060) by A. Gu et.al., 31 May 2024\n  * [S5: Simplified State Space Layers for Sequence Modeling](https://arxiv.org/abs/2208.04933) by J. Smith et.al., 9 Aug. 2022\n  * [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://arxiv.org/abs/2008.07669) by A. Gu et.al., 17 Aug. 2020\n  * [LSSL: Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://arxiv.org/abs/2110.13985) by A. Gu et.al., 26 Oct. 2021\n  * [DSS: Diagonal State Spaces are as Effective as Structured State Spaces](https://arxiv.org/abs/2203.14343) by A. Gupta et.al., 27 Mar. 2022\n  * [S4D: On the Parameterization and Initialization of Diagonal State Space Models](https://arxiv.org/abs/2206.11893) by A. Gu et.al., 23 Jun. 2022\n  * [The Hidden Attention of Mamba Models](https://arxiv.org/abs/2403.01590) by A. Ali et.al., 3 Mar 2024\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657088944_hlxvw4.png)\nWritten By\n## [Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[See all from Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Mamba Image Video Signal](https://towardsdatascience.com/tag/mamba-image-video-signal/)[State Space Model](https://towardsdatascience.com/tag/state-space-model/)[Transformers](https://towardsdatascience.com/tag/transformers/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ftowards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a%2F&title=Towards%20Mamba%20State%20Space%20Models%20for%20Images%2C%20Videos%20and%20Time%20Series)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ftowards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a%2F&title=Towards%20Mamba%20State%20Space%20Models%20for%20Images%2C%20Videos%20and%20Time%20Series)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ftowards-mamba-state-space-models-for-images-videos-and-time-series-1e0bfdb5933a%2F&text=Towards%20Mamba%20State%20Space%20Models%20for%20Images%2C%20Videos%20and%20Time%20Series)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1nRabuQl3JXU7is31QF2FxA.png)](https://towardsdatascience.com/which-is-better-for-your-machine-learning-task-opencv-or-tensorflow-ed16403c5799/)\n## [Which is Better For Your Machine Learning Task, OpenCV or TensorFlow?](https://towardsdatascience.com/which-is-better-for-your-machine-learning-task-opencv-or-tensorflow-ed16403c5799/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nIs one of them really better than the other? An explanation from a ML engineer who uses both frameworks.\n[Nour Islam Mokhtari](https://towardsdatascience.com/author/nourislam/)\nAugust 1, 20215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/05AHX37QW2yEkbCLw.png)](https://towardsdatascience.com/from-patches-to-slides-how-to-train-deep-learning-models-on-gigapixel-images-with-weak-supervision-d2cd2081cfd7/)\n## [From Patches to Slides: How to Train Deep Learning Models on Gigapixel Images With Weak Supervision](https://towardsdatascience.com/from-patches-to-slides-how-to-train-deep-learning-models-on-gigapixel-images-with-weak-supervision-d2cd2081cfd7/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA review of techniques for modeling whole slide histology images and recommendations for different situations.\n[Heather Couture](https://towardsdatascience.com/author/hdcouture/)\nMarch 31, 202112 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/07/meeting_to_issues.png)](https://towardsdatascience.com/automating-ticket-creation-in-jira-with-the-openai-agents-sdk-a-step-by-step-guide/)\n## [Automating Ticket Creation in Jira With the OpenAI Agents SDK: A Step-by-Step Guide](https://towardsdatascience.com/automating-ticket-creation-in-jira-with-the-openai-agents-sdk-a-step-by-step-guide/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLearn how to create AI Agents using the OpenAI Agents SDK to automate Jira ticket creation from a meeting transcript.\n[Juan Carlos Suarez](https://towardsdatascience.com/author/juancarlos.suarez/)\nJuly 24, 202514 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/0UqIQ8neIfoDljJVo.jpg)](https://towardsdatascience.com/building-a-biomedical-entity-linker-with-llms-d385cb85c15a/)\n## [Building a Biomedical Entity Linker with LLMs](https://towardsdatascience.com/building-a-biomedical-entity-linker-with-llms-d385cb85c15a/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow can an LLM be applied effectively for biomedical entity linking?\n[Anand Subramanian](https://towardsdatascience.com/author/anand.subu10/)\nMarch 19, 202427 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/non-L-pendulum.png)](https://towardsdatascience.com/from-physics-to-probability-hamiltonian-mechanics-for-generative-modeling-and-mcmc/)\n## [From Physics to Probability: Hamiltonian Mechanics for Generative Modeling and MCMC](https://towardsdatascience.com/from-physics-to-probability-hamiltonian-mechanics-for-generative-modeling-and-mcmc/)\n[Math](https://towardsdatascience.com/category/math/)\nMathematics, Hamiltonian Monte Carlo, generative models, and research frontiers ✨\n[Soran Ghaderi](https://towardsdatascience.com/author/soran-ghaderi/)\nMarch 28, 202515 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/11/08dKIzIzw9BWaYcN7-scaled.jpg)](https://towardsdatascience.com/courage-to-learn-ml-demystifying-l1-l2-regularization-part-3-ee27cd4b557a/)\n## [Courage to Learn ML: Demystifying L1 & L2 Regularization (part 3)](https://towardsdatascience.com/courage-to-learn-ml-demystifying-l1-l2-regularization-part-3-ee27cd4b557a/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nWhy L0.5, L3, and L4 Regularizations Are Uncommon\n[Amy Ma](https://towardsdatascience.com/author/amyma101/)\nNovember 29, 20236 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2017/07/0YyZpqjCKkoKx-au9.jpg)](https://towardsdatascience.com/galaxy-zoo-classification-with-keras-219184aff581/)\n## [Galaxy Zoo classification with Keras](https://towardsdatascience.com/galaxy-zoo-classification-with-keras-219184aff581/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nThe full code for this is available on my Github at https://github.com/jameslawlor/kaggle_galaxy_zoo.\n[James](https://towardsdatascience.com/author/jameslawlor/)\nJuly 12, 20175 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1vpZVjrjzotvtlJliJIiy3A.png)](https://towardsdatascience.com/3-ways-to-leverage-apple-silicons-gpu-for-deep-learning-2cbb5b268b76/)\n## [3 ways to leverage Apple Silicon's GPU for Deep Learning](https://towardsdatascience.com/3-ways-to-leverage-apple-silicons-gpu-for-deep-learning-2cbb5b268b76/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nApple Silicon has brought impressive performance gains with great power efficiency. Can we use these chips for Deep Learning as well?\n[Jiří Moravčík](https://towardsdatascience.com/author/jiri.moravcik/)\nMarch 1, 20244 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/07/0A_j6HyHskjnrSJgr-scaled.jpg)](https://towardsdatascience.com/7-tips-to-choose-the-best-optimizer-47bb9c1219e/)\n## [7 tips to choose the best optimizer](https://towardsdatascience.com/7-tips-to-choose-the-best-optimizer-47bb9c1219e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nBased on my experience.\n[Davide Giordano](https://towardsdatascience.com/author/davidegiordano/)\nJuly 25, 20205 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1\n",
  "body_base_path": "attachments/66cae8cbf72f/pages/",
  "body_markdown_path": "attachments/66cae8cbf72f/pages/archived-body.md",
  "local_html": "attachments/66cae8cbf72f/pages/archived-body.html",
  "body_source": "prior_capture"
}