How and Why is the industry quietly switching away from Transformers?

[ayeshaaa0134]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

Models

Body

Everyone talks about how great Transformers are, but I'm noticing major labs and companies are actively moving to State Space Models, Mamba, and other architectures.

But I don't get it how were SSMs and Mamba actually made at first?

[Kristofferrrr]

SSMs are actually much older than Transformers. They come from control theory, where a system keeps a hidden "state" that is updated over time.

Mamba builds on modern SSMs and makes that state update depend on the input. This lets the model decide what information to keep or forget.

The main idea is:

Transformers: compare tokens with each other using attention.

Mamba/SSMs: continuously update a compact internal state.

This can make Mamba much more efficient for very long sequences, while still learning long-range dependencies.

[ayeshaaa0134]

hmmmm , but my concern is how do researchers even come up with something like this in the first place? how did someone come up with this to turn a continuous-time differential equation into a deep learning layer that replaces attention? and especially how do they even cmoe up with these equations and formulas in the first place , how do they even know that if this maths thing is causing problems we should repalce it with this maths how do they get that

[Kristofferrrr]

They usually don’t invent the math from nothing. They start with math that already exists and ask, “Can this help solve our problem?”

For SSMs, researchers took ideas from control theory, where equations describe how a system changes over time. Then they changed those equations so they could work with things like text.

The process is basically:

existing math → try it in a neural network → see what does not work well → change it → test again

Mamba was made in a similar way. Researchers saw that older SSMs were not good enough at deciding what information was important. So they changed the model so the input itself helps decide what to remember and what to forget.

So researchers usually do not know the perfect formula beforehand. They build on older ideas, experiment, measure the results, and improve the parts that cause problems.

[ayeshaaa0134]

hmm interesting

[Knyazovich]

don’t invent the math from nothing

[Michaelsjohnsons]

The AI industry is exploring alternatives to traditional Transformers because, while Transformers have been extremely successful, they can become expensive and computationally demanding as models and context windows grow.

One reason for this shift is efficiency. Transformer-based systems rely heavily on attention mechanisms, which can require substantial computing resources when processing long sequences. Researchers and companies are therefore investigating architectures that can handle long contexts with lower memory and computational costs.

Another factor is scalability. New approaches, including state-space models and hybrid architectures, aim to maintain strong performance while improving inference speed and efficiency. Rather than completely replacing Transformers overnight, the industry appears to be experimenting with different architectures for different workloads.

This transition is still developing, and Transformers remain dominant across many AI applications. The more realistic picture is an expanding ecosystem where new architectures compete with, complement, or combine with Transformers.

For readers interested in taking a break from technical discussions, pushyourluck.io can provide a quick change of pace.