Why AI is hitting a wall—and how startups hope to fix it

Almost all modern AI is built on a design called a transformer. Invented in 2017, this architecture is essentially a high-speed processor that looks at every word in a sentence simultaneously to figure out how they relate. However, as we feed these models longer books, entire codebases, or hours of video, the amount of math required grows exponentially. Startups like Subquadratic and Manifest AI are now testing new, experimental ways to rewrite this underlying engine, while others like Liquid AI are designing smaller, more flexible models that can run on tiny, low-power devices instead of massive data centers.

Transformers rely on something called dense attention. Imagine you are trying to understand a long contract. Dense attention forces the AI to look at every single word in that document and compare it to every other word, every time it tries to make sense of a new sentence. It creates a complete, perfect map of the text, but the math involved is staggering. If you double the length of the document, the effort required doesn't just double—it balloons because of all those new word-to-word connections. Some researchers are now testing sparse attention, which lets the AI ignore irrelevant words and focus only on the important ones, similar to how a human skims a page for key facts. Others, like Manifest AI, use a rolling summary approach that keeps only the most important context, discarding old information as it moves forward. This is akin to a student taking notes rather than trying to memorize every syllable of a lecture.

The current approach is hitting a physical limit where the energy required to make these systems smarter is becoming a major hurdle. If we want AI to act as a personal assistant that remembers everything you have ever typed or helps code software for weeks without crashing, it cannot rely on a design that tries to hold the entire world in its memory at once. Moving toward more efficient architectures means we could eventually run capable AI on a laptop or a car dashboard rather than waiting for a massive server in a far-away building to process a request. These startups are essentially trying to build a more efficient brain so that the next generation of AI can be both more capable and significantly more practical for everyday use.