For years, the golden rule of local AI has been simple: if the model is bigger than your RAM, you're out of luck. You either buy a workstation with obscene amounts of VRAM or pay a cloud provider to rent a GPU. But a new wave of architectural experiments is turning that logic on its head, proving that you can run massive models on hardware that—on paper—shouldn't be able to handle them.
The Magic of Mixture-of-Experts
The secret sauce here is the Mixture-of-Experts (MoE) architecture. Unlike traditional "dense" models where every parameter fires for every word, MoE models only activate a small fraction of their "experts" per token. Developers like Andrey Mikhaylov are leveraging this by keeping only the essential model core and KV cache in memory, while streaming the idle experts from the SSD on demand.
This approach allows a 26B parameter model to run on a base MacBook Air with just 8GB of RAM, hitting surprising speeds of 5-6 tokens per second. By treating the NVMe SSD as a high-speed extension of memory, the hardware bottleneck shifts from capacity to disk throughput.
Pushing the Limits: From MacBooks to 744B Models
This isn't just a niche trick for small models. We're seeing research projects like Flash-MoE running 397B parameter models on 48GB of unified memory. Even more ambitious is the Colibri-inspired approach, which claims to run the massive GLM-5.2 (744B parameters) on consumer machines with roughly 25GB of RAM using pure C and zero dependencies.
While traditional swap space is notoriously slow, these optimized streaming pipelines are designed specifically for the way LLMs access data, making the "SSD-as-RAM" dream a practical reality for local inference.
The Future of Local AI
We are moving toward a world where the size of your GPU no longer dictates the intelligence of your local AI. As SSD speeds continue to climb and streaming optimizations improve, the gap between cloud-scale models and home-office hardware will continue to shrink, putting truly massive, private intelligence in everyone's pocket.
Sources
Media
