x
In the hyper-competitive race to scale production artificial intelligence, enterprise technology leaders have largely operated under a single, expensive rule of thumb: when AI performance slows down or capacity runs out, throw more Graphics Processing Units (GPUs) at the problem.
However, GPU memory remains the single most expensive and constrained resource in modern data centres. As large language models (LLMs) evolve to handle extended context windows and complex multi-turn conversational workflows, a glaring inefficiency has emerged: AI systems spend vast amounts of expensive compute power constantly re-processing information they have already calculated.
Data platform specialist Weka is challenging this brute-force hardware approach. With the launch of its NeuralMesh 6 software platform and Wekapod 3 hardware line, Weka introduces a technique called the Augmented Memory Grid. By leveraging high-speed flash storage to behave as a seamless extension of GPU memory, the platform can cache 100 per cent of an AI model's pre-calculated tokens—effectively removing the need to repeatedly re-compute contextual data.
To understand why traditional GPU scaling hits a financial wall, it helps to examine how modern LLMs process queries. Every interactive prompt involves two distinct mathematical phases:
In ongoing interactions—such as software engineering assistants, enterprise copilots, or customer service agents—each new user input re-triggers the prefill stage for every previous turn in the conversation, unless that work is preserved in memory (a process known as Key-Value or KV caching).
Without efficient off-GPU caching, the computational waste grows exponentially. As Weka co-founder and CEO Liran Zvibel points out, a conversation lasting 10 turns can lead to over-calculating previous context up to 100 times. Extend that to a 20-turn session, and the model may re-run the exact same calculations up to 400 times.
By aggregating ultra-fast NAND flash storage into an Augmented Memory Grid, Weka enables organisations to cache 100 per cent of these pre-calculated tokens off the GPU. This eliminates redundant prefill overhead and liberates precious GPU VRAM to serve additional users and generate new responses.
Beyond token caching, NeuralMesh 6 addresses several operational and architectural hurdles faced by modern AI cloud providers and enterprise IT teams.
Managing multi-tenant AI environments traditionally requires choosing between strict physical isolation and flexible cloud sharing. NeuralMesh 6 bridges this gap by offering:
Combined, a master environment managing 50 composable clusters can scale to support up to 50,000 tenants, with tenant provisioning completed in under 30 minutes.
AI pipelines typically rely on two distinct storage formats: file-based paths for data ingestion, training, and fine-tuning, alongside object-based paths (such as Amazon S3) for inference and cloud-native tools.
Traditionally, bridging these formats requires a translation gateway that duplicates data across disk. Weka eliminates this bottleneck by allowing the exact same physical data on disk to be read natively through file or object protocols simultaneously—with no translation layers or duplicate copies required. For non-AWS specialised GPU clouds (such as CoreWeave, Nebius, G42, and Lambda), this architecture delivers significantly higher performance while replacing per-API charges with transparent, capacity-based pricing.
When expanding workloads across distributed regions or acquiring new GPU allocations, data transfer delays can stall operations for days or even weeks. NeuralMesh 6 solves this through metadata-first replication: destination storage environments become instantly browsable, with full data hydrating dynamically on demand as applications request access. This reduces cluster setup times from weeks to less than an hour.
Storage clusters often force a compromise between expensive, high-performance Triple-Level Cell (TLC) flash and cheaper, slower Quad-Level Cell (QLC) flash. Weka’s AlloyFlash architecture mixes both within a single cluster, automatically directing latency-sensitive operations to TLC while placing bulk capacity onto QLC. Coupled with always-on data reduction guarantees, organisations can lower their cost per terabyte without sacrificing speed on critical inference workloads.
Over the past two years, legacy enterprise storage vendors—including Dell, NetApp, and Pure Storage—have actively repositioned their product portfolios around AI workloads. However, industry analysts note a distinct difference between adapted legacy storage and architectures engineered specifically for AI from the ground up.
Steve McDowell, chief analyst at NAND Research, highlights that while traditional storage vendors are adapting to manage data at AI speeds, native providers like Weka and VAST Data were built specifically for this paradigm. McDowell points to Weka's Augmented Memory Grid as a standout technical lead, calling it one of the most capable KV cache implementations available today. In a market constrained by GPU availability and sky-high memory costs, saving GPU cycles directly translates into substantial financial savings.
McDowell also advises enterprise technology buyers to look past marketing claims by evaluating real-world deployments and requesting contractual guarantees on data reduction capabilities before committing capital.
While smaller organisations running basic AI implementations may not immediately hit GPU memory limits, storage-assisted token caching becomes essential for enterprise environments operating AI at scale. Key use cases include:
By extending GPU memory into intelligent flash storage, enterprise organisations can dramatically improve the return on investment of their existing AI infrastructure—scaling capacity and throughput without waiting months for additional GPU deliveries.
For more information and the original source report, read the full article on VentureBeat:
Disclaimer: This article is provided for informational purposes only, mistakes may be made, and it's not offered or intended to be used as legal, tax, investment, financial, or any other advice.
