Memory Runs Out Before Compute, and Marvell Is Aiming at the Memory Wall
As artificial intelligence models grow larger and conversations grow longer, what runs short first is not computing power but memory capacity. That is exactly the point Marvell, a data infrastructure chip company, is aiming at.
The storage space AI needs to produce an answer is not only for the model itself. While a conversation is under way, space is also needed to hold the earlier parts of it, and the industry calls this the KV cache. The longer the context, meaning the length of input that can be handled at once, the larger this storage space grows in proportion.
When the high-speed memory attached to an accelerator chip cannot hold that space, the computing device spends time stopped and waiting. The chip’s computing power remains, but the answer comes late. The industry calls this the memory wall.
A representative measure is time to first token, the time from when a user sends a question until the first word appears on the screen. Marvell says the key to shortening that time lies less in computing speed than in memory capacity, bandwidth, and the width of the channel the data travels through.
The second-generation Structera X, which Marvell unveiled on the 12th to coincide with the OCP Global Summit, is a chip that widens that storage space beyond the server. Using CXL, an open interconnect standard that links processors and memory at high speed, it allows up to 6 terabytes of DDR5, the standard server memory, to be attached. Under the conventional approach, only memory plugged directly into the server could be used.
The new chip widened the paths memory travels to six channels, 50 percent more than the first generation. It also added a structure in which four servers share a single pool of memory. Because the amount of memory each server needs varies, letting another server borrow idle space reduces waste, the company says. It also says a feature that compresses data in real time produces the effect of using two to three times the actual capacity.
The performance figures should be read as the company’s claims. Marvell says that at the same context length, the time until the first token appears is 40 to 50 percent shorter. The figures have not been verified by an independent body. Sample shipments of the second-generation product are scheduled for the first quarter of next year.
Connectivity chips came with it. Racks packed tightly with accelerators are linked to each other by copper cables, and the longer the cable, the weaker the signal. Marvell plans to use connectivity chips made on a 3-nanometer process to keep signals intact over longer distances. Underlying the move is the judgment that the bottleneck is spreading from compute to memory, and from memory to the connections between servers.
Market interest has already moved to the products. The first-generation Structera X is shipping to large cloud companies, and Marvell’s presentation materials carried a supporting statement from a director of hardware engineering at Meta.
The stock’s reaction, though, is still cold. Marvell shares fell about 17 percent over five days last week amid the semiconductor sell-off. The industry view is that the problem the products target is clear, but valuations of chip companies will move only once the market’s doubts about the cost of data center investment are resolved.
