Alibaba's Aegaeon Pooling System Reduces Nvidia GPU Usage by 82%

Alibaba Cloud has introduced its Aegaeon pooling system, which reportedly reduces the number of Nvidia GPUs required for serving large language models (LLMs) by 82%. This claim emerged from a multi-month beta test conducted within Alibaba's Model Studio marketplace and was detailed in a peer-reviewed paper presented at the 2025 ACM Symposium on Operating Systems (SOSP) in Seoul. The Aegaeon system is designed to optimize inference-time scheduling, allowing multiple models to share GPU resources effectively. By virtualizing GPU access at the token level, Aegaeon can allocate compute resources dynamically, significantly increasing system-wide output, or "goodput," by as much as nine times compared to older serverless systems.

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192DigiTimes AsiaBenzinga

During the testing phase, the number of GPUs needed to support various LLMs, some with up to 72 billion parameters, decreased from 1,192 to just 213. The paper, authored by researchers from Peking University and Alibaba’s infrastructure division, highlighted two primary techniques for achieving these savings: packing multiple models per GPU and utilizing a token-level autoscaler.

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192

The results of Alibaba's Aegaeon system are likely to attract attention from other hyperscalers seeking to maximize their GPU resources amid rising inference demand. The ability to serve multiple models simultaneously could be particularly beneficial in constrained markets like China, where access to Nvidia's latest H20 GPUs is limited due to U.S. export controls.

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192

While the Aegaeon system shows promise, its effectiveness outside Alibaba's integrated environment remains uncertain. Critics may question whether similar results can be replicated by other cloud providers lacking Alibaba's specific infrastructure and optimizations.

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192

The paper does not specify the exact network fabric used during the beta test, leaving some ambiguity regarding the conditions under which the reported performance gains were achieved. Additionally, there is no detailed breakdown of which specific models contributed most to the GPU savings, raising questions about the generalizability of the findings.

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192

• “Instead of pinning one accelerator to one model, Aegaeon virtualizes GPU access at the token level, allowing it to schedule tiny slices of work across a shared pool.” — Jingren Zhou, CTO of Alibaba

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192

• “Alibaba says the gains came from two main techniques: Packing multiple models per GPU, and using a token-level autoscaler to dynamically allocate compute as output is generated, rather than reserving resources at the request level.” — Alibaba Cloud Paper

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192

• “Alibaba Cloud’s paper does not specify the exact network fabric used in the beta test, but we know the company offers its own eRDMA elastic RDMA network and has a record of building highly-integrated GPU serving stacks, suggesting the results may depend on an optimized, vertically integrated environment.” — Industry Analyst

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192

As demand for generative AI continues to grow, the cloud computing landscape may see increased competition among providers to adopt similar pooling systems. Alibaba's advancements could set a new standard for GPU utilization, prompting other companies to innovate in their cloud offerings.

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system— up to 9x increase in output lets 213 GPUs perform like 1,192