publications

2026

1. ATC

   Chorus: GPU-CPU Cooperative LLM Serving for Length-Skewed Decode

   Dong Chen^*, Yukun Zhou^*, Sheng Yao^*, Ruichuan Chen, Binhang Yuan, Shuai Wang, Luping Wang, Cheng Huang, Guodong Yang, Liping Zhang, and Wei Wang

   in the Proceedings of 2026 ACM SIGOPS Annual Technical Conference, Nov 2026
2. arxiv

   ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform

   Xiaoxiao Jiang^*, Suyi Li^*, Sheng Yao, Tianyu Feng, Lingyun Yang, Dapeng Nie, Haoran Yang, and Wei Wang

   Jul 2026

   Abs arXiv PDF

   Text-to-image (T2I) workflows are increasingly deployed on serverless platforms because users often compose customized workflows and invoke them intermittently. Existing platforms typically deploy each workflow as an opaque GPU function, provisioning, placing, and scaling all constituent models in the workflow together. This monolithic design obscures workflow structure, inflates scaling overhead, forces users to manage low-level GPU coordination, and limits fine-grained fairness in multi-tenant clusters. In this paper, we present ServerlessT2I, a serverless-native system that decomposes a T2I workflow into loosely coupled model functions that can be independently managed and scheduled. By explicitly managing individual model execution, ServerlessT2I enables per-model scaling, declarative workflow composition, transparent GPU-resident communication, and fairness-aware scheduling. To make this decomposition efficient, ServerlessT2I harvests slack GPU memory left idle by compute-bound T2I inference to build a data plane that reduces model loading and data communication overheads. \sys further introduces a fair scheduler for multi-tenant serving. Using production traces, ServerlessT2I sustains up to 2× higher request rates than existing T2I workflow serving systems with the same GPU budget; for a fixed request rate, it saves up to 3× GPU resources while satisfying service level objectives (SLOs).

3. arxiv

   From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

   Chaokun Chang^*, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, and Wei Wang

   Aug 2026

   Abs arXiv PDF

   Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state. However, the system behavior of these workloads—where latency, cost, and bottlenecks arise—remains poorly characterized, leaving serving systems to rely on assumptions built for conventional inference. We present AgentSysBench, a benchmark suite and measurement toolkit with ten representative agentic applications and unified systems-level instrumentation. Across controlled deployments and production traces, we identify six properties that distinguish agentic workloads from conventional LLM serving: (1) execution is heavyweight and stateful, with non-LLM components dominating latency in 5 of 10 applications and sandbox working-set memory peaking at 28 GB per session; (2) applications compose components with heterogeneous resource affinity—GPU-bound inference, memory-bound retrieval, CPU-bound sandboxes—whose task latencies diverge by up to 32x; (3) bottlenecks shift across requests, models, and deployments; (4) production sessions hold state idle for minutes to hours between active steps; (5) a control-plane tax—auxiliary LLM calls and context overhead from tool schemas and observations—crowds out productive compute and context; and (6) production traces from three applications reveal heavy cross-request redundancy in search queries and web fetches, exposing a large caching opportunity. Four design explorations demonstrate that these findings are actionable: task-aware serving reduces latency by 29–40%, communication-aware placement by up to 4.5x, state offloading reduces memory usage by 4.6x, and tool-result caching removes 35.2% of redundant search calls and saves 19.3% of aggregate search latency.

4. OSDI

   Heterogeneity at Hyperscale: Characterization and Scheduling of Large Production AI Clusters

   Suyi Li^*, Lingyun Yang^*, Haoxuan Yu, Sheng Yao, Tianyuan Wu, Xiaoxiao Jiang, Hanfeng Lu, Kangjin Wang, Chenhao Wang, Shenglin Xu, Lun Wang, Qingyang Duan, Shenghao Liang, Xiu Lin, Meng Zhang, Wenchao Wu, Yinghao Yu, Guodong Yang, Liping Zhang, and Wei Wang

   in the Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation, Jul 2026

   Abs PDF

   The rapid scaling of generative AI (GenAI), alongside the continued reliance on classical deep neural networks (DNNs), has pushed production AI infrastructure toward massive, heterogeneous GPU fleets. We present a comprehensive characterization of Alibaba Serverless Infrastructure (ASI), a hyperscale production cluster, based on a six-month trace covering 155,410 GPUs of multiple vendors and generations and jobs from 81 departments, spanning ad-hoc development, training, and online and offline inference. Our central finding is that high GPU demand does not yield high effective utilization: idle GPUs frequently become unallocatable because free capacity is stranded across nodes, lacks matching CPUs, or violates network-locality constraints, and because users reserve ample headroom for production safety. Notably, fractional-GPU fragmentation, a focus of prior work, is now negligible, as GPU sharing is rarely used. We detail deployed solutions that recover this capacity: a practical GPU defragmentation algorithm that cuts the number of nodes with slack resources by 20.2%, and SpotGPU, a preemption-cost-aware scheduling framework that safely harvests idle resources and raises the GPU allocation ratio from 68% to 93%. We further surface open challenges in skewed multi-vendor GPU adoption, bandwidth bottlenecks between heterogeneous GPUs, and interference among colocated workloads. We release the ASI trace, the most comprehensive to date in workload diversity and cluster scale, to support future research.

5. MLSys

   FaaScale: Unlocking Fast LLM Scaling for Serverless Inference

   Minchen Yu^*, Rui Yang^*, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Yue Cheng, Wei Wang, Ao Wang, and Ruichuan Chen

   in the Proceedings of the 9th Annual Conference on Machine Learning and Systems, May 2026

   Abs arXiv PDF

   Serverless computing is an attractive paradigm for cloud-based large language model (LLM) inference, but scaling LLMs on demand remains a major challenge due to high data transfer cost. We present FaaScale, a serverless LLM system that enables fast and resource-efficient model scaling. The key idea is a co-design principle—pipelined multicast inference—which synergizes multicast with dynamic, cross-node pipeline-parallel execution during model transfer. FaaScale implements this design through PipeCast, a model scaling scheme that adaptively multicasts model blocks and dynamically forms inference pipelines on the fly. Coupled with efficient memory management across GPU and host memory, FaaScale handles bursty LLM inference workloads effectively, achieving up to 5× lower tail time-to-first-token latency and 31.3% cost reduction on real-world LLM traces.

2025

1. SoCC

   ZipBatch: Multi-Tenant GPU Batching with Dual-Resource Regulation

   Haoxuan Yu, Sheng Yao, and Wei Wang

   in the Proceedings of the 2025 ACM Symposium on Cloud Computing, Nov 2025

   Abs DOI PDF

   GPU multiplexing is a widely-adopted strategy in GPU clusters for improving overall throughput and lowering the total cost of ownership. To mitigate inter-task interference in compute power and memory bandwidth on multiplexed GPUs, existing techniques divide a GPU into instances with limited predefined rigid configurations. Low utilization arises from the mismatch between heterogeneous burstiness and immutable resource configurations: 1) bursty inference traffic forces the scheduler to launch underfilled batches that cannot saturate the instance; 2) bursty kernel resource utilization leads to bubbles in compute power and memory bandwidth. This paper presents ZipBatch, a multi-tenant model serving system with elastic fine-grained GPU resource provisioning. First, ZipBatch launches dynamic batches with right-sized resource reservations to reduce macroscopic model-level over-provisioning while decreasing queuing time. Second, by observing that kernels exhibit imbalanced and inconsistent utilization of resources, we develop a prediction-based dual-dimensional regulator that allocates right-sized compute power and memory bandwidth in real time to suppress microscopic resource underutilization. Compared with state-of-the-art solutions, our experimental results show that ZipBatch achieves up to 60.84% reduction in average latency while increasing total throughput by 71.27%.

2024

1. SIGMOD

   WeBridge: Synthesizing Stored Procedures for Large-Scale Real-World Web Applications

   Gansen Hu, Zhaoguo Wang, Chuzhe Tang, Jiahuan Shen, Zhiyuan Dong, Sheng Yao, and Haibo Chen

   in the Proceedings of the 2024 ACM SIGMOD International Conference on Management of Data, Jun 2024

   Abs DOI PDF

   Modern web applications use databases to store their data. When processing user requests, these applications retrieve and store data in the database server, which incurs network round trips. These round trips significantly increase the application’s latency. Previous approaches have attempted to reduce these round trips by prefetching query results or batching database accesses. However, neither method can efficiently reduce the latency when some queries depend on previous queries’ results. In real-world applications, nearly 50% of the queries depend on the result of other queries.This paper presents WeBridge, the first system capable of synthesizing stored procedures for large-scale real-world web applications. First, WeBridge employs concolic execution technique to analyze the applications and generate stored procedures for hot program paths. Then, it seamlessly integrates the stored procedures into the application by extending the database access library. Finally, it improves the efficiency of the stored procedures with speculative execution. Evaluation using real-world web applications and workloads show that WeBridge achieves up to 79.8% median latency reduction and up to 2\texttimes peak throughput.