Cross-stack Design for Sustainable AI Infrastructure

Title: Cross-stack Design for Sustainable AI Infrastructure

Abstract: The rapid increase in LLM ubiquity and scale levies unprecedented demands on computing infrastructure. These demands not only incur large compute and memory resources, but also significant energy, yielding large operational and embodied carbon emissions. In this work, we present three main observations based on modeling and traces from production deployment of two Generative AI services in a major cloud service provider. First, while GPUs dominate operational carbon, host processing systems (e.g., CPUs, memory, storage) dominate embodied carbon. Second, offline, batch inference accounts for a significant portion (up to 55%) of serving capacity. Third, there are different levels of heterogeneity across hardware and workloads for LLM inference. Based on these observations, we design EcoServe, a carbon-aware resource provision and scheduling framework for LLM serving systems.
It is based on four principles - Reduce, Reuse, Rightsize, and Recycle (4R). With a cross stack ILP formulation and design, we demonstrate that EcoServe can lower carbon emissions by up to 47%, compared to performance, energy, and cost-optimized design points, while maintaining performance targets and SLOs. 

Bio:  Yueying Lisa Li is a PhD candidate at Cornell Computer Science department, specializing in sustainable AI system and data center scheduling design. She was fortunate to be mentored by Edward Suh, Christina Delimitrou and Udit Gupta. Before and during her PhD, she had a wide range of industry experience, and conducted research at Microsoft Research, Intel Labs, and Apple, focusing on improving the efficiency of various system stacks with learning and optimization methods, shipping to production systems like Azure and M1 chips. At Cornell and MIT, Lisa’s research focuses on efficient reinforcement learning (RL) and sustainable ML systems. Her career goal is to advance ML model/system co-design methodologies in data center computing and improve the sustainability of design and operating AI workloads.  
Lisa is active in the research community, she serves on the Artifact Evaluation Committees of MLSys, SOSP, and ASPLOS, reviewing for ICLR and CVPR, and co-organizing MICRO workshops. A co-founder of CALM initiatives, she is a steering member of the Computer Architecture Student Association (CASA) and co-founded the  Women in System Podcast to bridge system and ML communities.