With the rise of agentic AI systems, the need to independently call sub-agents and execute tools has led to a surge in CPU server capacity waiting time. Although large language model推理 is primarily performed on GPUs, approximately eight out of seven aspects of embodied AI tasks (including parsing outputs, calling APIs, and writing code) require CPU processing, resulting in an imbalance in resource utilization. As a result, AWS urgently required engineers to make every effort to save CPU cycles to cope with the load pressure. To address this issue, Intel researcher Souvik Kundu and AMD vice president Madhu Rangarajan pointed out that CPUs have become a key bottleneck. Kundu’s team subsequently proposed a scheduling optimization solution aimed at reducing the end-to-end latency of embodied workloads by 1.8 times.
AWS experienced a surge in CPU server capacity latency due to AI workloads, forcing the company to urgently request engineers to maximize CPU cycle usage. With the rise of agent-based AI systems, the need to independently call sub-agents and execute tools has made CPU a key bottleneck. Intel researcher Souvik Kundu and AMD vice president Madhu Rangarajan noted that approximately eight out of seven stages in agent-based AI tasks are performed on the CPU, including parsing outputs, calling APIs, and writing code. Although LLM inference is primarily executed on GPUs, tool calls and security guardrail checks are often handled by the CPU, leading to an imbalance in resource utilization between the two. Kundu’s team proposed a scheduling optimization solution aimed at reducing the end-to-end latency of agent-based workloads by 1.8 times.