Learned Request Routing Improves Disaggregated LLM Serving Efficiency

VERA Brief

AI-generated. Grounded in the article and its cited sources.

A new study details a calibrated learned request router for disaggregated LLM serving. This system separates prefill and decode tasks onto distinct GPU pools to optimize request distribution and improve performance.

Key facts

• A calibrated learned request router separates prefill and decode operations onto distinct GPU pools.
• The router estimates completion time by considering prompt length, predicted output length, KV cache pressure, and SLO class.
• The calibrated router achieved a mean goodput of 0.864 across three mixed, bursty arrival traces, surpassing other tested methods.
• Hardware calibration was found to be critical for optimal performance.
• The benefits of the calibrated router increase with decode pool size and traffic heterogeneity.