Cloud Native Computing Foundation
·
Why goodput matters more than throughput for LLM serving
原文英文，约2000词，阅读约需8分钟。
📝
内容提要
When we benchmark an LLM serving setup, the number almost everyone reaches for first is throughput: how many requests per second the system can push through. It is easy to measure, easy to...
🏷️
