Small language model serving for search and recommendation
搜推场景小语言模型在线推理工程实践
English summary
Production SLM serving under 20 ms mean / 70 ms P99 (80 ms SLA): prefill-only MFU 67%, about 10x SGLang throughput; selective FP8 +24%, shared-prefix attention +20%; peak throughput is not SLA capacity (more concurrency gave a few % QPS for +30% latency).
Read the original on tech.meituan.com
Chinese·Web page·opens tech.meituan.com
- Tags
- Lane a, Inference cost