Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

A recent study on large language model services confirms that quantitative operations significantly exacerbate the issue of non-reproducibility of execution results caused by caching mechanisms. The researchers compared the workload of using proxy tools under two engines and four weight formats, with fixed models, decoding parameters, random seeds, and request order. The results showed that with caching enabled, approximately 36.2% of rounds had changed proxy trajectories at 16-bit precision, while this proportion increased to 75.0% at 4-bit precision; in contrast, when caching was disabled, all execution results were identical. The study identified three key reasons: a single server-level cache setting for prompts expanded the difference between runs by 37.5 percentage points, and it only took effect when the execution order setting was activated; moreover, after resuming caching, while both the cached path and the recalculation path were reproducible, their results were different from each other. The study pointed out that although the service is deterministic under a given caching state, since the state is not included in the requests and is not reset by default, actual runs become non-reproducible. Additionally, single-round tests showed that this divergence affected the results of specific tasks…