Measuring LLM performance drift: observations and methodology from 31,352 repeated benchmark measurements [D]
A study on the performance drift of large language models analyzed 31,352 observations from repeated benchmark tests. The study transformed benchmark tests from ranking-based problems into longitudinal measurement problems, aiming to distinguish between changes in model behavior and fluctuations in infrastructure. Data showed that the standard deviation of scores within the same day was 2.80 points, while the standard deviation of daily medians across days was 8.43 points, with a ratio of approximately 3:1. The research team used methods such as versioned benchmark configurations, repeated evaluations, and change detection, and published a public method paper containing a measurement design but hiding specific task sets. The authors invited peers to provide criticism and suggestions on issues such as the choice of time series units for longitudinal evaluations, the distinction between drift and infrastructure effects, benchmark contamination control, and methods for detecting non-stationary data.