Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions

The research team conducted stress tests on the benchmarks for AI, finding that reducing computational costs significantly affects the stability of model behavior conclusions. The study was conducted on the BBQ and BBQ-V datasets, evaluating three types of dense and hybrid expert models under seven conditions including batch processing, quantization, benchmark reduction, and their combinations, and comparing them with the full-bench BF16 baseline. The results showed that increasing batch size could maintain accuracy within 0.35 percentage points of the baseline level, while reducing energy consumption for six out of five model-data set settings; INT8 quantization significantly saved energy (1.79–4.26 times the baseline), but INT4 led to larger and more dependent on model and context conclusions; benchmark reduction provided consistent savings, but the choice of the subset of retained items was extremely sensitive. Therefore, efficient evaluation should be considered a measurement intervention that must verify its effectiveness.