SharedSAE: One Feature Dictionary Across Language Models
2026-09-07 12:00Science🔥 42.2 heat score
1sources
1days unfolding
42.2heat score
1mentions
SummaryAI generated
The researchers proposed the SharedSAE method, aimed at addressing the inefficiency of training sparse autoencoders (SAEs) separately for each language model. This method utilizes a shared feature dictionary combined with a design of model-specific encoding and decoding mechanisms, and was validated on 1 billion-parameter base language models from four different families and word splitters. Experiments showed that SharedSAE retained 96.6% of the average explained variance of the proprietary SAE, its potential activation had 1.8 times higher cross-model correlation compared to SAEs aligned separately, and its potential descriptions possessed cross-model transfer capabilities. With the dictionary frozen, the new model could efficiently adapt and achieve reconstruction quality close to that of the proprietary SAE.
The researchers proposed the SharedSAE method, which uses a shared dictionary and model-specific codecs to replace the traditional approach of training SAEs separately for each language model. This method was trained on 1B-scale base language models from four different families and tokenizers. Experiments showed that SharedSAE retained 96.6% of the average variance explained by the proprietary SAE, its potential activation had 1.8 times higher cross-model correlation compared to SAEs trained separately, and its potential descriptions could be transferred between models. After freezing the dictionary, the new model could be efficiently adapted and achieve reconstruction quality close to that of the proprietary SAE.