RTX 4060 running the 35B model, 39 Tokens per second? Berkeley and MIT have joined hands to open-source FreeToken

The University of California, Berkeley, and MIT have jointly opened-source a project called FreeToken. This project aims to enable consumer RTX 4060 graphics cards to efficiently run large language models with 35B parameters by optimizing quantitative algorithms and memory management techniques, achieving an inference speed of 39 Tokens per second. FreeToken plans to lower the hardware requirements for large models on consumer devices and promote the widespread use of AI models; the related code is now available for developers to use.