Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory

A study based on knowledge space theory evaluated the performance of eight open-source and closed-source large language models in mathematical reasoning. The study found that these models do not follow human knowledge structures, frequently violate knowledge dependencies, and fail to effectively utilize the relevant knowledge provided by context to improve their ability to solve dependent problems. Additionally, there is a lack of consistency in the knowledge structures among different models, and the overlap in their knowledge distributions is low. It is worth noting that these structural flaws are almost invisible in evaluations based on accuracy and using “large language models as judges”. The study conclusions indicate that the knowledge structures of current large language models do not follow human-like patterns.