PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Prompts

The researchers proposed the Prompt2Box method, which uses a trained encoder to embed prompts into a box-shaped space to capture semantic similarity and specificity relationships. This method optimizes visualization and comparison effects through dimensionality reduction techniques. Experiments show that Prompt2Box predicts a 45% lower error rate in specificity compared to the length-based baseline. When hierarchical clustering was performed on 17 large language models using the UltraFeedback dataset, Prompt2Box identified 13.5% more weaknesses than the vector-based baseline, and the correlation between hierarchical depth and instruction specificity increased by approximately 33%. The related code has been made open source.