EVENT DOSSIER
The HarvestBench research paper has been published, showing significant differences among nine models in animal avoidance tasks.
HarvestBench is the first benchmark that aims to avoid harming living animals in pricing decisions, proposed by Jasmine Brazilek and others. The study simulates the decision-making behavior of LLM agents driving tractors during corn harvesting in farmland, covering 7,201 pricing decisions across nine models. Experiments revealed significant differences in the killing rates of various models, ranging from 0.4% to 98.8%. Terra and Sol were the most humane, while GPT-4o-mini was the most cruel, and its performance had nothing to do with the model’s capabilities. Under default settings, all models more frequently crushed wild animals than livestock. After introducing a moral briefing, the killing rate of five-sixths of the reasoning models dropped below 6%; removing this briefing increased the killing rate to over 84%. Additionally, three-quarters of the models were sensitive to prices at 5%. The benchmark uses game log counts for scoring, without the need for an LLM evaluator to ensure reproducibility.
arXiv:2609.04444v1 HarvestBench 首个将动物伤害定价的 LLM 代理基准
The HarvestBench research paper has been published, showing significant differences among nine models in animal avoidance tasks.
arXiv cs.AI and Hugging Face papers published HarvestBench benchmark results, simulating LLM agents harvesting corn while facing animal decisions. It covers 7,201 pricing decisions across nine models, with bacterial killing rates ranging from 0.4% to 98.8%.
2 reportsHarvestBench 是首个将避免伤害活体动物作为定价目标的基准测试。该研究由 Jasmine Brazilek 等人提出，通过模拟 LLM 代理驾驶拖拉机在农田中收割玉米的场景，要求模型在遇到动物阻挡时选择付费绕行或继续行驶。实验涵盖九个模型和 7,201 次决策，发现杀生率从 0.4% 至 98.8% 不等，其中 Terra 和 Sol 最为仁慈，GPT-4o-mini 最为残忍，且表现与能力无关。在道德简报下，五分之六的推理模型杀生率低于 6%，移除该简报后则升至 84% 以上。
HarvestBench is the first benchmark that aims to avoid harming living animals in its pricing criteria. It measures the moral performance of models by simulating their decision-making behavior when LLM agents drive tractors to harvest corn. The study ran nine models through 7,201 pricing decisions in a reinforcement learning environment with animals, and found that the rate of killing animals ranged from 0.4% to 98.8%. Terra and Sol were the most benevolent, while GPT-4o-mini was the most cruel. The results showed that all models more frequently crushed wild animals rather than domestic livestock under the default map; after introducing a moral briefing, the killing rate of five out of six inference models dropped below 6%, but removing the briefing caused the killing rate to soar above 84%. Additionally, three-quarters of the models were sensitive to a price of 5%, and the benchmark uses game log counts for scoring, without the need for an LLM evaluator to ensure reproducibility.