Many decision-making AI models are tested on well-known where prior exposure can inflate scores. Real-life decisions often involve long contexts, many options, and complex rules. Existing tests don't capture these together.
The authors built MMDM, a set of 5,448 questions across 16 task categories. Questions are new or rewritten to avoid memorization. They test single correct answers, sets of valid options, and probability distributions, with up to 1,024 options and 64,817 tokens.
Models that do well on world knowledge often fail on logic, constraints, and selecting all valid options. The best model achieved only 36.81% accuracy on multi-select tasks. This shows current models lack joint reasoning and constraint awareness, crucial for real agent workflows.