Financial systems vary by country, but existing mostly test U.S. and English-language knowledge, leaving Japanese finance, with its own tax rules and regulations, underevaluated. (LLMs) are computer programs that answer questions, and they need proper tests to see if they can handle real financial tasks in Japan.
The authors built a new test called JFinBench by taking 321 questions from six Japanese certification exams from 2026. They used a system where a computer program writes code to convert exam PDFs into test questions, and a human checks every one, making it easier to update the test with new exams each year.
They tested 12 recent LLMs and found the best proprietary models scored above 93%, while the top open-weight model (a freely available type) trailed by 12 percentage points. Most errors came from missing knowledge, not calculation mistakes. Giving models web access narrowed but did not close the gap.
