The authors evaluate language models on air traffic control understanding using a framework that weights errors by operational consequence, not just semantic similarity. They built a diagnostic benchmark from aviation standards and input from 40 controllers across three countries, with two tasks: extracting structured operational fields and judging readback safety. Eight models were assessed under both conventional and consequence-aware metrics.
They find a systematic semantic-safety gap: conventional scores like F1 substantially overestimate operational reliability, including for models that look reliable under standard metrics. Fine-tuning with a risk-aware loss narrows the gap but does not close it.
This matters because it gives evidence that standard NLP evaluation can miss failures in high-stakes settings where confusing an altitude or dropping a condition is costly. It argues for as a complement to standard metrics before safety-critical deployment.