The paper introduces llmmas-otel, a lightweight, framework-agnostic tool that integrates OpenTelemetry-based distributed tracing with fault injection for LLM-based multi-agent systems in software engineering. It instruments agent executions across workflow phases, agent steps, inter-agent communication, tool calls, and LLM invocations, and supports targeted fault injection at interaction points. This enables reproducible comparison of baseline and faulty executions through aligned traces and run artifacts. The tool is validated on a minimal demo workflow and a real LLM-based multi-agent system for software development, though specific results are not detailed in the provided text. The contribution is addressing the lack of structured observability and controlled failure testing in LLM-MAS, which are notoriously difficult to debug due to their distributed, stochastic nature.
Tracing and fault injection make multi-agent software workflows easier to inspect
The authors combine distributed tracing with targeted failure injection to compare normal and faulty agent runs.
Academic
Zahra Seyedghorban · Egor Klimov · Arie van Deursen · Annibale Panichella · Burcu Kulahcioglu Ozkan
Delft University of Technology · JetBrains Research
Research Digest··2 min read
Thread:Multi-Agent Coordination
In one line
llmmas-otel combines OpenTelemetry tracing with fault injection to inspect and test LLM-based multi-agent systems for software engineering.
Before you read this
Three ideas this paper builds onmulti-agent architectureA system where multiple AI agents collaborate, each handling specialized roles, communicating and coordinating to complete complex tasks.99 papers →runtime monitoringAudits an AI agent's live actions, internal states, or effects against constraints or policies to catch errors and attacks.107 papers →failure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.115 papers →
Read it as
The authors present llmmas-otel, a framework-agnostic tool for observing and deliberately disrupting LLM-based multi-agent software engineering systems.
The authors built llmmas-otel on OpenTelemetry, a standard for collecting distributed-system telemetry.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.