Docker adds MicroVM Sandboxes to Harbor for reproducible AI agent evaluations

New integration lets each evaluation task run in its own isolated environment, addressing reproducibility and security concerns in AI agent testing.

By LineZotpaper

Published22 October 2026

Read Time2 min

Docker has announced that Harbor, the open-source framework for evaluating AI agents, can now run evaluation tasks in isolated microVMs through Docker Sandboxes. The integration aims to improve reproducibility and security by giving each task its own kernel-level sandbox, allowing agents to install packages and run commands without risking interference between tasks or access to the host system.

Docker's new integration between Harbor, the open-source agent evaluation framework, and Docker Sandboxes provides each evaluation task with its own lightweight virtual machine, or microVM, instead of sharing a container. Docker says this approach isolates agent activity, preventing a task from reaching the host, other tasks, or external networks.

The need for such isolation is driven by the nature of coding-agent benchmarks, which can involve thousands of repeated tests where an agent is allowed to install packages, edit files, and run commands. Docker's engineers describe the process as running untrusted, AI-written code at scale and highlight reward hacking as a specific concern, where an agent might tamper with test files or overwrite scoring records rather than solving the intended task.

Harbor orchestrates the evaluations, managing task execution, score collection, and logging, and supports benchmarks such as Terminal-Bench. With Docker Sandboxes, each task runs in a microVM with its own kernel, providing stronger separation than shared containers. Configurable network policies in Docker Cloud Sandboxes allow dependencies to be pulled during setup and then restrict internet access while the agent works, a measure Docker says helps ensure reproducible results across runs.

Docker Cloud Sandboxes also support Docker Compose, letting multi-container tasks run inside a single microVM. Docker has tested the setup with up to 64 concurrent tasks, though higher limits are available on request. The announcement does not include data on startup overhead, per-task cost, or evaluation-score variance; Docker says teams sensitive to latency or expense will need to measure those themselves.

The integration is available in the nightly release of Docker Sandboxes and is expected to reach stable release on October 23. Harbor users can enable the sandbox backend by installing the appropriate extra package and using the -e sbx flag. Local runs are free, while cloud scaling depends on Docker's Cloud Sandboxes offering. Docker's announcement also notes that the Harbor integration adds to a growing list of sandbox backends for the framework, including Daytona, Modal, E2B, Runloop, and others.

Analysis

Why This Matters

Reliable agent evaluation is critical for AI development; inconsistent environments can produce misleading benchmark results.
Isolated microVMs reduce the risk of reward hacking and cross-task contamination, potentially raising trust in evaluation outcomes.
The integration makes cloud-scale agent testing more accessible, but teams must assess the cost and performance tradeoffs for their own workloads.

Background

AI agent evaluation has become an active area in development tooling, with frameworks like Harbor providing orchestration and benchmark management. A persistent challenge is ensuring that evaluations are both secure and reproducible. Traditional container isolation can be insufficient when agents have privileges to modify files and install software; microVMs offer stronger boundaries. Docker has been extending its Sandboxes product to support agent runtimes, including earlier integrations with GitHub Agentic Workflows and NanoCo's NanoClaw platform.

Key Perspectives

Docker: The new integration provides a secure, scalable execution environment for agent evaluations, with configurable network policies and support for multi-container tasks. Developers and AI teams: The promise of reproducibility and isolation is valuable, but the lack of published performance benchmarks means adopters must invest time in testing latency and cost for their specific use cases. Critics/Skeptics: The integration does not prevent all forms of reward hacking; an agent that can write to test files may still tamper with results. Additionally, cloud sandboxes require a Docker Cloud account and access token, adding overhead for some teams.

What to Watch

Adoption of the local vs. cloud sandbox option as teams weigh cost, isolation needs, and scale.
Publication of independent benchmarks comparing microVM-based evaluations with other sandbox backends.
Expansion of Docker Sandboxes to broader agent-runner ecosystems beyond Harbor.

Sources

Harbor Adds Docker Sandboxes to Run Reproducible AI Agent Evals in Isolated MicroVMs · InfoQ

Checked

12 claims in this story checked against 1 source: 11 supported.

Show each claim

SupportedDocker adds MicroVM Sandboxes to Harbor for reproducible AI agent evaluations.InfoQ: “Harbor Adds Docker Sandboxes to Run Reproducible AI Agent Evals in Isolated MicroVMs.”
SupportedThe integration lets each evaluation task run in its own isolated environment.InfoQ: “This change allows each evaluation task to run in its own microVM.”
SupportedThe integration addresses reproducibility and security concerns in AI agent testing.InfoQ: “In Docker Cloud Sandboxes, configurable network policies let a task pull dependencies during setup, then restrict internet access while the agent works. Every agent is then evaluated under the same network conditions, which addresses a common source of non-reproducible results.”
SupportedDocker announced that Harbor now works with Docker Sandboxes.InfoQ: “Docker announced that Harbor, the open-source agent evaluation framework, now works with Docker Sandboxes.”
SupportedEach task runs in a microVM with its own kernel, keeping agent activity from reaching the host or other tasks.InfoQ: “Each task runs in a Docker Sandbox, a microVM with its own kernel, so agent activity stays inside the VM and cannot reach the host, other tasks, or anything outside the environment.”
Partly supportedThe integration prevents tasks from reaching external networks.InfoQ: “The local integration currently supports only public and no-network modes, subject to the machine's existing sandbox policy.”
SupportedCoding-agent benchmarks can repeat thousands of tests, allowing agents to install packages, edit files and run commands.InfoQ: “Evaluating a coding agent means letting it install packages, edit files, and run commands, and a benchmark repeats this thousands of times.”
SupportedDocker's engineers identify reward hacking as a concern, including tampering with tests or overwriting score records instead of solving the task.InfoQ: “They also flag reward hacking, where an agent earns a passing score by tampering with tests or overwriting the file that records its score instead of solving the task.”
SupportedHarbor orchestrates evaluations by running tasks, collecting scores and logging, and supports benchmarks such as Terminal-Bench.InfoQ: “Harbor manages orchestration. It runs tasks, collects scores, and logs. It also supports benchmarks like Terminal-Bench.”
SupportedDocker Cloud Sandboxes can restrict internet access after dependencies are pulled during setup, helping make network conditions consistent across evaluations.InfoQ: “In Docker Cloud Sandboxes, configurable network policies let a task pull dependencies during setup, then restrict internet access while the agent works. Every agent is then evaluated under the same network conditions”
SupportedDocker Compose support lets multi-container tasks run inside a single microVM.InfoQ: “Docker Compose is supported, so Harbor can run multi-container tasks inside a single cloud sandbox, with services and sidecars sharing the same microVM.”
SupportedDocker tested Harbor with 64 concurrent cloud tasks, and higher limits are available on request.InfoQ: “Docker says it has tested Harbor with 64 concurrent tasks in the cloud. Higher limits are available on request through Docker's contact channel.”