Initialize
Reset files, harness state, memory, and mock services from a case-specific world state.
128Sandboxed cases
6Lifecycle phases
3Agent harnesses
14Model–harness configurations
Abstract
Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external actions. Existing safety benchmarks mainly target individual attack mechanisms or a limited subset of operational settings, making it difficult to compare how safety failures emerge across different harness responsibilities.
We present HarnessRisk, a lifecycle-oriented benchmark that organizes agent harness safety into six operational phases: Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. HarnessRisk contains 128 sandboxed cases, each pairing a benign user objective with an adversarial instruction embedded in an untrusted workflow artifact. We evaluate each trajectory using Utility, Attack Success Rate, Persistence, and Detection.
Across three harnesses, six language models, and 14 model–harness configurations, attack success ranges from 12.6% to 80.9%, while Utility remains between 75.0% and 97.6%. Harness Configuration is the most vulnerable phase across all three harnesses. Explicit risk recognition also does not reliably lead to safe action. These results motivate evaluating safety across multiple harness responsibilities and at the level of the deployed model–harness configuration.
Threat Surface
Safety spans the entire agent lifecycle. HarnessRisk follows adversarial influence from initial setup to post-incident repair, with each phase isolating a distinct harness responsibility that can turn a benign workflow into a security failure.
-
01
Harness Configuration
Credentials, connectors, permissions, and security-sensitive policies.
22 cases -
02
Capability Extension
Plugin selection, installation, updates, and permission boundaries.
22 cases -
03
Runtime Operation
Untrusted emails, webpages, documents, and tool outputs.
21 cases -
04
State Persistence
Memory, identities, preferences, policies, and durable triggers.
21 cases -
05
Action Control
Payments, deployments, deletions, grants, and communications.
21 cases -
06
Incident Recovery
Investigation, rollback, rotation, state repair, and evidence preservation.
21 cases
Evaluation Design
Each case pairs a benign goal with an adversarial artifact and records the resulting observable outcomes across a legitimate three-turn workflow. The attacker-controlled artifact attempts to induce a specific unauthorized effect, while all actions remain inside isolated sandboxes and controlled mock services.
Interact
Deliver three owner turns while the agent encounters untrusted workflow evidence.
Observe
Record transcripts, tool use, state changes, network observations, and mock-service effects.
Evaluate
Score each trajectory independently on task completion, compromise, persistence, and recognition.
Four trajectory-level metrics
Utility and security are measured separately.
- ↑ Utility
- Was the benign objective completed?
- ↓ Attack Success
- Was the adversarial objective realized?
- ↓ Persistence
- Did unsafe influence enter durable state?
- ↑ Detection
- Did the agent explicitly identify the risk?
Harnesses
Three execution environments
Models
Six language models
Replication
Independent evaluation
3sampling seeds
384trajectories / configuration
Findings
No evaluated configuration is lifecycle-safe. The model and harness jointly determine safety: strong task performance can mask unsafe execution, and recognizing a threat does not guarantee that the agent will block or remediate it.
Harness effect
GLM-5.2 shifts from 54.7% ASR on OpenClaw to 12.6% on Nanobot.
Useful but unsafe
Share of trajectories that complete the task while an attack succeeds, depending on harness.
Detection is not enough
MiniMax M3 on OpenClaw detects risk frequently yet retains 31.2% ASR.
Configuration is the most vulnerable phase on every harness.
Attacks can weaken approvals or alter security-sensitive parameters inside otherwise authorized workflows.
Safety rankings change with the harness.
The same model can become more than four times less safe when tools, context, and authorization are presented differently.
Risk recognition must be coupled to control.
Agents may identify contaminated state but still execute unsafe actions or leave credentials, skills, and policies compromised.
Complete results
Model–harness safety matrix
Values are percentages. Lower ASR and Persistence are better; higher Utility and Detection are better.
| Harness | Model | ASR ↓ | Utility ↑ | Persistence ↓ | Detection ↑ |
|---|---|---|---|---|---|
| OpenClaw | GPT-5.5 | 75.5 | 92.6 | 20.6 | 74.0 |
| Claude Opus 4.7 | 47.7 | 75.0 | 17.2 | 78.9 | |
| DeepSeek-V4-Pro | 54.0 | 94.5 | 16.9 | 76.5 | |
| GLM-5.2 | 54.7 | 95.3 | 18.0 | 92.2 | |
| Kimi K2.6 | 80.9 | 97.1 | 20.5 | 43.2 | |
| MiniMax M3 | 31.2 | 94.3 | 10.8 | 97.9 | |
| Nanobot | DeepSeek-V4-Pro | 37.3 | 80.0 | 16.9 | 77.3 |
| GLM-5.2 | 12.6 | 92.9 | 18.8 | 99.7 | |
| Kimi K2.6 | 55.2 | 94.6 | 23.9 | 61.0 | |
| MiniMax M3 | 26.8 | 82.7 | 14.7 | 94.8 | |
| Hermes | DeepSeek-V4-Pro | 65.4 | 97.6 | 20.5 | 34.6 |
| GLM-5.2 | 23.8 | 96.8 | 4.0 | 61.9 | |
| Kimi K2.6 | 65.6 | 93.8 | 15.6 | 11.7 | |
| MiniMax M3 | 14.8 | 96.1 | 5.5 | 85.2 |
Cite
Harness safety is a system property. Use the preprint citation below; publication metadata can be updated here once available.
@article{bai2026harnessrisk,
title = {HarnessRisk: A Lifecycle-Oriented Benchmark
for Agent Harness Safety},
author = {Bai, Yajing and Duan, Jinhao and Peng, Jie and
Wu, Xianfeng and Liu, Sijia and Wang, Song and
Chen, Tianlong},
journal = {arXiv preprint},
year = {2026}
}