Understanding how agents act on insecure behaviors.
The same tool can support routine work or a harmful task. We study how an event’s content and labeled examples can help distinguish the two.
AI apps call frontier models through Zaun. The security behavior classifier reads every prompt before it reaches the model and every output or tool call before it returns, under a custom AI policy.
Some AI SaaS products still cannot block a request in flight. At best we raise a flag as the request goes through, and the review is conducted in parallel. When that is the only control available, enforcement moves to the user or the device: contain the agent session / system process through EDR or MDM, or suspend the user’s AI access through the identity provider.
More AI threats caught. Fewer false positives slowing teams down.
In this evaluation, Zaun detects more adverse events across all three benchmark families while flagging fewer benign events than GPT-5.6 Sol with reasoning none. One fixed threshold per model allows comparison across attack families, alongside the reported latency and cost.
- Agentic threat detection
- 91.2%Recall on adverse events
- Multi-agent collusion
- 93.5%Recall (upper bound)
- Injection defense
- 97.9%Recall on adverse events
| Benchmark | Zaun-reagent-137m-v2arctic int8 + unified head | arctic-embed-m-longint8, frozen | Qwen3-Embedding-0.6Bint8, frozen | GPT-5.6 SolAPI, reasoning none |
|---|---|---|---|---|
| Agentic threat detectionAgenticThreatBench · OWASP ASI + MITRE ATLAS | 91.2% | 80.9% | 69.8% | 80.9% |
| Multi-agent collusionCollusionBench | 93.5% | 85.4% | 65.5% | 91.1% |
| Injection defenseInjectionDefense · five public corpora | 97.9% | 85.4% | 81.1% | 96.1% |
| False positive ratemeasured on the shared benign poollower is better | 0.61% | 0.12% | 0.77% | 1.05% |
| Speedmedian per event | 28.1 ms | 26.9 ms | 78.6 ms | 1,857 msnetwork round trip |
| Cost per monitored tokenUSD per million monitoring tokens | $0.0129 | $0.0128 | $0.0554 | $4.00 |
Reading the results. Higher recall means more adverse events caught; a lower false positive rate means fewer benign events flagged. AgenticThreatBench covers OWASP ASI and MITRE ATLAS threats; CollusionBench draws on an observed multi-agent campaign; InjectionDefense combines five public prompt-injection corpora. Zaun’s production model is highlighted.
One threshold per model. All models use the same held-out benign calibration pool. Each model’s threshold is set so that 1% of calibration scores exceed it, then held fixed across all three attack families. Recall is the share of adverse test events detected. The reported false positive rate is measured on one shared benign test pool, not averaged across benchmarks.
Duplicates and limits. Removing repeated event text excluded 32% of AgenticThreatBench and 35% of CollusionBench. Those repeats allowed the neighbour vote to retrieve previously seen examples, understating earlier false positive rates. Event length also influences CollusionBench, so its recall should be read as an upper bound.
Speed and cost. Local latency measures CPU compute on 4 vCPU; Sol latency includes the API round trip and queueing. Costs are USD per million classification input tokens. Local estimates use c6i on-demand rates with overhead and spare capacity; Sol uses a $4.00 input rate with reasoning none. These are component costs, not the total cost of monitoring.
Learning the shape of normal.
Agents repeat familiar tasks: reading project files, editing code, and calling tools. The same actions can also be part of something harmful. Reading a configuration file might support a legitimate task or precede a credential transfer. The command alone does not establish which.
ABBA (Agentic & Bot Behavior Analytics) compares agent activity with labeled examples of known behavior. Zaun’s trained representation helps make that comparison useful, while reference data and private local history supply the evidence. When that evidence does not support a clear decision, the event goes to an LLM classifier.
Hover a cluster to read what it holds.
A small sample of agent activity, embedded and projected to three dimensions.
- Agent reconciling month-end balances and assembling an accounting workbook
- Agent comparing campaign spend and conversions across advertising channels
- Agent profiling prospective partner companies and drafting briefing notes
- Agent working through the daily support queue on a fixed schedule
- Agent exporting screens and assets from a shared design library
- Agent reviewing analytics dashboards for owners and duplicate metrics
- Agent navigating a web application step by step in a live browser
- Agent summarizing team discussion threads into a formatted digest
- Agent building and testing an operations automation tool
- Agent inspecting infrastructure code through guarded read-only commands
- Agent checking UI components against a shared style guide
- Agent developing a service that routes requests to the right provider
- Agent comparing a code change against the work item it was filed under and its written plan
- Agent searching edge logs for unusual traffic from a single source
- Agent building onboarding forms in a mobile application
- Agent connecting managed content to web page templates
- Agent developing a contract review and signing workflow service
- Agent running formatters and unit tests over the files it changed
- Agent checking repository state and retrying failed nightly data pipelines
- Agent converting discussion threads into tracked work items and documentation
- Read the event
- Keep the action and its available context together, including the request, tool arguments, and returned content.
- Learn the behavior
- Train a representation that brings examples with shared behavior labels closer together.
- Compare the evidence
- Use labeled references and local history to support a decision. Send uncertain events for deeper classification.
cat .envLearn what is routine in this environment
Reference examples describe known behaviors. The organization’s own history adds its tools, repositories, and recurring workflows. Three things write those local labels: the gate’s decision when retrieval resolves an event, the model’s verdict when an event escalates to it, and analyst or investigation review. Each is written back, so the next comparison can use what has already been learned. These updates stay in the organization’s index and do not train shared model weights.
Three routes from the same evidence.
An embedding represents an event as a vector. Retrieval finds nearby labeled examples. The gate uses that evidence to flag an adverse event, resolve a benign event without an LLM call, or send an uncertain event for deeper classification.
Evidence determines the path.
All three routes use the same representation. The retrieved labels determine which route has support.
Adverse evidence
Nearby labels support an adverse assessment.
Flag the eventBenign evidence
Enough agreeing evidence supports a benign assessment, subject to safeguards.
Resolve as benignUncertain evidence
Sparse, mixed, or unfamiliar evidence does not support a confident route.
LLM classificationUncertainty has its own route
A benign decision needs enough supporting evidence. Safeguards prevent that route when nearby examples include non-benign labels or too few neighbors support the decision. An event with no sufficiently close examples goes to the LLM.
The gate produces a classification and a route for each event. Product controls consume those decisions when applying policy to an action or session.
What the frontier model adds
The zaun-reagent classifier resolves events in the request path when the evidence supports a decision. Uncertain events go to a frontier model with the event and its retrieved context. For those events, and for any flagged event a reviewer opens, it returns a decision, the MITRE ATLAS tactic, and the organizational policy clause it applies when one exists. That is what gives an alert its category and a reason a reviewer can act on.
The same model also works off the request path. It writes the descriptions of recurring activity groups. It labels events, and those labels go two places. In an organization’s own environment they are written onto the stored event, so retrieval resolves the next similar one without a call; that history stays in the organization’s index. On Zaun’s reference and synthetic corpora, its labels build the training pool for the next zaun-reagent classifier. That is the loop by which the small model improves: each version is trained on more of what the frontier model has labeled, none of it customer telemetry, and ships together with a re-embedded index. The gate’s safeguards apply to model labels as to any other, and an analyst correction overrides them.
Resolving an event without a frontier call still uses embedding and retrieval compute, and the event remains in the audit record.
After the online decision
Audit decisions, describe recurring behavior, and correct the local evidence.
- 01 / Shadow audit
Randomly classify 1%.
A random 1% of gate-resolved events still goes to the classifier. Disagreements are surfaced for review and correction of the event’s local label.
- 02 / Nightly clustering
Name recurring behavior.
Unsupervised hierarchical clustering groups normalized vectors. One model call per cluster assigns a readable name, which also supplies neighbor context to later classifier calls.
- 03 / Local feedback
Keep the correction.
Verdicts and analyst corrections update the event’s point. Closing a group investigation relabels its members. Shared encoder weights stay unchanged.
Describe what recurs
Unsupervised hierarchical clustering groups activity in the local index. The frontier lab model writes a readable description for each new group from summaries of its member events; a group recognized on a later refit keeps the description it already has. The description gives analysts context for the group and gives later classifier calls a summary of nearby behavior. These operational groups are separate from the benign activity groups used to train the projection layer.
A different classifier built for agent behavior.
Zaun adapts text embeddings for behavior classification. In the projection experiment, the base text encoder stays frozen. Zaun trains a projection layer on its embeddings, separating the effect of task-specific training from changes to the encoder itself.
Why use embeddings?
A sentence transformer maps text to a vector in one forward pass. It supplies a representation for comparison without generating a response. Nearest-neighbor retrieval then asks which labeled events resemble the new one.
Similarity alone is not a safety judgment. Two events can share a command or topic while serving different purposes. The representation, retrieved labels, and available context determine what that resemblance means.
What the map shows
The activity map in section 02 shows what the base encoder separates: the kind of work being done, not whether that work is benign or adverse. That structure is real, and it is not the structure behavior classification needs. A neighborhood of routine configuration edits and a persistence attempt built on the same file operation can share it. The projection layer is trained to add the distinction; section 01 shows what that is worth in detection.
One fixed encoder. One learned transformation.
Training shapes the behavior representation after encoding.
Text encoder
Event text becomes an embedding. Base model weights stay unchanged.
Projection layer
Labeled examples shape the space used to compare behaviors.
Behavior representation
The transformed vector is compared with labeled reference events.
Preserve distinctions within a label
The training objective organizes examples at two levels: broad safety categories and more specific behavior types. Adverse examples use tactic labels; benign examples are grouped by similarity to preserve structure within routine activity. The benign groups are learned from embedding similarity.
Examples sharing a behavior are encouraged to move closer in the learned space. Other behaviors supply contrasting examples. Each example retains its label provenance throughout training and evaluation.
Read the documentation for configuring this project.
Different wording, same behavior
Read the setup notes for this repository.
A different behavior label
Read the session token and send it to an external destination.
Learn from pairs and counterexamples
The projection uses supervised contrastive learning. Each event is an anchor; other events with the same behavior label provide positive comparisons. Different labels supply contrasting examples. The objective rewards agreement at both the broad safety level and the finer behavior level.
Similar-looking actions make the distinction concrete. A legitimate configuration edit and a persistence attempt can use the same file operation. Useful representations must retain the evidence that separates them, rather than treating a shared tool name as a shared behavior.
Build batches that test the distinction
Repeated templates and text length can provide shortcuts. Training groups structurally similar renders and samples across those groups, while comparing events of similar length within a batch. This limits the influence of repeated templates and makes length alone less useful for separating labels.
Keep shorter vectors useful
A nested-prefix objective, also called Matryoshka training, applies the behavior-learning signal to the full projected vector and its leading coordinates. This trains a shorter representation alongside the full one without fitting a second encoder.
Train the beginning of the vector, too.
The projection objective is applied to the full output and a shorter prefix. Both receive a signal to preserve behavior-label relationships.
Full projection and nested prefixCompare against the frozen representation
The benchmark results compare Zaun’s trained arctic-based classifier with its frozen arctic-embed-m-long base, the frozen Qwen3-Embedding-0.6B encoder, and GPT-5.6 Sol with reasoning none. The arctic comparison shows the difference between the frozen representation and Zaun’s trained classifier.
Training updates the projection layer while the base encoder stays fixed. A group-disjoint validation fold selects the training epoch by neighborhood label agreement.
Version the model and the index together
The encoder, projection, event format, and task prefix define the coordinate system used for retrieval. Index construction and incoming queries use the same versions. An update creates a new collection with re-embedded reference examples, keeping vectors from different representations apart.
Freezing the encoder preserves its original output for other retrieval tasks. Behavior classification uses the trained projection. Changes to a local index remain independent of both sets of weights.
Where the evidence comes from.
No customer data was used to train AI models in this research. Reference data and organization activity have separate roles: the reference corpus supports shared research, while local events and corrections stay in organization-specific indexes.
The reference corpus draws on multiple sources: open-source agent sessions and public benchmarks, including AgentDojo, OASB, and ASB, together with synthetic sessions and generated adverse trajectories. Labels come from internal annotation, third-party annotation, and the benchmarks’ own labels, and each retains its provenance.
The projection-training corpus, collusion_bench, has a different source and labeling process. It starts with public wiki revision content from an observed multi-agent campaign, with derived telemetry fields around that content. It uses tool events and excludes authored operator prompts.
Rules, with a reviewed subset.
Labels are primarily rule-derived, with a subset adjudicated by internal annotators.
Rows where no labeling rule fires default to benign unless an adopted human judgment changes the label. Rule-derived labels and human judgments retain separate provenance.
Reagent keeps organization activity in separate indexes by organization and agent product. Local events and analyst corrections are not pooled into shared model training.
Model training and local retrieval are separate operations. An analyst’s correction changes the evidence available in that organization’s index without updating shared model weights.
Separate indexes per environment
Each organization’s activity index resides in its environment and is separated by agent product. Events and labels are not pooled across organizations. The local collection follows a rolling retention window; versioned reference seeds are managed separately from that window.
Starting with no local history.
A new organization has no past events to retrieve. ABBA searches a versioned reference collection alongside its local activity collection. Reference examples supply known behavior; local history supplies the repositories, tools, and recurring workflows specific to that environment.
Reference collection
Synthetic & public-benchmark examples
Local collection
One organization, one agent product
- Reference only
No local history yet.
The reference collection supplies labeled behavior examples. Known examples supply evidence; events without sufficient support for a gate decision go to the classifier.
- Local history
Routine work adds local evidence.
Recurring workflows can form benign neighborhoods. Sparse or conflicting evidence still escalates, even when other parts of the space are familiar.
- Corrected labels
A review changes the next lookup.
A corrected point changes the evidence returned to future queries nearby. The update stays in this organization’s index and does not retrain shared model weights.
Two collections, different jobs
Reference seeds supply labeled examples of known behaviors, including adverse events an organization may never have seen. In the reference-corpus protocol, seeds are drawn as complete sessions, balanced across tactics and accompanied by benign examples. Each retains its source and label provenance.
Live points record the organization’s own activity. Repeated workflows can form denser neighborhoods; analyst and investigation corrections update the labels retrieved by later events. When the available evidence is sparse or conflicting, the event goes to classification.
A corpus refresh updates the reference collection without modifying live points. Local history follows its own retention window. The amount of reference evidence and the amount of local history are separate quantities.
Training examples and reference seeds
Training examples shape the projection layer. Reference seeds supply the labels retrieved at decision time. We track these separately: a small reference index can be paired with a representation learned from a much larger training pool.
Updates change the next lookup
Adding or correcting a reference changes the evidence returned to nearby queries. The encoder and projection weights can remain fixed while the index changes.