NVIDIA turns agent consent failures into tests NVIDIA turns agent consent failures into tests

The trace supplies the failure

The demonstration uses Hermes with a fictional company's communication and work tools. A changed instruction separates preparation from permission to send. NVIDIA reports six successful held-out runs, compared with four before the change. The deliverable is a testable distinction between two actions that can look similar in a conversation.

Compass sorts several kinds of evidence

NeMo Compass examines traces using five kinds of signals. These cover unusual behavior, tool problems, user complaints, divergence from declared rules and poor evaluation results. Its anomaly detector uses an Isolation Forest, while tool checks look for invalid arguments, failures and unproductive retries. The resulting findings point back to recorded runs. That connection lets a reviewer inspect the action sequence behind an issue rather than accepting a generated diagnosis on its wording alone.

The test needs a definition of success

NeMo Eval Author supplies skills for a coding assistant working with an agent's repository and intended behavior. It can help build an initial suite without production traces. Harbor and NeMo Gym are the supported evaluation formats. Its setup reviews or creates an ETHOS.md document describing purpose, boundaries and success criteria. This makes the expected behavior an input to the evaluation rather than something inferred only from whatever the agent happened to do.

Missing evidence has a narrower meaning

The Eval Author documentation distinguishes an unobserved behavior from an absent test. An audit might find no recorded recovery from rejected tool arguments. That finding alone does not establish that the suite lacks a recovery case. A full coverage audit needs traces from completed evaluation runs. The distinction matters when deciding whether to write another case or collect another run: those are different repairs to an evaluation process.

Validation and evaluation answer different questions

Eval Author's known-issues guide separates checking a task's environment and grader from measuring the target agent. The first establishes that the machinery for a test works. The second examines the agent's actual behavior inside that machinery. The guide describes inspecting inputs, variations, success criteria and completed runs as distinct steps. A test can execute correctly while still asking the wrong thing. That distinction gives reviewers a specific place to investigate a surprisingly clean score.