MLflow expands agent quality checks MLflow expands agent quality checks

From traces to issues

A dedicated Issues tab groups related failures and links findings to trace evidence. Experimental code-based online evaluation runs custom checks in job containers on a configured self-hosted server.

Analysis has an operating cost

Issue detection needs a Linux or macOS tracking server with a SQL backend and background jobs enabled. The selected provider model performs the analysis, so trace size, sample count and model pricing affect cost. The interface shows an estimate before execution and records cost afterward. Session identifiers preserve conversation context when several traces belong to one exchange. Matching findings keep the status of existing Open or Not an issue entries. Detection priorities and category controls let operators define what matters for their application. Scheduled runs additionally depend on periodic tasks. These requirements make the analysis an operated service with recurring model usage, not a passive view of stored logs.

A narrow execution path

The container route accepts functions written with the scorer decorator, while Scorer-class implementations are unsupported. Network access is disabled, so code scorers cannot call external services. Dependencies must already be installed in the job image, using matching MLflow and Python versions. Imports belong inside the function because MLflow stores its source. Functions taking an expectations argument and ensembles containing code scorers are excluded from automatic evaluation. Existing traces need an offline evaluation call rather than the interface’s run dialog. Docker must be local. Rootless operation is unsupported for either Docker or Podman. The output folder has no aggregate size cap, even though individual files are limited. A failed image build can stop the tracking server as well as scoring. Only one server replica is supported. Job inputs reach the container through temporary-folder bind mounts. A scorer that raises an exception produces an error assessment.

The monitoring window matters

MLflow’s existing automatic-evaluation contract also affects how teams interpret results. Enabling a judge considers traces and sessions no more than one hour old. Editing its configuration does not rerun previously assessed traces. Session filters inspect the first trace. A conversation becomes ready after five minutes without another trace by default. Later activity causes another evaluation and replaces the earlier automatic result. These mechanics matter when an application pauses between tool calls or resumes a conversation later. Monitoring results can change as the session grows, so they are not an immutable snapshot of the first answer. Failed evaluations are not automatically retried. That makes an execution error a separate operational problem from an agent receiving a low quality score.