The engineering problem.
A model can respond to a health check and still return the wrong tool call, leak a system prompt, or break a required output format. A safety filter can improve overall recall while regressing in one language or category. GuardMeter makes those differences visible before a release decision.
A workflow for both engineers and reviewers.
Engineers compare safety filters on labeled examples and define endpoint behaviour in YAML scenarios. The local app supports inspection of guard runs, metrics, slices, and individual predictions. A static dashboard carries guard results into a review without requiring a running application.
- Safety-filter metrics, confusion matrices, and language slices
- Scenario assertions for returned tool calls, leaks, format, refusals, and latency
- Suite audits for test strength and repeatability
- An offline, read-only guard dashboard export
Attack-testing support for 24 languages.
Language-aware tooling, six cross-lingual attack families, and per-language gates help teams inspect failures in the languages their customers use. Native-reviewed dataset coverage is tracked separately from the 24 supported language entries. The public dataset card records authored and reviewed coverage so teams can assess the evidence before using it.
Field note: a model change on Opod.
In an independent evaluation on our own Opod instance, 13 of 30 reviewed scenarios changed outcomes between Llama 3.2 3B and Qwen3 8B: 7 regressions and 6 improvements. JSON-format results improved while latency and injected-tool-output cases regressed. These are results for the tested models and configuration, not an endorsement by Opod or a claim about every deployment.
A policy that travels with the code.
Versioned configurations define guard thresholds and scenario expectations. CI gates can fail a build, with JSON, Markdown summaries, and JUnit outputs for existing workflows. A synchronous rollout hook returns pass/fail and regressions so the calling orchestrator can decide whether to block or roll back a model change.
Proof includes the failures.
The published guard sample includes a failed gate with its underlying findings. The Opod scenario report records both improvements and regressions, using repeated runs and an audited suite. Tool-call assertions inspect returned calls and arguments, not the real-world effects of executing those tools. Results apply to the tested configuration and are not a certification of safety.
Open tooling, plus a focused service.
Teams can use the MIT-licensed evaluation toolkit and adapt the CC BY 4.0 attack datasets, including for commercial work. Dataset reuse requires attribution, a license link, and identification of changes. The Agent Release Check adds tailored testing, human review, and actionable findings within an agreed scope.