For teams changing a guard or an agent.
This work fits engineering teams changing a model, prompt, threshold, or provider. We compare safety classifiers on labeled examples and test OpenAI-compatible endpoints against expected behaviour. The scope defines the policy, target configuration, and examples behind each release decision.
Test endpoint behaviour with YAML scenarios.
Scenarios check returned tool calls and arguments, responses to injected tool output, system-prompt leaks, JSON format, latency, and refusals. Tool assertions inspect the calls an endpoint returns; executing tools and verifying downstream effects require separate integration checks.
- Define the expected behaviour and assertions in a YAML suite
- Exercise legitimate requests and adversarial cases
- Compare baseline and candidate outcomes by scenario
- Retain failed examples and the configuration used
Audit test strength and repeatability.
Before a suite becomes a release gate, its audit checks review status, whether assertions can fail against a broken target, weak categories, near-duplicates, and flakiness across repeated runs. Rubric judgments can be cross-checked with a second judge when configured. These checks test the suite’s usefulness; observed repeatability is limited to the evaluated runs.
Measure the filter tradeoff.
A guard can catch more unsafe inputs while blocking more legitimate ones. GuardMeter reports recall, false-positive rate, precision, F1, and latency, with views by category, language, and attack type. Confidence intervals and paired comparisons put observed differences in context.
- Agree on labels, policy, and representative examples
- Compare baseline and candidate predictions
- Inspect failures and affected slices
- Document coverage limits and unresolved questions
Test the languages your customers use.
GuardMeter supports attack testing in 24 languages through language-aware tooling, six cross-lingual attack families, and per-language release gates. Agree the languages and cases for your engagement, then inspect results by language. Dataset authoring and native review are tracked separately; supported tooling does not mean every language has a completed, reviewed dataset.
Open datasets you can inspect and reuse.
The GuardMeter attack datasets are licensed under CC BY 4.0 for reuse and adaptation, including commercial use, with attribution, a license link, and changes identified. The evaluation toolkit is separately MIT-licensed. Our service adds tailored tests, human review, and recommendations around your agreed scope.
Connect the check to CI and rollouts.
A versioned gate defines the thresholds a run must meet. The GitHub Action and CLI provide JSON, Markdown summaries, JUnit outputs, and alerts on gate failure. The synchronous POST /api/hooks/rollout hook runs a scenario suite and returns pass/fail plus regressions; the calling orchestrator decides whether to block or roll back the model change.
- A reviewable suite and gate configuration
- An evaluation report and machine-readable summary
- CI checks tied to the evaluated model change
- A rollout decision for your orchestrator to enforce
Inspect the evidence and its scope.
The sample report and evidence pack expose a failed gate and its underlying checks. Our independent Opod field note shows how scenario outcomes change between models. A pass applies to the evaluated examples and policy; evaluation is not certification.
EU AI Act high-risk obligations for Annex III systems apply from 2 December 2027; those for high-risk systems embedded in Annex I products apply from 2 August 2028. Scope varies by system and use, and an evaluation does not establish regulatory compliance.
Start with one release check, or a broader integration.
The GuardMeter page describes the fixed-scope Agent Release Check, its pricing, and the deliverables. If the evaluation needs to become part of a larger product or delivery pipeline, describe the existing system and the decision the checks need to support.