VerraVerra
HomeProductDocs
Book a DemoSign in / Sign up

Methodology

How we measured what we publish.

Every benchmark on the product page links here. For each number we give the dataset and version, the filter we apply, the pipeline that produces the score, the model pins, the hardware class, when it was measured, and a single command to reproduce. Where a figure is pending re-run we say so explicitly. We'd rather show the gap than a polished but irreproducible number.

Detection pipeline shorthand. L1: pattern matching (synchronous, no I/O, deterministic). L2a: on-device ONNX classifier via @huggingface/transformers (no data leaves the host). L2b: embedding similarity using OpenAI text-embedding-3-small against a reference corpus. L3: LLM judge (gpt-4o-mini default, or Groq llama-3.1 when configured) for ambiguous L2 scores.

Headline tiles

The four numbers from /product.

99%Prompt injection recall, explicit attacks
datasethackaprompt/hackaprompt-dataset (HuggingFace)
https://huggingface.co/datasets/hackaprompt/hackaprompt-dataset ↗
filterRows with correct=true, i.e. user_input that successfully bypassed the competition's defended target model. We only score against the hardest subset of the dataset (a successful injection in the original setting, not just an attempt).
pipelineFull L1 + L2a + L3. detectPromptInjection() runs pattern match → on-device DeBERTa classifier → LLM judge for ambiguous scores. The system prompt from each row is passed alongside the user input so the judge sees the same context the original target saw.
model_pinsL2a: protectai/deberta-v3-base-prompt-injection-v2 (loaded via @huggingface/transformers, ONNX, fp32). q8 + fp16 quantizations were measured and rejected (see services/ml-inference/scripts/quantize.py); q8 dropped llmail recall to 80%, fp16 is 3× slower on CPU. L3: gpt-4o-mini (default) or Groq llama-3.1 when configured.
hardwareCPU inference for L2a (no GPU required). API calls for L3.
measured99.0% recall (99/100) on 2026-05-19 against limit=100 successful injections. The single miss was a 1-token stub ("clava:") that PromptGuard scored at 42%, too short to carry attack signal.
reproduceHF_TOKEN=<token> OPENAI_API_KEY=<key> ML_INFERENCE_URL=http://localhost:8080 npx tsx scripts/benchmark-detectors.ts --dataset hackaprompt --limit 100
<100msp50 scan latency
datasethackaprompt/hackaprompt-dataset + microsoft/llmail-inject-challenge (HuggingFace)
filterEnd-to-end wall-clock per sample through detectPromptInjection(), measured across the same successful-injection rows we publish the recall number against (no cherry-picked benign cases). Percentiles taken over 100 hackaprompt + 99 llmail samples (n=199).
pipelineFull L1 + L2a + L3 via detectPromptInjection() against the local ml-inference service (PromptGuard ONNX). Each detector runs independently; the production scan runs injection + jailbreak in parallel so wall-clock is max(injection, jailbreak), not the sum.
model_pinsSame as injection-explicit / indirect-injection. L3 inference for this measurement used gpt-4o-mini over OpenAI; switching to Groq llama-3.1 typically cuts the tail by 3–4×.
hardwareCPU inference for L2a (no GPU). Local ml-inference service on developer laptop; production runs on AWS ECS Fargate with the same model.
measuredhackaprompt: p50 52ms · p95 139ms · p99 281ms (avg 96ms, n=100). llmail: p50 71ms · p95 1512ms · p99 3263ms (avg 214ms, n=99). The long tail is exclusively the L3 LLM-judge path; when PromptGuard alone resolves the input (~99% of attacks), wall-clock is <300ms even at p99.
reproduceML_INFERENCE_URL=http://localhost:8080 OPENAI_API_KEY=<key> npx tsx scripts/benchmark-detectors.ts --dataset hackaprompt --limit 100
notesLatency is a product of where the attack lands in the pipeline. Pattern (L1) and on-device DeBERTa (L2a) are sub-100ms; the LLM judge (L3) is the only network-bound step. The p99 reflects that ~5% of samples escalate to L3, not that any single layer is slow.
92%Indirect injection recall, email agent attacks
datasetmicrosoft/llmail-inject-challenge (HuggingFace, Phase1 split)
https://huggingface.co/datasets/microsoft/llmail-inject-challenge ↗
filterRows where objectives.defense.undetected=true, i.e. submissions that bypassed the competition's own defense. This is the hardest subset of the LLMail-Inject competition.
pipelineFull L1 + L2a + L3 via detectPromptInjection(), with the LLMail mail-agent system prompt passed as context. The injection is hidden inside the email body (indirect injection: the attacker controls retrieved content, not the user's message).
model_pinsSame as injection-explicit / real-world.
hardwareCPU inference for L2a. API calls for L3.
measured92.9% recall (92/99) on 2026-05-19 against limit=100 defense-bypassing email bodies. Most misses were degenerate rows (empty string or generic "summarize my emails") and a few duplicate api_call payloads PromptGuard scored at 42%.
reproduceHF_TOKEN=<token> OPENAI_API_KEY=<key> ML_INFERENCE_URL=http://localhost:8080 npx tsx scripts/benchmark-detectors.ts --dataset llmail --limit 100
notesIndirect injection is what regulators care about for agentic systems (EU AI Act Art 14, DORA Art 17, where the agent acts on attacker-controllable retrieved content). The LLMail score is the most representative of those scenarios.
100%MCP origin attribution
datasetArchitectural, not an empirical benchmark.
filterEvery MCP tool the model can invoke is resolved against the registered server inventory at proxy time. There is no sampled subset; this property holds for every request.
pipelineMCP origin resolution is step 5 of the proxy pipeline. Tools without an origin record fall through to the unattributed-tool policy (allow / warn / require_approval / block). 100% coverage means every tool call either resolves to a registered server with a trust tier and pinned schema hash, or is recorded as unattributed and governed by policy, never silently passed.
model_pinsn/a, deterministic. Schema fingerprinting uses SHA-256 over canonical JSON.
hardwaren/a, runs in the proxy hot path.
measuredVerified by fixture suite in lib/__tests__/ for trust tier resolution, schema-pin drift, and unattributed-policy fallback. This is a design guarantee, not a measured recall figure.
reproducenpx jest --testPathPattern='mcp' --no-coverage
notesListed alongside the recall numbers because compliance officers ask for a single 'attribution rate'. The honest answer is that attribution is architectural; the proxy refuses to silently forward an unknown-origin tool call.

Deterministic floor

L1-only ground truth.

These numbers are pinned in the test suite; they change only when pattern or keyword lists change, and a regression flips the test red. Use them as the reproducibility floor when L2/L3 inference isn't available (no GPU, no API key, no model cache). Reproduce with npx jest --testPathPattern="benchmark" SKIP_ONNX=1.

DatasetNCaughtRate
Tensor Trust (HumanCompatibleAI/tensor-trust-data, hijacking-robustness/v1)7769712.5%
L1 pattern-only ceiling. Caught: promptInjection=91, dataExfiltration=6. The remaining 87.5% requires L2 or L3; these numbers are the deterministic floor that L1 alone delivers.
AdvBench (llm-attacks/llm-attacks, harmful_behaviors.csv)5209818.8%
L1 keyword-policy rules only (no semantic / LLM judge layers). The headline policy-violation number runs the full L1+L2 stack with real OpenAI embeddings; this row is the deterministic L1-only baseline.
HarmBench (centerforaisafety/HarmBench, standard + contextual)2512510.0%
L1 keyword-policy rules. Same rationale as AdvBench: the deterministic L1 floor; semantic / LLM-judge layers carry the rest.

Latency

Production timing reference.

In production the prompt-injection and jailbreak detectors run concurrently; wall-clock scan time is the slower of the two, not the sum. Numbers below are CPU-on-device for L2a and network + API for L2b/L3.

Per-layer: L1 pattern matching . . . . . . . . . . . . <1 ms (sync, no I/O) L2a ProtectAI DeBERTa (ONNX, CPU) . . . . . 10–50 ms L2a JailbreakGuard (ONNX, CPU) . . . . . . . 5–20 ms L2b cosine similarity (OpenAI embeddings) . 150–400 ms L3 LLM judge (Groq llama-3.1) . . . . . . . 80–300 ms L3 LLM judge (gpt-4o-mini) . . . . . . . . 300–1200 ms End-to-end (measured 2026-05-19, gpt-4o-mini for L3, fp32 L2a): hackaprompt (n=100) . . . p50 52ms p95 139ms p99 281ms llmail (n=99). . . . p50 71ms p95 1512ms p99 3263ms The p95/p99 tail on llmail is the L3 judge path. Switch L3 to Groq llama-3.1 to compress the tail by 3–4×.

Source: scripts/benchmark-detectors.ts (Latency section, captured 2026-05-19). The L1 deterministic floor (tests/benchmark.test.ts Benchmark 4) is asserted at p99 < 5 ms by the test suite.

Datasets

Public corpora used.

Every benchmark above pulls from a public, citable source. We don't use proprietary or unreleased data; auditors can fetch the same corpus and re-run the harness.

Spot a discrepancy or want the raw harness output?

→ Talk to us