THE QUESTION, ANSWERED
WHICH AI SECURITY VENDORS PUBLISH THEIR TEST RESULTS?
Few do, and fewer publish the misses. As of October 2026 we found ten public sets of results for AI security products and defenses, plus our own. Two questions tell them apart: who ran the test, and whether the failures are printed next to the successes.
How to read a published result
- Who ran it: the vendor, a lab the vendor paid, or someone with no commercial tie.
- Whether the misses are printed: attacks that got through, and legitimate work that was wrongly blocked.
- Whether you can rerun it: a published method, a public dataset or harness, a named version.
- Whether it tests the product you would buy. A benchmark of generic defenses is not a test of a vendor.
The results we found
| Who | What they published | Who ran it | Misses printed? | Can you rerun it? |
|---|---|---|---|---|
| F5, AI Guardrails | A lab efficacy test, 2026 | SecureIQLab, commissioned by F5. F5 says there was “no F5 involvement in test design or execution.” | Totals of attacks blocked and missed, by category | Method described in the report |
| SecureIQLab, AI Security CyberRisk Validation | A multi-vendor comparison, described as non-commissioned; test plan filed with AMTSO | SecureIQLab | Results not out yet; publication listed for November 18th, 2026 | Method and test plan are public |
| Pipelock | Results against its own open egress benchmark, version 3.6.0, October 3rd, 2026 | The vendor; the page calls every entry “self-run, self-reported” | Yes: 187 of 190 attacks contained, 2 of 76 benign cases wrongly blocked, failed cases named | Yes, open harness and corpus |
| Deconvolute Labs | An SDK benchmark in a public repository | The vendor | Yes: 1 attack missed and 7 benign samples flagged, of 300 | Yes, run commands published; one narrow scenario |
| Lakera (Check Point), PINT | An open prompt injection benchmark with scores for several vendors | Lakera, which also ranks first | Partly: one combined score per product, no separate false-positive rate | Partly; part of the dataset is not public |
| AgentDojo (ETH Zurich, Invariant Labs) | An open benchmark of prompt injection attacks and defenses on agents | Academic authors and contributors | Yes: utility, utility under attack, attack success rate | Yes, fully open |
| Meta, LlamaFirewall | A research paper, May 2025 | Meta | Yes: 1.75% of attacks still succeeded with all defenses on, and task completion fell from 47.73% to 42.68% | Framework is open source |
| Protect AI, prompt injection classifier v2 | A model card | Protect AI | Implied by precision and recall, with stated limits including false positives on system prompts | Model is public |
| Mindgard with Lancaster University | A red team of six vendors’ guardrails, 2025 | Mindgard researchers, testing other vendors | Yes, the paper is about evasions | Method published |
| BELLS-O (CeSIA) | An independent benchmark of 28 systems from 17 providers, June 2026 | Independent researchers | Yes: detection and false-positive rates | Authors release the benchmark |
| Mountain Theory | Recorded runs under a published protocol: an agent after a model update, and the same 10 actions against NVIDIA OpenShell in three configurations | Mountain Theory | Yes: the OpenShell run names the two actions we had no policy for. We have not yet published a rate of legitimate work wrongly blocked | The protocol is public; the runs are recorded, not a harness you can rerun |
Checked October 9th, 2026 against each source. A vendor missing from this table may publish results we did not find; tell us and we will add them. For who does what in AI security, see the landscape.
What is still missing across the field
No AI agent security product we found has results from a party with no commercial tie, run on an action set the vendor did not write, with both misses and false blocks printed. SecureIQLab’s non-commissioned comparison, listed for November 18th, 2026, may be the first. Until then the strongest evidence anyone has is a result they ran themselves with the failures left in. Ask every vendor for the four things above, us included.
Common questions
Which AI security vendors publish their test results?
As of October 2026, F5 (through a SecureIQLab test it commissioned), Pipelock, Deconvolute Labs, Lakera, Meta, Protect AI and Mountain Theory publish results for their own products or defenses. AgentDojo, Mindgard and BELLS-O publish results across products. SecureIQLab's first non-commissioned comparison of AI security products is listed for November 18th, 2026.
Do any AI security vendors publish their false positives?
Pipelock prints 2 of 76 benign cases wrongly blocked, and Deconvolute Labs prints 7 of 150 benign samples flagged, each with a harness anyone can rerun. F5's commissioned report includes a false-positive figure. BELLS-O reports false-positive rates across 28 systems from 17 providers.
Is a commissioned test independent?
Partly. The lab runs the test, and the vendor pays for it and decides whether to publish. F5 says it had no involvement in test design or execution. A non-commissioned comparison, where the lab picks the products and publishes regardless, is the stronger kind.
More answers
The product, if you want it
Mountain Theory is the check described above, shipped. It works with any AI you already use, in your environment or the cloud.