OWASP Sentinel: benchmark results

Model gemini:gemini-3.1-flash-lite, temperature 0.3. 450 Benchmark cases (stratified sample, Java), 388 findings triaged per run. Only confident false positives are dropped.

In plain English: precision is how many flagged issues are real; recall is how many real issues get found; false-positive rate is how many safe code samples are wrongly flagged. Higher precision and recall are better; lower false-positive rate is better.

Configuration	Precision	Recall	FPR	Youden
A. Opengrep (sast-rules)	0.675	0.600	0.289	0.311
B. A + custom rules	0.676	0.889	0.427	0.462
C. B + LLM triage (1 run)	0.712	0.889	0.360	0.529

Per Benchmark category (precision / recall / FPR)

The OWASP column comes from owasp_2025.yaml, checked against the official OWASP Top 10 2025 CWE lists. Each cell rests on about 20 real and 20 fake cases.

Category	OWASP 2025	A. Opengrep (sast-rules)	B. A + custom rules	C. B + LLM triage (1 run)
cmdi	A05	0.60 / 1.00 / 0.67	0.60 / 1.00 / 0.67	0.62 / 1.00 / 0.62
crypto	A04	0.82 / 0.86 / 0.19	0.82 / 0.86 / 0.19	0.82 / 0.86 / 0.19
hash	A04	1.00 / 0.76 / 0.00	1.00 / 0.76 / 0.00	1.00 / 0.76 / 0.00
ldapi	A05	0.54 / 1.00 / 0.86	0.54 / 1.00 / 0.86	0.54 / 1.00 / 0.86
pathtraver	A01	0.57 / 1.00 / 0.76	0.57 / 1.00 / 0.76	0.58 / 1.00 / 0.71
securecookie	A02	1.00 / 1.00 / 0.00	1.00 / 1.00 / 0.00	1.00 / 1.00 / 0.00
sqli	A05	n/a / 0.00 / 0.00	0.61 / 0.52 / 0.33	0.69 / 0.52 / 0.24
trustbound	A06	n/a / 0.00 / 0.00	0.58 / 0.90 / 0.67	0.63 / 0.90 / 0.52
weakrand	A04	n/a / 0.00 / 0.00	0.88 / 1.00 / 0.14	0.95 / 1.00 / 0.05
xpathi	A05	n/a / 0.00 / 0.00	0.67 / 0.93 / 0.47	0.67 / 0.93 / 0.47
xss	A05	0.57 / 0.81 / 0.62	0.57 / 0.81 / 0.62	0.71 / 0.81 / 0.33
overall		0.68 / 0.60 / 0.29	0.68 / 0.89 / 0.43	0.71 / 0.89 / 0.36

Limits

Baseline on the full OWASP Benchmark

Opengrep + GitLab sast-rules, no LLM. 2740 cases, 1579 findings. Precision and recall are shown only where ground truth exists.

Full Benchmark

Category	TP	FP	TN	FN	Precision	Recall	FPR	Youden
cmdi	126	94	31	0	0.57	1.00	0.75	0.25
crypto	105	27	89	25	0.80	0.81	0.23	0.57
hash	89	0	107	40	1.00	0.69	0.00	0.69
ldapi	27	27	5	0	0.50	1.00	0.84	0.16
pathtraver	127	115	20	6	0.52	0.95	0.85	0.10
securecookie	36	0	31	0	1.00	1.00	0.00	1.00
sqli	0	0	232	272	n/a	0.00	0.00	0.00
trustbound	0	0	43	83	n/a	0.00	0.00	0.00
weakrand	0	0	275	218	n/a	0.00	0.00	0.00
xpathi	0	0	20	15	n/a	0.00	0.00	0.00
xss	182	118	91	64	0.61	0.74	0.56	0.18
overall	692	381	944	723	0.64	0.49	0.29	0.20

450-case sample

Category	TP	FP	TN	FN	Precision	Recall	FPR	Youden
cmdi	21	14	7	0	0.60	1.00	0.67	0.33
crypto	18	4	17	3	0.82	0.86	0.19	0.67
hash	16	0	21	5	1.00	0.76	0.00	0.76
ldapi	21	18	3	0	0.54	1.00	0.86	0.14
pathtraver	21	16	5	0	0.57	1.00	0.76	0.24
securecookie	21	0	21	0	1.00	1.00	0.00	1.00
sqli	0	0	21	21	n/a	0.00	0.00	0.00
trustbound	0	0	21	21	n/a	0.00	0.00	0.00
weakrand	0	0	21	21	n/a	0.00	0.00	0.00
xpathi	0	0	15	15	n/a	0.00	0.00	0.00
xss	17	13	8	4	0.57	0.81	0.62	0.19
overall	135	65	160	90	0.68	0.60	0.29	0.31

Benchmark categories with no rule coverage

OWASP Top 10 2025 coverage

ID	Category	Status	Cases
A01	Broken Access Control	Measured	268
A02	Security Misconfiguration	Measured	67
A03	Software Supply Chain Failures	Not covered	0
A04	Cryptographic Failures	Measured	975
A05	Injection	Measured	1304
A06	Insecure Design	Measured	126
A07	Authentication Failures	Not covered	0
A08	Software or Data Integrity Failures	Not covered	0
A09	Security Logging and Alerting Failures	Not covered	0
A10	Mishandling of Exceptional Conditions	Not covered	0

"Measured" means ground truth exists for at least one CWE in that category, not that the whole category is tested. "Not covered" categories have no metrics.