Evidence helps.
The model still makes mistakes.
We tested ProveIt on 77 curl reports held back during development. It found useful evidence in both false alarms and real bugs. It also got two real bugs wrong.
What happened in the test
18 reports labeled AI slop by curl's maintainer. 59 reports linked to published vulnerabilities.
No slop report was marked likely valid. Of the rest, most needed human investigation, more information, or a policy decision.
See every finding
| ProveIt's finding | Labeled AI slop | Known vulnerabilities |
|---|---|---|
| References do not check out | 2 | 0 |
| Code contradicts the claim | 3 | 2 |
| Needs a policy review | 4 | 3 |
| More detail needed | 2 | 1 |
| Needs human review | 7 | 31 |
| Code supports the claim | 0 | 22 |
For the proportion flagged for lower priority, 95% bootstrap intervals are 11–50% for slop and 0–8% for real reports. This is a small test set.
Two real bugs deserved more attention.
Misread cookie behavior
The model treated cookie injection through a file named “none” as intended behavior.
Inspect the mistake → False alarm · CVE-2022-27776Misread a redirect condition
The model missed an authentication and cookie leak in the redirect logic.
Inspect the mistake →These failures are why ProveIt provides evidence for a maintainer to review and never closes a report.
17 judgments
were withdrawn.
The model produced 490 citations. 50 did not point to evidence it had been shown. Invalid citations are discarded; a judgment that loses required evidence becomes “Needs human review.”
A valid citation means the referenced line exists in the evidence. The model can still misunderstand it.
See a made-up quote get caught →What this test can tell us
One project. The current implementation supports curl. Results do not establish how well it works on other repositories.
Reading code is not running it. These results cover reference checks and model analysis. Optional offline sandbox reproduction is implemented separately and is shown in a saved example; it is not included in these scores.
Uncertainty is common. 32 of 59 real reports needed human review or more information. Three more needed a policy decision.
Dataset, method & cost
The full collection contains 49 reports labeled AI slop by curl's maintainer and 148 HackerOne reports linked to curl CVEs. Reports were split by a hash of their IDs: 120 for development and 77 for the held-out test.
Rules were developed on the development split, then evaluated once on the test split. Checkers and models receive only the report's title, body, date and attachment names. Triage discussion, labels and CVE mappings are withheld.
The test cost $1.34 in model calls, about $0.017 per report. This excludes optional sandbox reproduction and external documentation search.
Reference checks alone flagged 2 of 18 slop reports and none of the 59 real reports. Many misleading reports cite real code; the harder question is whether their claims match its behavior.