Why this matters
A benchmark is only as honest as its answer key. Every ground-truth row in RealVuln is a claim — "this line is a vulnerability" — and a scanner is rewarded for agreeing with it and penalised for disagreeing. That works when the claim is solid. It stops working the moment we write down a row we are not sure of ourselves, because from then on we are grading scanners against a coin toss and calling the result a score.
Until 3.0.0 we had two ways to handle an uncertain location: label it anyway, or delete it. Both are wrong in ways that matter to anyone using the numbers. Labelling it manufactures false negatives for careful scanners and false positives for cautious ones. Deleting it hides the location from anyone who wants to check our work, and quietly makes the corpus look cleaner than it is. Reviewing 74 TypeScript/JavaScript applications for this release produced enough of these cases that we could no longer pretend the problem was small.
What we did
The ground truth now has three states instead of two. A row is scoring (a real vulnerability, counted), a false-positive trap (a location that looks vulnerable and is not, counted the other way), or non_scoring: a location we reviewed, could not settle from the source alone, and chose to record with a written reason rather than guess at. There are 176 of them in 3.0.0, all in the TypeScript/JavaScript corpus; the 2,182 Python rows that were already published are unchanged and all still score.
The scorer treats a non-scoring row as invisible in both directions. A scanner that reports one is not charged a false positive. A scanner that misses one is not charged a miss. Critically, the row takes no part in matching at all: the matcher pairs each scanner finding with at most one ground-truth row, and if a non-scoring row were allowed into that pairing it could absorb a finding that should have matched a real vulnerability next to it — silently turning a true positive into a miss. We found and fixed exactly that behaviour while building the release, and it is now covered by a regression test.
The bar for marking a location non-scoring is deliberately narrow. It is for cases where the question "would a production reviewer call this a defect?" is genuinely open — most often because the application is a deliberately gamified training target and the behaviour is a challenge mechanic as much as a flaw. It is not for locations that are merely awkward to label, and it is never a way to excuse an exposure just because the surrounding application is intentionally insecure: being inside a vulnerable app is not evidence that any particular line is deliberate. Every non-scoring row carries its reason in the data file, so anyone can disagree with a specific call and rescore.
What we observed
The non-scoring rows cluster almost exactly where you would expect them to. 64 of the 176 are in OWASP Juice Shop and 37 in its Goof fork — applications whose challenge design deliberately blurs the line between "vulnerable" and "intended". The largest count outside those two is 23, in one generated CRM; only 12 of the 74 TS/JS repositories carry a non-scoring row at all, and 62 carry none. This is a property of Juice Shop, not of our reviewers: in an ordinary codebase the same standard almost never fires.
The effect on scores is visible and, we think, correct. On Juice Shop alone, 33 of Kolega DevSec Max's reports land inside non-scoring locations and are counted as neither hit nor miss. Under the old two-state rule those 33 would have been either 33 true positives or 33 false positives depending on which way we had guessed — a swing of several F3 points on a single repository, decided by a judgement call rather than by the scanner.
Why we think that happened
The alternative designs all leak the reviewer's uncertainty into the scanner's score. Partial credit ("count it as half") still rewards or punishes a scanner for a decision we did not make with confidence. Dropping the rows makes the corpus unauditable. Keeping them, labelling the uncertainty explicitly, and removing them from the arithmetic is the only option that leaves both the data and the scores honest. It costs us a slightly smaller denominator on the TS/JS tab; we think that is the right price.
What's next
The Python corpus has no non-scoring rows today because it was reviewed before this state existed. We will re-review it against the same standard rather than assume it is clean; if that produces non-scoring rows, the Python scores will move slightly and we will say so. False-positive traps for TS/JS — the counterpart mechanism, for locations that look vulnerable and are not — are the next ground-truth job. As always, every 3.0.0 figure is frozen at realvuln.com/v/3.0.0/ and regenerable from the repository.