Until now their scan read the source code. The new one looks at the app the way an attacker does: through a browser, over the network, with no access to anything internal. The two find almost entirely different things.
- The new scan tests the app through a browser, with no access to the source code.
- It runs against a full copy in a separate environment, so nothing reaches real users.
- The findings of the two scans barely overlap.
The scan that reads your code found a user whose access had been revoked, while the app never rechecked whether the old sign-in still held. That kind of miss is hard to catch, because nowhere does it look like a mistake.
The scan that looks from outside found the admin dashboard at a guessable address, with no login in front of it at all.
The first one had read that same page and moved on, because nothing in the code was wrong.
Why the two do not overlap
Because they look at different things. Reading code finds broken logic. Looking from outside finds unlocked doors that the code says nothing about, because they are not broken. They are simply there.
A dashboard with no login is not a mistake in the code. It is a mistake in the assumption that nobody will guess the address, and addresses do get guessed.
This is what I liked more than the announcement itself: they ran the two against the same apps and admitted the findings barely overlap. They could have kept quiet and sold one of them.
The small print
An automated scan is not a full human penetration test and nobody claims otherwise. It catches the known shapes quickly and cheaply, while a human finds the strange ones.
But for somebody who stood up an app with an agent and will put real customer data in it tomorrow, the difference between a scan like this and none at all is enormous.