If the people you're paying to find weaknesses in the security system are assuredly never going to find a way to access internal data then how did you conclude you needed a pen tester in the first place? I mean, it's probably the right conclusion but only precisely because they'd find a way to access things they shouldn't be able to.
It's relatively common to have pen testers attack a cloned environment w/ sanitized data. This is especially true in cases where your policies (or those you've agreed to from customers) require you to present evidence that you are having a pen test done every X years.
We log all accesses and flows. So eg if our pentesters found a vulnerability in an endpoint, we can retrieve every post against that endpoint and (1) verify the pentesters didn't exploit it against prod, and (2) verify that it hasn't been exploited by anyone else.
Of course, that only works if the vulnerability is reported. There is no reason for the malicious actor to report the vulnerability they have chosen to exploit.
What percentage of the vulnerabilities discovered are independently discovered by multiple pen testers?
It sounds like you're suggesting that pen testers by default will not reveal discovered vulnerabilities with clients.
Then you talk about "discovered and revealed vulnerabilities". But, your first sentence talks about "discovered vulnerabilities not revealed".
What you may be wanting is a honeypot, where a pentest client intentionally puts some vulnerabilities of various exploit difficulty into the clone environment to ensure pentesters are doing their job.
> It sounds like you're suggesting that pen testers by default will not reveal discovered vulnerabilities with clients.
How so? Presumably most pen testers are working in good faith. But, if there is a malicious actor in their midst, that individual would not disclose any vulnerabilities they intend to exploit, no. What would be the point? That's just a really good way to get caught.
> Then you talk about "discovered and revealed vulnerabilities".
Yes, that's right. While it is theoretically possible for all your pen testers to be working together maliciously, if you are careful in your employment practices you can make this highly unlikely.
As such, if your data shows that 100% of all known vulnerabilities were independently discovered by multiple testers, then there is reasonable confidence that any malicious actor's failure to disclose a vulnerability will still be reported by someone else.
But if that figure is less than 100%, and especially if it is considerably less than 100%, then there is much more doubt cast on another pen tester in your organization's ability to find the same vulnerability. Here you have a problem.
You don't need to be, but there are some big advantages:
1. You get to test the flaws in an environment where nobody will raise an eyebrow. If you go straight for the production system, it is likely your early attempts will visibly show up in the logs.
2. You get paid to carry out malicious deeds. That's a double win.
Why do you think it would be silly to take the job?
The second two sentences read like excellent reasons why you should take the job (even if they are just a repeat what I already said in different words).
Then what do you need pen testers for? With an offer like that, any threats to your system will come work for you instead.
The reality is that you don't get paid well if the data is worthless. You only get paid well when the data is worth orders of magnitude more than what you're being offered. If you are inclined to break that law, that's a pretty nice carrot dangling there.
If you are so inclined, why wouldn't you take the job and report the not so crafty exploits to bring in the sweet, sweet paycheque and use the really juicy exploit to also go after the even sweeter data? It's a total win-win situation...
...unless you get caught, but if you are so inclined that's not exactly on your radar.
My claim is that people tend not to do crime if there's a very well-paid alternative, and I think I have pretty good empirical backing on that one. Also, our data is probably not worth that much. We do pen testing so we don't get popped and leak our customers data, likely losing some of our customer base (even if it isn't worth much, not having it leaked is); because soc2 essentially demands it; and because smart customers care more about pentests done by good firms than soc2.
>What percentage of the vulnerabilities discovered are independently discovered by multiple pen testers?
I'd warrant nearly all of them, though it may take a while.
If you have ever submitted or worked with a bug bounty program you will run into dozens of duplicates.
I've personally performed and overseen assessments in which the company had already done a complete blackbox pentest and wanted a second whitebox review to make sure the first company knew their stuff and validate they found the same bugs. Also did a few of the honeypot assessments in which companies put purposely vulnerable code in to make sure 'we are doing our job', I hate those most.
Depending on the testers speciality of course, the reports often found the same or similar issues.
Source: 15 years as a pentester, offensive security engineer, and now security architect.
> What percentage of the vulnerabilities discovered are independently discovered by multiple pen testers?
Zero because we patch them as soon as we are notified. Generally at the end of the test / before the retest, but if they found something serious they would notify immediately,
Patch production, sure, but naturally you would leave them in the pen testing environment for some time in order to collect data. No data and you’re just guessing. That’s fine for amateur hour, but not business.
