By Declan Ingram
Over the past few years we have seen more and more automated scanning tools being used as the primary source of application assessment. A couple of years ago, when we were S-A.com, one of the guys did a very comprehensive test of all the available scanners, and the results were mediocre at best. In fact, as a result of these tests, we decided at the time that they added little to no benefit to our testing tool-chain.
Recently, with the enforcement of PCI Web Application Security Assesment requirements, clients need to have the coverage for all of their applications and do not have the funds available for full manual testing.
The three that we have been looking at recently are AppScan, Acunetix, and Burp Professional. Burp is a little bit different, in that it’s primarily a manual assessment tool with some scanning features.
We have been judging the quality of these products based on false positives, false negatives, and code coverage. The applications have all been web apps: HTML, JSP, ASP, PHP, old, new, good, bad, ugly, etc.
The results were……interesting:
- All scanners needed a lot of manual work to get any reasonable amount of code coverage.
- There were a huge amount of false positives.
- There were many false negatives. (Probably more than we know
)
However, these flaws can all generally (possibly excepting false negatives) be negated with a qualified person running the scans, and verifying the results. So this is really not a problem, right? I mean, it’s how the vendors advertise their low false-positive and false-negative rates.
The big problem, as I see it, is that these applications are not sold or targeted to specialist testers anywhere near as much as they are marketing to coders and auditors that do not have the skills to use them effectively. This negates the whole idea and provides a false sense of security!
The outstanding product here is burp, it’s a semi-automatic scanner, so it requires a skilled tester to use, but it’s a fraction of the cost and is targeted at the right market to get results.
Seconded – Burp pro is awesome! Especially on value for money, compared to IBM and HPs offering.
These tools should be sold with a big warning notice.
Just checkout some of the marketing content around them. For a laugh at least.
I agree that even the better developers struggle with these programs. The progression towards better automated code testing is slow.
“he three that we have been looking at recently are AppScan, Acunetix, and Burp Professional”
Is there …mmmm… you know, one missing from the list here?
Hey Declan, you guys ever look at W3AF?
-C
The comparison tests for these tools when run stand-alone without experienced testers are worth zip in my opinion. The bottom line is that a few percentage points variation here or there is meanlingless if major vulns escape detection over and over again – app after app after app. (Akin to WAFs capabilities?). Understanding their limitations is key and positioning them into the SDLC requires that knowledge. Do they have a place on their own without experienced testers running them and analyzing results in more depth (plus using more manual analysis)?
@xntrik
yes, I’ve played with w3af, but not for a year or so. At the time, it didn’t have the flow of Burp.
@Anton
haha yes there is one missing. A very important one which is going to get a big going over soon – and you will know every little detail. These tests have been based upon our experiences directly with client requests.. AppScan marketing seems to be very successful!
@Declan what sort of data were you getting? Is this also a case of no matter how many tools you run, the sum of their results is never going to equal the number of vulnerabilities in an application?
sick of hearing these tools get the low lying fruit and that that is worth their purchase. How?
>sick of hearing these tools get the low lying fruit and that that is worth their purchase. How?
Easy. With tools, low hanging fruit are gone (some/most are), without them – they stay.
To many orgs, that is motivation enough to buy them, since they are never going to fix anything BUT the low hanging fruit.