CWE Shows Automated Vulnerability Scanners Aren't Perfect

Via ComputerWorld -

Security experts working on the CWE (Common Weakness Enumeration) project claim that the initiative to create a central resource of software vulnerabilities for developers is gaining momentum.

Sponsored by the Department of Homeland Security (DHS) and maintained by a team of workers at nonprofit Mitre and other security professionals, the ongoing effort is roughly four months shy of publishing a final draft of its vulnerability encyclopedia, said leaders of the project.

Presenting at the ongoing Black Hat 2007 conference, CWE initiative leaders said they are busy aggregating and organizing the mountains of vulnerability data they have gathered and said they are working more closely than ever with applications security testing companies to help compare the abilities of various software scanning tools.

...

CWE's research will not list the names and performance results of the products it is testing -- provided by over 20 firms, including Cenzic, Fortify, SPI Dynamics, Veracode, and Watchfire -- but the work to compile a resource that offers developers an idea of the types of vulnerabilities missed by the tools should provide a great deal of value, Martin said.

Officials with the project said that they have been pleasantly surprised by the variety of methods employed by the commercial scanning tools and the different types of flaws found by the various products. Going into this phase of the research, the group expected to find that many scanning systems identified the same types of issues, said Sean Barnum, director of knowledge management at Cigital, a software quality assurance company.

The tests also revealed that the products were looking for only 45 percent of the 600 common vulnerabilities that have already been entered into the CWE index.

"We found that less than half of what we already have in CWE is covered by these tools, so this helps prove that there are a lot of known issues out there that aren't being addressed," Barnum said. "We also thought that the tools would look for the same types of things, but they are actually very different, and there's not a lot of overlap; that's something that developers need to be aware of as they choose tools; you want to right set for aggregated coverage."

------------------------------------------

Tools are good, but they aren't the silver bullet of security either.

Tools will make a good security tester better, but they won't make a bad security tester good.