Part of my early career was spent in Quality Assurance (QA) looking after divisional software and hardware procedures and auditing very large private and public sector projects. Before moving on to lead similar software development projects, the time in QA taught me much about the benefits of standards and the importance of appropriate processes. Later in my career - as a business consultant - I was to revisit this when I spent some time as a Software Capability Maturity Model (SW-CMM) assessor.

From an objective viewpoint, the body of knowledge that has been refined in the IT industry is invaluable to how we should build systems for the Web and, although the Web presents us with additional challenges, many of the principles from the body of knowledge are as applicable today as they were 20 years ago - just because the Web appears to be different, it doesn’t mean we have to reinvent the body of knowledge, just refine it further to encompass the additional challenges.

Looking specifically at Web applications (i.e. websites, whether Internet, Extranet or Intranet), the greatest risk for any organisation using them is one of security.

Open Source software is potentially more vulnerable to security attacks than proprietary software because the code is openly available to everyone - including hackers who can find and exploit security weaknesses to their benefit (I think of these people as “test teams gone bad”). However, one of the benefits of Open Source software is that there is a wide community of supporters who find, report and fix any problems that are found, and so resolutions for security problems are usually quickly implemented but require regular upgrades to benefit from them. This is not to say that Open Source is a bad choice - it is often an excellent choice - but clients (and agencies) need to understand and balance the risks before making decisions about which systems to use. Unfortunately, there does not appear to be a uniform standard for assessing the maturity of a web application.

In the SW-CMM there are 6 levels of maturity, from zero (meaning none) through to five (being the highest level). The level of maturity is applied to the organisation that develops the software applications, and is based on the processes that are followed and the systems that are in place. The theory being that the more mature the organisation on the scale, the more robust the application should be. However, this is not a direct relationship, only an implied one.

If we were able to apply the principles of the Capability Maturity Model to software applications (in general) and Open Source Software applications (more specifically) we would be able to develop a solid foundation for assessing the maturity of the software product itself. We could then compare this with our own requirements and so decide if the product is a suitable fit to the overall project aims.

Projects such as Sourceforge.net provide an indication of a product’s development stage (release status) by indicating whether the product is in planning, in development, at alpha release, beta release, etc. To the casual observer, it is unclear exactly how products on SourceForge reach these status levels (I only spent 10 minutes looking for it, but I would have expected to be able to find it quickly). Unfortunately, the release status does not tell us how secure the product is, and since security is the greatest risk on the Internet, it should be embodied in a product’s maturity rating, or be a separate rating in itself.

What I would like to see is some form of Security Maturity for released products - particularly Open Source - but also proprietary (Windows is well known for its vulnerabilities yet the source code is not openly available to the general masses). The question is then one of how to measure it.

Websites such as Security Focus provide long lists of security vulnerabilities of Open Source software produced by a myriad of organisations. However, there is no abstraction of this information into a relative score that can be used as a tool for pre-selection of applications based on project requirements.

In my opinion, the metric does not need to be complex.

The key piece of information is the list of bug reports for an application. These would need to be classified as “security related” and “non-security related”. Security related bugs would then need to be quantified against a standard list of security errors, for example cross-site scripting vulnerability, SQL injection vulnerability, code injection vulnerability, filesystem vulnerability, authentication vulnerability, etc. Each one of these classifications would have a defined impact according to the SMM standard. By counting the number of vulnerabilities of each type we would have an idea of the current security status of an application. However, if there are no current vulnerabilities, this does not mean the application is not vulnerable. It just means we haven’t found the next weakness.

Consequently, we should factor in both time and the number of installed versions of the application into the formula.

When the number of security vulnerabilities increases over time, this suggests that the application is more vulnerable and so the security maturity would decrease. If the number of detected vulnerabilities decreases over time, we would infer that the application is more secure. The reason for this is that if we are detecting less vulnerabilities across the installed user base, we can assume the system is more stable and less open to security vulnerabilities.

When an application is young - ie only recently launched and/or with very few live installations - it would be difficult to assess the security maturity accurately because it would only take a few vulnerabilities to radically alter the security maturity score. For this reason, young applications (defined as, for example, less than 1 year and/or 500 installations) would only warrant a bracketed rating as an indication of the potential security maturity because of the potential fluctuations that a few vulnerabilities would cause in the value.

I think a methodology of this nature would give much greater confidence for teams or organisations looking to select an off the shelf application, but it relies on software producers being open about the security maturity of their application. For professional organisations, this should be a benefit and a unique selling point, and for Open Source organisations it helps people make a decision between the Open Source system and a similar proprietary one.

By creating a level playing field (an equal Security Maturity Model standard) for all applications, Open Source and proprietary alike, we cut through the misinformation that we often find in the Internet. Open Source applications often have much higher numbers of installations compared to proprietary rivals (for example bulletin board software), plus the Open Source versions are discussed at length (both good and bad) in public forums. However, the proprietary products have smaller user bases and often don’t have the public presence. Consequently, searching for information on bulletin board software can easily bias a decision one way or the other because there is no way we can measure either product equally.