Standards for Software Quality Assurance
Start course

This course covers section one of CSSLP Domain Five and looks at security quality assurance testing. You'll learn about important and foundational concepts on the process and execution of testing, topics regarding quality and product integrity, and various other considerations.

Learning Objectives

Obtain a solid understanding of the following topics:

  • Security testing use cases
  • Software quality assurance standards
  • Testing methodologies and documentation
  • Problem management
  • The impact of environmental factors on security

Intended Audience

This course is intended for anyone looking to develop secure software as well as those studying for the CSSLP certification.


Any experience relating to information security would be advantageous, but not essential. All topics discussed are thoroughly explained and presented in a way allowing the information to be absorbed by everyone, regardless of experience within the security field.


So, what do we have? Standards for software quality assurance can vary. Now, quality is the term we use most frequently and it is a term of many definitional shades. It holds the meaning of goodness of fit for its intended use according to specified requirements, we of course have the ISO 21827 and the OSSTMM. Now, when we think again of quality we need to think carefully about what that means. And as I've already said, we're talking about the goodness of fit for its intended use.

Now, what this means though, is that if we take, for example an apple, now, the shine on an apple or the brightness of the gold that we might be where working with does not improve the item's quality as such, in the strictest sense. So long as that item, whatever it may be measures fully up to the set of standards and requirements that were applied to its making, then its quality is said to be appropriate or fine.

Now, the quality of the product cannot be assured when it comes to software is so simply because it appears to be performing as intended. It must be verified as working as intended as I said before, taking nothing for granted. And we must do this with validated evidence that proves in fact that it does.

So let's examine the ISO 9126. Now, it establishes the basis of a quality program for software. Now, this particular standard focuses on modeling for both operationality and security and it covers these areas, the functionality, reliability, utility, efficiency, maintainability and portability of software. Now it requires that each attribute has the criteria set, the metric that goes along with it and they must bet tested appropriately as defined by what quality would mean in the particular context. And that of course refers back to the requirement specification and the related metrics. Now, as I had mentioned, and this is simply to make sure that we cover everything thoroughly the 9126 was superseded by the 25010 in 2017.

Another standard that we can apply is the system security engineering capability maturity model an adaptation to the software development world of the traditional capability maturity model concept. Now, this encapsulates the software security engineering practices that we find in the ISO 21827 of 2008. Now, this is intended as a tool to evaluate the quality and maturity of software security engineering practices over the systems development life cycle of which the software development life cycle as a subset.

Now, we have other standards that accompany the 21827 and these alongside the 15288 and the new 25010, these are the standards models used to evaluate software engineering maturity in any entity. Now, one question that may be occurring to you is the 21827 still valid given that it was written and published in 2008? As the ISO, like NST often does they reevaluate the standards that have been published seeking to determine if any updates or changes of significance are necessary.

The 21827, the SSE-CMM was reevaluated in 2020 and deemed to still be valid and correct. So let's take a quick examination of what it looks like now in 2021. As all traditional CMMs are we have five levels. The SSE-CMMI level one, like them all refers to the first level as the initiating. Things that are performed irregularly haphazardly ad hoc, and are in shape to be governed with my greater organization and rigidity.

Having successfully moved past the level one we move into level two, which again refers to the common term repeatability. Here, we've instituted controls that allow us to plan and track what we're doing. And this forms the basis for the lessons learned that will carry forward to level three. Here we go for more definition. Here we have well defined processes, things that facilitate not only repeatability but tracking and the ability to improve them.

In addition to this, they have the ability to establish metrics which will serve to track how well we are doing and help us measure our progress yet more refined. Moving to level four, this is where we take those metrics and we manage the process in a quantitative fashion. Here we use the metrics to measure and control what we're doing so that we can see where things are working well and where things are not working so well.

Moving on to level five, here's where we are not only doing the job well, regularly, consistently, we are also looking at improving the process itself. So we have a two pronged approach here by the time we've reached level five. Let's do it right. Let's do it right all the time and let's see if we can't improve how doing it to make sure that we continually improve the product and our methods for producing it. So here we have, what is commonly referred to as the OSSTMM. OSSTMM of course stands for Open Source Security Testing Methodologies Manual.

Now, the Institute for Security and Open Methodologies, ISECOM, which operates out of the University of Barcelona in Spain and has a partner here in the US. Now, they developed this standard and this is a peer reviewed testing methodology for conducting security tests and how to measure the results using defined and applicable metrics.

One of the most important things that it does is it defines a scientific methodology for the accurate characterization of security through examination and the correlation of test results in a consistent and reliable way. I am sure that you're beginning to notice a pattern of repetition here. Not only are we going to build it and work with it properly, but we're going to continually measure it so that we can repeat the good things that we do and stop doing the bad things that we have been doing.

In addition to providing the methodology for how the software can be designed to and built and then have its quality measured, The OSSTMM also provides guidelines for auditors to perform an assurance audit to show that the tests themselves did the job that they were thorough, complete, which are not always the same things, that they were compliant and that the results of the test were quantified correctly that they too are reliable, that they are consistently performed, that in other words, as the test itself is repeated in the correct way, that it would essentially deliver the same results. And that they are accurately representative of the subjects being tested. Now, OSSTMM and ISECOM, its organization, does offer training and internationally recognized certification of a variety of disciplines along this area.

About the Author
Learning Paths

Mr. Leo has been in Information System for 38 years, and an Information Security professional for over 36 years.  He has worked internationally as a Systems Analyst/Engineer, and as a Security and Privacy Consultant.  His past employers include IBM, St. Luke’s Episcopal Hospital, Computer Sciences Corporation, and Rockwell International.  A NASA contractor for 22 years, from 1998 to 2002 he was Director of Security Engineering and Chief Security Architect for Mission Control at the Johnson Space Center.  From 2002 to 2006 Mr. Leo was the Director of Information Systems, and Chief Information Security Officer for the Managed Care Division of the University of Texas Medical Branch in Galveston, Texas.


Upon attaining his CISSP license in 1997, Mr. Leo joined ISC2 (a professional role) as Chairman of the Curriculum Development Committee, and served in this role until 2004.   During this time, he formulated and directed the effort that produced what became and remains the standard curriculum used to train CISSP candidates worldwide.  He has maintained his professional standards as a professional educator and has since trained and certified nearly 8500 CISSP candidates since 1998, and nearly 2500 in HIPAA compliance certification since 2004.  Mr. leo is an ISC2 Certified Instructor.