Moshe Cohen and Jon Chard both work in the Rational division at IBM Software.
The challenge to any software development effort is that qualitymanagement is a tricky balancing act that must factor in time, cost, andrisks. Get it wrong, and you could face issues ranging fromunsustainable costs, missed windows of opportunity, and unhappycustomers, to a massive recall or the complete failure of a system at acritical moment. Get it right, and you can achieve a positiveoperational return on investment from efficiencies gained in developmentactivities. With effective quality management, you can also createopportunities to deliver critical (but difficult-to-qualify) benefits,such as improved marketshare, higher customer satisfaction, andincreased brand equity.
What’s the problem with quality?
Complexity is now a given in many products and systems. Software,microelectronics, sensors, and mechanical technologies are oftencombined to create products that can measure conditions, respond tochanges, and interact with other products, people, and IT systems in newways.
Consider the potential overall effect of quality issues: In manyindustries, the entire success of an organization can hinge onconsistently solving complex challenges. Yet success and brand imagecultivated over many years can evaporate quickly when something goeswrong. Examples are plentiful: an aerospace company faced a $1 billionloss when a rocket self-destructed because of a bug in a guidancesystem. In the healthcare industry, software problems and poor qualitycontrol in cancer treatment systems designed to deliver preciseradiation treatments have led to tragic consequences for a number ofpatients. In these types of cases, what may seem to be a small defect isactually critical to the mission.
One of the biggest challenges related to quality management ishow to invest intelligently to minimize risk, given economicconstraints. Figuring out how to relate quality to outcome and whatconstitutes the right level of quality for individual products, however,is not always clear.
The safest way to address quality problems early in the lifecycleis to use proven approaches and process frameworks, such as the Capability Maturity Model (CMM), Capability Maturity Model Integration (CMMI),agile development, or other general or domain-specific approaches.Focusing on best practices that can deliver a positive return provides away to separate practices that will deliver tangible business results.
During any software delivery process, “When do we release?” is akey question with no simple answer. You must consider project-specificvariables, such as the cost of delays, the opportunity value of earlydelivery, marketplace quality expectations, and the costs associatedwith defects. Ultimately, the delivery strategy will be based on theactual or perceived importance of each variable.
Schedule-driven delivery effectively implies “deliver on time,regardless of other factors” and is often used in consumer products tohit marketplace windows or in contractually governed environments, wheredelay penalties may apply. Quality issues are often magnified, giventhat software contractors frequently get paid on a time and materialsbasis regardless of the quality of software they deliver. In many cases,you may even end up paying extra for them to fix their own defects, sothe potential costs of defects can add up quickly.
According to the Carnegie Mellon Software Engineering Institute, “Data indicate that 60-80% of the cost of software development is in rework.”
Quality-driven delivery can also be costly but for differentreasons. The release timing for this approach is governed by achievingthe right quality — but how do you define that? Zero defects ispractically impossible to achieve, given that there is no way todetermine how many defects still exist in a piece of code or theprobability of detecting those defects in use. A target based on defectsfixed might be more realistic — but it’s still impossible to know thenumber of remaining defects in the product.
To cost-effectively balance quality-risk versus time-to-marketconsiderations, a risk-driven approach may be ideal. A risk-drivenstrategy is a refinement of a quality-driven approach that optimizesrisk exposure against development cost and time.
The optimal time to release is when the total risk exposure isminimal, typically around the time where the risk associated withcompetitive threats starts to outweigh the risk reduction associatedwith further quality improvements, as illustrated in Figure 1.
The best time to release can vary widely: Move too soon, andcustomers may be upset with quality issues. Wait too long, and yourcompetition will beat you to the punch and dominate the space. In theend, the optimal time to release lies where your overall risk isminimal. Consider the following examples.
- Mass-market products — In the consumer electronics marketplace, time to market is critical to product success. Here, the opportunity cost of delaying a release to improve quality can be high, thus pushing the optimum release point to an earlier date.
- Safety-critical applications — In applications such as flight control software, quality may be the most critical parameter, given the potential size of loss caused by a defect. In this case, the optimum release point would be a later date.
- Systems with high-availability requirements — Mobile communications in the telecommunications industry, for example, have extremely high availability requirements because of the commercial implications of system outages. As a result, the potential loss resulting from a defect makes quality critical, pushing the optimum release point to a later date.
In practice, the required time-to-market is always ahead of youroptimal time to release, where your overall risk exposure is the lowest.The only way to hit your time-to-market window without increasing therisk is by accelerating your risk reduction over time.
Quality management versus testing
If a faster reduction in risk is the goal, how do you achieve it? Theanswer is not testing, which is focused solely on discovering defects.Rather, an approach that is focused on reducing risk throughout thelifecycle is need. This is what quality management is all about. And bychoosing quality management best practices that deliver a quick returnon investment, you can focus on reducing risks in your project knowingthat your management is behind you and supporting you.
Traditionally, testing has been a late-stage activity in thedevelopment lifecycle, conducted between the construction of softwareand its eventual release. Quality management can be considered as itsown lifecycle within the overall software development lifecycle, asshown in Figure 2. Its activities take place across the developmentprocess and are synchronized with the development process in key points.
The entire process is closely linked to change and configurationmanagement because discovered defects can be considered as changes thatmust be managed along with other changes progressing through the changemanagement process.
There are many strategies for reducing risk, so it is essentialto distinguish between those that are good ideas and those that alsodeliver a positive returns. The following are six "best practice"strategies that are proven to deliver a return on investment.
1. Drive testing from requirements. Linking testdefinition to requirements can help ensure that what you ultimately testreflects requirements. This approach can help reduce risk by ensuringthat tests demonstrate compliance with requirements. However, manuallybuilding such traceability is time-consuming. A dedicated qualitymanagement solution with support for capturing traceability could speedup the process by a factor of ten, typically leading to savings in theorder of $150-300K for projects with approximately 5000 requirements and10,000 test cases.
2. Reduce planning overhead with collaborative quality management.Quality management is a lifecycle activity that affects everyoneinvolved in a project. As such, managing the testing plan is acollaborative activity involving all stakeholders. A central repositorywhere stakeholders can share and access updates and a “single version ofthe truth” can significantly cut the time spent on collaborative tasks.For a team of 100 testers, this could lead to savings of around$750,000 over 12 months. It also means that a team of 92 testers coulddo the work of 100, leaving the other 8 to accelerate the schedule.
3. Prioritize testing according to risk. Although youcannot completely remove risk from a development program, you can manageand measure it by taking proportionate mitigation actions. Thisapproach prioritizes the execution of the test cases, where thoseaddressing high risk requirements are held above those with a lowerrisk. This approach leads to significant reduction in the total numberof test case executions, corresponding to large savings.
4. Integrate testing and quality management to improve accuracy and repeatability.Integrating both manual and automated testing within the qualitymanagement environment can deliver efficiency and quality benefits,including improved management of tests; likelihood of executing thecorrect set of tests; and collaboration resulting from betterinformation on testing status and results
5. Automate reporting to improve efficiency, consistency and decision making.Reporting is a key element of the quality management process because itfacilitates informed decision making based on information capturedacross the process. By automating the process, you can producesubsequent reports with minimal effort; therefore you are more likely toproduce reports more often, helping to detect potential problemssooner. Automation also helps improve the accuracy and consistency ofreporting — factors that can positively influence project management,productivity, and quality.
6. Improve efficiency by applying smart defect management to eliminate duplicates.As project complexity increases, the number of detected defectsincreases, as does the occurrence of duplicate reporting of defects.Identifying such duplicates is a time-consuming activity that adds novalue to the project. Tooling that automates the detection of duplicatedefects as testers log them can therefore help reduce overhead costs andshorten delivery timelines.
The overall effect of risk reduction strategies
Each of the best practices discussed here can produce significantresults. But collectively, they also contribute to earlier defectdetection. And as the cost of fixing defects is lower the earlier it isfixed, applying these best practices leads to significant savings due toearlier defect resolution.
As an example, let’s assume a team that is at a CMMI level 2, anda project with 1000 defects detected by unit and integration testing.Applying the best practices discussed above can lead to the earlydetection of over 900 additional defects.
To realize these benefits, organizations would do well to investin software that provides a collaborative, customizable, web-based testand quality management hub for the quality management lifecycle. Whetherthey are down the hall or across the globe, quality professionals andother decision makers can use quality management software to collaborateon test planning and management, risk-based testing, and defectmanagement, including duplicate defect detection.
Conclusion
The momentum of technology, coupled with customer demands, is pushingorganizations to deliver ever-smarter products. Given the addedcomplexity in development processes, finding ways to continuallyoptimize time, cost, and quality is critical to successful innovation. Arisk-driven development approach can help you balance quality risk withtime-to-market imperatives, and it also can do so while delivering asignificant financial returns. The cost benefits are achieved throughaddressing defects earlier in the development process when they are lessexpensive to fix, as well as through productivity and efficiency gainsin development processes.
What’s more, such an approach can also support long-term benefitsthat cannot be easily quantified but are key to success, includingcustomer satisfaction, reduced in-service costs, and a strong reputationand brand.
