As we all execute testing programs whether it is a integration, system or user acceptance testing phase, the two questions that consistently arise are:
- Is the testing phase on schedule?
- When are we going to complete the testing phase?
For many people, the questions are answered with statments such as:
- The test cases will be executed by the end of the week
- If the developers can solve all the bugs by a certain date, I will complete executing the tests by a certain date
In a nutshell, the responses tend to be relative to some other event while the people asking the question are looking for an absolute answer. How do we quantify the response in terms we can defend? I use a set of two metrics and trends over time for each that address the fundamental questions. These metrics are test case pass percentage (testing progress) and trend over time and open bug count and trend over time. These two metrics provide a quantitative measure and a backup measure so the project manager has two data points that point to the same timeframe. These metrics assume test phase exit criteria of all test cases pass and all severity bugs are closed which I find typical.
Let's start with the Test Progress Metric which documents the percentage of all test cases defined and passed for the testing phase. The numerator is the number of test cases passed and the denominator is the total number of test cases in the test plan. Over time, the number passed is recorded on a daily basis and percentage plotted over time. This trend line is then projected ahead to 100% for the completion date of the test phase. The trend of test case passing follows a linear pattern for most of the curve so any linear projection method affords a good view of the completion date.
The main advantage to this metric is it combines all the major elements that impact test phase completion but are either unknown or not easily measurable. These include:
- At any point in the testing phase, the number of additional bugs that will be found.
- The rate at which developers will close bugs
- The rate at which testers can regression test repaired bugs
I have seen many organizations attempt to measure these parameters or define quantatively these unknowns but with little success. It is more cost effective to use the defined metrics which naturally address all these questions.
Stability in the total number of test cases is critical here as instability in the denominator creates an unstable trend line so any risks associated with test plan instabiity need to be mitigated up front. If the instability occurs, the best approach is to recalculate the daily completion percentages, replot the curve, and the projections.
The second metric is open bugs and the trend line. Similar to the testing progress metric, the number of open bugs is recorded on a daily basis and plotted over time. Unlke the testing progress which uses a percentage, the open bugs is an absolute number. The severity level of the bugs needing closure for the test phase to end should be the set used in the plot. In a typical project, this is the severity level 1 and 2 bugs but it could differ depending on the individual organization and the bug management process established.
The open bug curve will follow a hypergeometric distribution which has a big hump early and a long linear tail as it approaches zero. This linear section is where the PM can also use any linear projection method to determine when the bug count will go to zero. Given the upward portion of the curve early on, this curve is only helpful after 60%-70% of test cases are pass. It will typically fall very close to the testing progress curve and provide a backup to the date projection derived from testing progress.
I have found these metrics to be very helpful in answering the questions from sponsors and are easy to defend. If anyone would like to see examples of these trends, feel free to contact me either on the blog or twitter @mdgibson. In a future post, I will walk through some test phase management techniques and how they compliment the testing metrics.
Comments
You can follow this conversation by subscribing to the comment feed for this post.