In my previous post, the discussion centered around a minimum set of metrics needed to effectively manage a testing program whether that program is an integration, system, or user acceptance test and exclusive of the development model used. In nearly all cases, a project will have a more formal test phase in which a set of pre-defined tests are executed with the outcome being a move to production. Given the assumption of utilizing the minumum metrics set defined in Testing Metrics, several management principles come to mind to move through the tests in a quick and efficient fashion.
The managment principles that have assisted me are:
- Focus on completing the maximum number of test cases between builds
At the outset of any test program, the complete set of test cases must be run and no bugs exist. At the point that bugs are issued and fixed, the questions become when to rebuild the test environment and the extent of the regression tests. My preference is to execute as many test cases as possible before rebuilding the environment and ideally execute the entire set until all test cases are either passed or blocked by a bug. What the PM achieves is to identify the maximum number of issues and identify the most stable and least stable components of the solution. By maintaining a substantial queue of issues, the development community can maintain a steady effort without any wait time as this community will typically be the driver for schedule.
In most test programs, this will take several days and assuming the development community begins bug fixing immediately a set of bugs will be ready for a build. At the point where all test cases are either passed or blocked, the test program is ready for another build thus freeing up additional test cases for execution. This principle makes a steady work queue for both the testing and development communities and affords the highest likelihood of the quickest progress towards completing the test phase.
- Focus on the completion of tests and not fixing bugs.
Once periodic builds commence, the test program is in an iterative cycle where all the tests are run and the developers work to unblock the maximum number of test cases with each build. The testers are faced with a question of whether to regression test bugs or focus on the newly unblocked test cases.
With each build, I focus on the new unblocked test cases to get through all test case steps in the quickest possible time. What this achieves is to uncover as many bugs as possible and ideally getting to the full set of bugs as early as possible. Once the complete set of bugs is known, the PM has additional techniques to determine the ultimate questions of: Am I on schedule? and When will I be done?
Regression testing of the bugs will need to occur but is a secondary concern early in the test program and can occur after all test cases have run through all their steps. If the testers focus on bug fixing and not test case completion, it drives the test program into a state of constant regression testing where builds are requested frequently, the progress through the test cases both slows and becomes unpredictable, and the early test steps are executed many more times than needed as they are executed with each build. All of these ideas result in a slower and less predictable test program.
- Define bug severity or priority levels and stick to them
One of the cornerstones of any test program is the categorization of identified bugs and the communication to the development community as to how big a deal each bug is. Bugs are either categorized using a severity measure which speaks to the impact on the requirements associated with each bug or priority which speaks to the order in which bugs should be fixed. At the end of the day, I beleive either is a viable manner to identify the urgency of the fixes but the main theme is to publish the definitions well before the outset of a test program and stick with those defintions.
The tendency of many organizations is to define the levels but get more "flexible" with the definitions in the heat of the testing program. Individuals have their own points of focus and can stray from the definitions as their preferred functionality identifies issues. Maintaining the definitions enables the development community to focus on the issues that have the greatest impact on the overall requirements and as stated above enables the test program to move through all the test cases in the quickest fashion.
- Structure the tests around business functions
The structure of the test cases plays a big role in the reporting of progress through the test program. When speaking to the project sponsors, the conversations are more fruitful when speaking in the terms of the business functions. If the test program centers on taking sales orders, then a structure around web orders, store orders, returns, exchanges, etc makes sense to the sponsors vice a structure around a set of interfaces or technical solution components. By structuring in this fashion, the PM can report for instance that web orders are working except a detailed scenario such as multi-line orders when a single line has more than one delivery date.
This structure also typically dovetails nicely with functional individuals within the organizations and enables the PM to focus developers with an individual tester to more quickly understand the issues and confirm fixes.
- Manage using the metrics
The metrics defined in my previous post provide the PM a framework for priortization. When executing the maximum number of tests, the PM quickly finds a situation where all the issues cannot be fixed in each build and typically more of each severity/priority level that can be fixed. The question becomes where to begin. The bugs blocking test cases metric provides a quantitative measure of which bugs block the greatest number of test cases. By working issues first that are blocking the maximum number of test cases, the PM can unblock the maximum number with each bug fix and keep moving through the test cases in the quickest possible fashion.
In this model, there may be instances where a priority level 2 bug is fixed before a priority level 1. This may seem counter productive but is in sync with the notion of moving through the test cases in the quickest possible fashion and identifying all the bugs as quickly as possible.
By using these principles to define priorities during any testing phase, movement through the testing phase has become quicker, easier, and more predictable. All of these elements are important to the PM as validating proper operation of the solution in a predictable and quantitative fashion is a key element late in the project and a cornerstone to get a decision to move to production.
Comments
You can follow this conversation by subscribing to the comment feed for this post.