Mark Gibson; PgMP, PMP, IBM certified Executive Project Manager

Basic tenants for Enterprise Project Management

For those medium to large enterprise projects, having a set of basic tenants that keep me focused has proved valuable to run the projects and to assist the project teams in understanding the basis behind decision making.  Each of these tenants is unique but also are used in concert.  These tenants are:

• Customer collaboration is the key to a better solution

In today’s Enterprise 2.0 environment, openness and the use of collective intelligence are proving valuable avenues to achieving higher levels of success.  This has been true in the realm of enterprise project management for some time as the methodologies that include early prototyping of either commercial or custom software embrace this concept vice the more documentation centric approaches that produce large and detailed requirements specifications.  In my experience, putting more working software in front of the customer sooner enables a better review of the proposed solution in a faster time period.  The end result is less requirements change and shorter delivery time periods which is the position we all want to be in.

This collaboration is especially true in the commercial software realm where the customer business processes are mostly molded into the processes defined by the software.  This affords the project manager an opportunity to build working prototypes of the software to both educate the customer on the software features and engage customer personnel on the most appropriate manner to configure the software.  This results at least in a solution into which the customer has more ownership due to participating actively in how it was configured as opposed to being told how best to configure.  Most technicians like to have creative license in these phases.  The same is true for the customer technical resources.

• Requirements will change during the project

When operating within the document centric methodologies, we all strive to develop requirements that are stable.  In the 80s and 90s, we developed metrics around this concept to understand the risk when the requirements change.  The one constant is that the requirements did change and always will. Any approach we take to requirements gathering is going to be imperfect so planning for this change both in the budget and schedule are crucial to project success.  Planning extra development capacity or planning an incremental drop with limited functionality so the project manager can add to it when the requirements change is something I try to always do.

• Statements of Work and all project deliverable are firm commitments

In most cases, project managers are given statements of work upon which to execute.  These documents are written and approved by people other than the PM or the project team.  In some cases, this can put the PM into a difficult situation when commitments stretch the bounds of the possible or in timeframes that stretch our capabilities.  In all instances, these documents as well as all documents developed by the project teams are firm and binding.  In my experience and in rare cases, these documents are considered guidelines that can be renegotiated at any time.

When these renegotiations occur, the customer trust in the PM is diminished.  The natural continuation of the theme is if this is not a commitment then what else is also not a commitment.  The degradation of trust becomes an issue when any decisions are made and most acute when the difficult decisions are made.  On my projects, I treat the SOW as a document as a firm commitment and use the project documentation to add detail to the commitments defined within.  For example, if the SOW defines custom functionality for creating and modifying purchase orders, then the requirements define the number of screens, the screen layout, etc.  The requirements would not attempt to remove the purchase order modification functionality.  This can occur when the effort estimates by the project team are greater than by the SOW writers and creates an atmosphere where the reduction in scope is justified by the differing estimates.  Resisting these temptations will go a long way to building and maintaining the trust within the whole project team. When requirements do change as stated above, the burden or proof is on the PM.

• Satisfy the Project Sponsor(s) to gain acceptance

One of the key time periods on any project is when the customer is determining whether to accept the project. The key decision makers should be identified early and are typically the Project Sponsor.  In my opinion, this is the person or persons who pay for the project and will use the results of the project.  In many organizations, this can more than one person such as a business sponsor such as a CFO for a financials project and the CIO for the technical aspects of the project.

Where the confusion lies is the day to day contacts on the project are seldom these individuals. They are individuals who work for and implement the guidelines defined by these individuals.  I believe it important for the project manager to stay in close contact with the sponsors and understanding their strategic goals and issues so the project can achieve these goals.  In my experience, if the project achieves the strategic goals it will be considered successful.  This is not to say the tactical goals of the project team and all the detailed requirements are not important because they are.  When choosing between achieving a tactical goal and either a strategic goal or undermining a strategic goal, I will choose the strategic goal unless I have the consent of the sponsors to do otherwise.

When the time comes to make the decision on the project acceptance, most sponsors will take the input from the project team but make the decision themselves.  If the project manager has maintained close contact with and met the goals of the sponsor(s), the acceptance decision will be much easier.

• Plan for the unexpected… It is going to happen.

In every project I have worked, unplanned work occurs.  The reasons for this are varied and typically unique to the project but it happens nearly every time so the project manager has to plan for the unexpected.  What happens is a change in the triple constraints: schedule, budget or scope.

Planning for these items comes down to leaving extra time in the schedule and budget.  In an incremental delivery model, keeping the final drop with less than a full team’s effort affords the opportunity to move additional functionality into that release.  This is akin to padding the critical path a concept on which many people have written.  In the single release models, the padding of the critical path is the mechanism for planning against unplanned schedule issues or increases in scope.

I typically try to keep 10% of the budget back for contingencies.  Operational managers have historically used this concept for unplanned work as well.  The same concept applies well for the managers of projects.  The extra money can be used to increase staff to cover unplanned losses of staff, increases in scope, clarified requirements that increase effort, increases in actual effort over the original estimates, or even to pay for change requests.  The use for change requests has the added benefit of maintaining the original budget when the project team or sponsor receives unplanned requirements or changes in company policy impact the project.

In concert with tenant #1, I am completely open with my customers on this concept and collaborate with them on if or how to use the additional funds.  In the rare instances when the contingency is not used, the project is under budget which is a good position in which to be.

• Do what you say you are going to do.

This tenant seems like common sense but we all know common sense is not common.  This applies to even the small things such as accepting meeting requests, being on time for meetings, and publishing deliverables on time. For traveling consultants, this includes being on-site when agreed, not missing flights, or planning for flight delays.  In all too many cases, consultants use airline schedule and delays as reasons for being late to work.

This also applies to deliverables such as meeting all the requirements in a SOW or requirements specification, conducting scheduled design reviews when and how agreed.  The word of a PM should be sound and unwavering.  What we all know is that projects are built with many interdependent activities and the commitments we all make.  The plans are all built on these activities, commitments, etc so the plan and therefore project breaks down when we do not do what we say we are going to do.

When the difficult project decisions arise, I keep these tenants in mind and they both drive the decisions and make them easier by focusing me on what is truly important.

Requirements Documentation- Let me Count the Ways

As I finish preparations to begin a new project, I have spent some time discussing several methods of documenting requirements.  More specifically in a requirements collection phase, at what point and with what audience do we put pen to paper with the deliverable being a requirements document.

The discussions in which I have been engaged in the last week have defined four basic scenarios that can occur, each with their own advantages and disadvantages and degrees of passion about the method.

     Conduct all meetings first and document later

In this method, the project team conducts the days or week worth of meetings, meeting minutes are kept and published.  After all meetings are conducted, a subset of the team begins the writing.  The writing is typically accomplished by a business analyst but any team member(s) can also accomplish. 

The advantages of this method are: 1) decreased time defining requirements so less business user time, 2) complete focus on requirements without need to discuss verbiage, 3) during writing, rest of team can accomplish other startup tasks. 

The disadvantages are: 1)  longer approval cycle as requirements wording is left to the end,    2) results of discussions may be lost.  This is a greater risk the longer the requirements meetings occur,  3) writers must spend time initially preparing and then reviewing meeting minutes to write final document.  Initial sets of meeting minutes are non-value add as they are duplicative to the final requirements document.

     Conduct meetings and document weekly

In this method, the meetings are conducted for a number of days during week which is typically either 3 or 4 days.  The remaining 1 to 2 days are used to document the results of the week.  A subset of the team is typically used for the actual document preparation and is well suited for a business analyst. Typically, the emerging document is reviewed weekly and prior to beginning the new week of material.

The advantages of this method are: 1) all parties can focus on requirements definition without overhead of documentation, 2) writers have dedicated time to document at the end of each week, 3) less delay between definition and documentation so time lapse issues are less

The disadvantages are: 1) some time lapse issues and verbiage issues remain, 2) time to define requirements is extended as time is spent dedicated to documentation, 3) time lapse issues created additional uncertainty on the approval cycle length and thus the associated schedule risks, 4) end user subgroups may be required to attend sessions split across disjointed weeks.

      Conduct meetings and document daily

With this method, the writers keep up with the daily definition of requirements and document said requirements during the evening hours.  This enables up to a 5 day workweek to document which may be useful when limited time is available either all together or with a particular end user group.  A business analyst remains well suited to the writing tasks but technicians with good functional skills can also accomplish easily. Typically, the prior day's language is reviewed prior to beginning the day with new material.

The advantages are: 1) lag time between definition and language agreement is substantially smaller eliminating the loss of agreement over the lag time, 2) the team is better able to monitor progress as the definition and language are agreed upon in a more incremental fashion, 3) individual subgroups of users are needed for a confined time as they are not required to come back at a later time to reach agreement on language.

The disadvantages are: 1) writers are required to put in additional hours at night and will loose effectiveness due to fatigue, 2) the days can be disjointed as language is reviewed at the beginning of each day causing greater stress on the end users, 3) the time lag of even a day can result in loss of agreement on the material from the previous day.

     Conduct meeting and build document in real time

In this final method, the requirements document is populated in real time as the requirements are defined.  All parties agree review the language on either a PC projector and/or via a web-based conferencing solution such as NetMeeting or Webex.  As individuals articulate the requirement, the writer types the new requirement into the final deliverable.  If the need for diagrams is identified, the writer can shift to a presentation graphics or business process flow application such as PowerPoint or Visio.  In some cases, PMs may decide to have a short document review after the sessions are complete, but great care should be taken to ensure topics are not reopened.

The advantages of this method are: 1) progress is incremental and easily tracked against the available schedule, 2) once all material is covered, the document is ready for signature and all parties have agreed on the language, 3) no time lag is created between requirement identification and documentation, 4) each end user subgroup is needed for only a single period of time vice coming back for a document review.

The disadvantages are: 1) writers have great pressure of them to either type in real time while interpreting the requirement or creating a diagram in real time, 2) end user may feel additional pressure since requirements are documented in real time vice having some number of days to review after documentation is complete.

In summary, I prefer the build as you go method as it provides a better gauge of progress, keeps all project team members focused on the goal of completing the deliverable, and typically has a smaller approval tail then the other methods.  The key assumption here is that parties are held accountable to the verbiage written as they go along and wholesale revisions are not made at the end.  This risk is further mitigated by publishing the draft on a daily basis.

Test Management Philosophy

In my previous post, the discussion centered around a minimum set of metrics needed to effectively manage a testing program whether that program is an integration, system, or user acceptance test and exclusive of the development model used.  In nearly all cases, a project will have a more formal test phase in which a set of pre-defined tests are executed with the outcome being a move to production.  Given the assumption of utilizing the minumum metrics set defined in Testing Metrics, several management principles come to mind to move through the tests in a quick and efficient fashion.

The managment principles that have assisted me are:

• Focus on completing the maximum number of test cases between builds

At the outset of any test program, the complete set of test cases must be run and no bugs exist.  At the point that bugs are issued and fixed, the questions become when to rebuild the test environment and the extent of the regression tests.  My preference is to execute as many test cases as possible before rebuilding the environment and ideally execute the entire set until all test cases are either passed or blocked by a bug.  What the PM achieves is to identify the maximum number of issues and identify the most stable and least stable components of the solution.  By maintaining a substantial queue of issues, the development community can maintain a steady effort without any wait time as this community will typically be the driver for schedule.

In most test programs, this will take several days and assuming the development community begins bug fixing immediately a set of bugs will be ready for a build.  At the point where all test cases are either passed or blocked, the test program is ready for another build thus freeing up additional test cases for execution.  This principle makes a steady work queue for both the testing and development communities and affords the highest likelihood of the quickest progress towards completing the test phase.

• Focus on the completion of tests and not fixing bugs.

Once periodic builds commence, the test program is in an iterative cycle where all the tests are run and the developers work to unblock the maximum number of test cases with each build.  The testers are faced with a question of whether to regression test bugs or focus on the newly unblocked test cases.

With each build, I focus on the new unblocked test cases to get through all test case steps in the quickest possible time.  What this achieves is to uncover as many bugs as possible and ideally getting to the full set of bugs as early as possible.  Once the complete set of bugs is known, the PM has additional techniques to determine the ultimate questions of:  Am I on schedule? and When will I be done?

Regression testing of the bugs will need to occur but is a secondary concern early in the test program and can occur after all test cases have run through all their steps.  If the testers focus on bug fixing and not test case completion, it drives the test program into a state of constant regression testing where builds are requested frequently, the progress through the test cases both slows and becomes unpredictable, and the early test steps are executed many more times than needed as they are executed with each build.  All of these ideas result in a slower and less predictable test program.

• Define bug severity or priority levels and stick to them

One of the cornerstones of any test program is the categorization of identified bugs and the communication to the development community as to how big a deal each bug is.  Bugs are either categorized using a severity measure which speaks to the impact on the requirements associated with each bug or priority which speaks to the order in which bugs should be fixed.  At the end of the day, I beleive either is a viable manner to identify the urgency of the fixes but the main theme is to publish the definitions well before the outset of a test program and stick with those defintions.

The tendency of many organizations is to define the levels but get more "flexible" with the definitions in the heat of the testing program.  Individuals have their own points of focus and can stray from the definitions as their preferred functionality identifies issues.  Maintaining the definitions enables the development community to focus on the issues that have the greatest impact on the overall requirements and as stated above enables the test program to move through all the test cases in the quickest fashion.

• Structure the tests around business functions

The structure of the test cases plays a big role in the reporting of progress through the test program.  When speaking to the project sponsors, the conversations are more fruitful when speaking in the terms of the business functions.  If the test program centers on taking sales orders, then a structure around web orders, store orders, returns, exchanges, etc makes sense to the sponsors vice a structure around a set of interfaces or technical solution components.  By structuring in this fashion, the PM can report for instance that web orders are working except a detailed scenario such as multi-line orders when a single line has more than one delivery date.

This structure also typically dovetails nicely with functional individuals within the organizations and enables the PM to focus developers with an individual tester to more quickly understand the issues and confirm fixes.

• Manage using the metrics

The metrics defined in my previous post provide the PM a framework for priortization.  When executing the maximum number of tests, the PM quickly finds a situation where all the issues cannot be fixed in each build and typically more of each severity/priority level that can be fixed.  The question becomes where to begin.  The bugs blocking test cases metric provides a quantitative measure of which bugs block the greatest number of test cases.  By working issues first that are blocking the maximum number of test cases, the PM can unblock the maximum number with each bug fix and keep moving through the test cases in the quickest possible fashion.

In this model, there may be instances where a priority level 2 bug is fixed before a priority level 1.  This may seem counter productive but is in sync with the notion of moving through the test cases in the quickest possible fashion and identifying all the bugs as quickly as possible.

By using these principles to define priorities during any testing phase, movement through the testing phase has become quicker, easier, and more predictable.  All of these elements are important to the PM as validating proper operation of the solution in a predictable and quantitative fashion is a key element late in the project and a cornerstone to get a decision to move to production.

Role of Use Cases on a Project

With the advent of more recent development and implementation models, the concept of use cases and user stories has arisen.  For the purposes of this post, we will use these terms synonymously.  Historically, the Information Technology (IT) community used requirements to document "What the system needed to do."  These functions were documented and cataloged then eventually tested at the end of the project.  Over time, we discovered that requirements were a difficult to define as end users typically do not think in terms of software functions but more in terms of what business functions they perform.  From this realization came the concept of user stories and use cases.  The concept has caught on making user stories a very common artifact on today's projects.  The questions then becomes how do we are Project Managers use user stories and are there benefits beyond what requirements afforded us?

Development or Implementation Requirements

The definition and control of scope continues as a focus area for PMs throughout the project lifecycle.  The early definition and continual control of user stories as the basis of development and implementation are central.  The creation of a backlog of user stories and the inclusion of software bugs in the backlog creates a different dynamic for PMs using a sprint based methodology but is similar enough to a requirements based model that organizations can make easy transitions.  Quality control mechanisms such as traceability matrices can continue to have a place in projects with user stories.

Basis for Test Planning

Where user stories begin to have additional value over requirements is in test planning.  In a typical requirements based approach, a team may have created testing scenarios from the requirements.  In many cases, this is an additional organizing step performed by the test team to organize the requirements into identifiable business functions to enhance and put context around test results reporting.  With user stories, this structure already exists natively.  Since the user community defined the business processes this organization naturally falls out and in terminology easily identified and understood. 

The inherent advantage both in test planning and reporting is this business context.  When a base user function like assign a lead in a sales operations scenario or put away an item in a warehousing scenario are defined, the users have a strong understanding of both what the function is and how is applies to their business and day to day operations.  When planning the tests, defining the individual test cases becomes easier and the business community can participate more fully and with more confidence.  The same applies in the reporting scenario where understanding which business functions are working becomes a critical input into the decisions to release new functionality to production.

Basis for Training Material Development

Working with the same theme, training material is best organized around basis business functions.  It can start with some high level like distribution, general ledger close, or even a little deeper like sales lead sourcing.  With the business context of user stories, this same organization is usable as an outline for end user or even administrator training.  Better leverage is gained when the UAT testers are trained using the training material before user acceptance testing (UAT) so the training material can be flushed out before the larger set is trained just before going live.  The organization of the training material and test cases are in sync giving the UAT testers an advanced step before training in addition to having attended the draft end user or administrator training class.

Basis for End User Guides

End user guides or quick reference guides are the final beneficiary of user stories.  These guides need to identify the key business functions and define the key strokes needed to accomplish.  The user stories identify the major and detailed business functions and the test cases define the key strokes leaving only reducing those two artifacts down to the material desired in a user or quick reference guide.

Given the user context in which user stories afford us, these stories have benefits the project manager can leverage during all phases of the project.  Whereas requirements require organizing and reorientation to an end user context that requires additional effort, the user stories already have this context so reduce administrative work.  It appears to me user stories have all the benefits of requirements but also enable efficiencies not attainable otherwise so PMs should consider user stories as their basis for scope definition and control whether the project is custom development, a commercial software implementation or a combination.

Have multiple releases?: A different approach to maintaining the branches.

Have you been in a operational environment where multiple development streams each with unique functionality are in development but the release to production dates are not in a definitive sequence.  Or perhaps, an incremental release development model where the development is broken into blocks that are larger than any sprint/iteration.  These are examples of the scenarios whereby the question of when to merge the development in the configuration management tool branch becomes an acute question.  The timing of the merge to the trunk places the operations or project manager in a position whereby the production environment cannot be updated until the new functionality is also deployed.  This becomes an increasingly acute problem when the number of changes in production is higher, of high priority, or the size of the new functional release increases as the regression test cycle is necessarily longer.  So how do we solve this problem?

Traditionally, we have merged the branches to the trunk and built our pre production testing environment from the trunk which creates the circumstances noted above.  I suggest reversing that model.

As changes are made in the trunk (production) merge those changes out to the branch(s).  This creates in the branch(s) a code baseline that is identical to the future production environment and relieves the need to know definitively the order in which the branch(s) will be deployed to production.  It also creates in the branch a code baseline that is identical to the future production state but is in the branch and not the trunk so the capability to update production with either high priority fixes or any other branch remains.  In this instance, the issues associated with merging of the new functionality to the production baseline are flushed out in the branch by building the pre-production testing environments from the branch and testing. 

Once the functionality is flushed out in the branch and this branch is determined to be the next going to production, then the branch is merged to the trunk, regression testing and release to production occur.  By testing out in the branch, the regression testing from the trunk (after the merge) is greatly reduced thus minimising the risk of production maintenance during the regression testing window.  The positive side effect is the decision on which new functional release will go to production is delayed until after the functional testing in the branch and easing any decisions in this area.

The downside to this approach is more maintenance of code out in the branches.  The decision becomes a trade-off between the risks associated with longer regression test cycles and the inability to maintain production during this cycle with the additional effort associated with maintaining the branches.  In those circumstances where there are a lot of high priority updates in production and/or the new releases are large, numerous, or have long regression test cycles, I suggest considering this alternative approach.

Relationships between use cases and requirements

In some organizations, the terms use case and requirements are used interchangeably.  In some respects this is an understandable point as both are ways to define the basic functions that software must accommodate.  They are both typically defined early in the project, are used as a basis for estimating the effort involved in a project, are grouped so developers or implementation consultants have some structure during the development or implementation phase of the project, and they are also used as the basis for testing planning and completion.  In many ways, they are similar.  In terms of usefulness for test case and training material development, they are very different.

At the outset of any project, we project managers look to gain insight into what the solution must do.  In many cases, we ask our business users what they would like the new software solution to do for them and we call this list of items, requirements.  This approach is well seasoned and has manifested itself in techniques like Joint Application Design (JAD) sessions where groups of business get together and discuss their needs of the new solution.  Project team members furiously take notes as the items are identified.  Even though this is a well seasoned approach for developing requirements, it is difficult question for business users to answer because they do not think of their business in this manner.

In my experience, business people are well versed in how the business is conducted, what they do on a day to day basis and what information and in what format they need this information to be effective in their varied positions within any company.  This business context is a key differentiator between use cases and requirements.

When developing use cases, the project manager asks the business team what they do on a day to day basis, what they do on rare occasions, and finally what they do on a very intermittent basis.  These questions have the context of the business activities they perform.  In order to be complete, project managers can start with more senior executives or even the project sponsors to get an idea of the breadth of individuals from whom to solicit their day to day activities.  These are the use cases.  A similar interviewing process can occur with individuals but with much smaller groups so each group can focus on their area of responsibility.  The project team can collect the use cases from each team and consolidate them.

By collecting from each group, a natural organization occurs since each group is typically a functional group enabling the project manager to begin to organize the use cases and therefore the development or implementation activities into a structure well known to the people who will both be served by the new solution but who also are likely to determine when the solution is ready to deploy.

Besides functional business people, a prudent project manager will also interview technical staff who will both administer the solution, network, interfaces, and batch processing.  These technical use cases are equally as important as the functional ones.

When defining requirements, an interview is typically the well traveled path to definition.  Many services companies have defined templates or questionnaires to assist with getting a complete set of requirements.  All of these approaches prove difficult as each software solution is unique to the customer.  We typically find our requirements sets of incomplete and this realization does not occur until deep into the testing cycles or even as late as the first few days after going live.  As consultants, we are very conscientious about doing a good job but we are behind the eight ball.

A natural mechanism for developing use cases is to start with business process flows.  Business process re-engineering (BPR) was very popular in the late 90's as companies began to look critically at their internal operation.  On many occasions as part of an Enterprise Resource Planning (ERP) implementation, my project teams would start with the business processes, take the opportunity to identify areas of streamlining since the process data was consolidated and became easier to identify these areas.  Although BPR is less popular today, the definition of business processes can be a good starting point for use case definition as the use cases are each unique path through the business processes.

In previous posts, I have mentioned the advantages of process flow diagrams as they identify new paths that are not always apparent and provide a mechanism for being more systematic in the approach to define use cases.  My best projects have started with using process flows as the starting point for use case definition and then leveraging these use cases further down the line when writing test cases and even training material

Once the project manager has a good set of use cases including the happy and non happy paths through the business processes, these same paths become the basis for the test cases.  The difference with test cases is a test case should address each inflection point within the continuum of data that could pass through the use case.  For example, we may look at closing the AP books for the month.  The use cases will be the same but the test cases include open AP across the boundary as well as open and closed within the month.  This different data condition may lead to different handling within the solution.  Another example could be a calculation algorithm whereby negative numbers are not permitted such as in a financial example or were a zero is handled differently. All could be the same use case but different test cases so the unique data conditions are exercised.

Finally, the use cases can be the basis for the training material.  Given the use cases define all the business activities needed for a good solution, the end users will need to understand the keystrokes to perform each of these business activities within the solution.  At a high level, the trainers can take a representative set of test cases including the organization defined during the collection process and build training material off this set.  The unique activities are defined within a structure that makes sense to the business and is broken down into functional sets that align to the set of users to be trained.  What is remaining for the trainers is to capture screen shots where appropriate, define exercises and pre-stage data into a training environment if so inclined.

Although requirements and use cases are perceived by many organizations to be very similar or synonymous terms, the manner and context in which they are derived as well as the additional value of use cases in the testing and training realms define some separation between the two.

Test Phase Management - Simple metrics for effective management.

As we all execute testing programs whether it is a integration, system or user acceptance testing phase, the two questions that consistently arise are:

• Is the testing phase on schedule?
• When are we going to complete the testing phase?

For many people, the questions are answered with statments such as:

• The test cases will be executed by the end of the week
• If the developers can solve all the bugs by a certain date, I will complete executing the tests by a certain date

In a nutshell, the responses tend to be relative to some other event while the people asking the question are looking for an absolute answer.  How do we quantify the response in terms we can defend?  I use a set of two metrics and trends over time for each that address the fundamental questions.  These metrics are test case pass percentage (testing progress) and trend over time and open bug count and trend over time.  These two metrics provide a quantitative measure and a backup measure so the project manager has two data points that point to the same timeframe.  These metrics assume test phase exit criteria of all test cases pass and all severity bugs are closed which I find typical.

Let's start with the Test Progress Metric which documents the percentage of all test cases defined and passed for the testing phase.  The numerator is the number of test cases passed and the denominator is the total number of test cases in the test plan.  Over time, the number passed is recorded on a daily basis and percentage plotted over time.  This trend line is then projected ahead to 100% for the completion date of the test phase.  The trend of test case passing follows a linear pattern for most of the curve so any linear projection method affords a good view of the completion date.

The main advantage to this metric is it combines all the major elements that impact test phase completion but are either unknown or not easily measurable.  These include:

• At any point in the testing phase, the number of additional bugs that will be found.
• The rate at which developers will close bugs
• The rate at which testers can regression test repaired bugs

I have seen many organizations attempt to measure these parameters or define quantatively these unknowns but with little success.  It is more cost effective to use the defined metrics which naturally address all these questions.

Stability in the total number of test cases is critical here as instability in the denominator creates an unstable trend line so any risks associated with test plan instabiity need to be mitigated up front.  If the instability occurs, the best approach is to recalculate the daily completion percentages, replot the curve, and the projections.

The second metric is open bugs and the trend line.  Similar to the testing progress metric, the number of open bugs is recorded on a daily basis and plotted over time.  Unlke the testing progress which uses a percentage, the open bugs is an absolute number.  The severity level of the bugs needing closure for the test phase to end should be the set used in the plot.  In a typical project, this is the severity level 1 and 2 bugs but it could differ depending on the individual organization and the bug management process established.

The open bug curve will follow a hypergeometric distribution which has a big hump early and a long linear tail as it approaches zero.  This linear section is where the PM can also use any linear projection method to determine when the bug count will go to zero.  Given the upward portion of the curve early on, this curve is only helpful after 60%-70% of test cases are pass.  It will typically fall very close to the testing progress curve and provide a backup to the date projection derived from testing progress.

I have found these metrics to be very helpful in answering the questions from sponsors and are easy to defend.  If anyone would like to see examples of these trends, feel free to contact me either on the blog or twitter @mdgibson.  In a future post, I will walk through some test phase management techniques and how they compliment the testing metrics.

No Test Team: Ideas for System Testing

I recently started a new project in Canada which is an enterprise implementation of a commercial order capture and order management product.  My initial discussions with the PMO went well until we began discussions around the testing strategy where I typically discuss the different testing phases, the number and availability of testing environment, etc.  When we came to the composition and contact information for the system test team, the customer's face went blank...

I discovered this customer did not have a test team only a business analyst.  Needless to say, I was a little taken a back but went to work on how to test an enterprise level order capture and order management system without a formal test team. I also learned there would be no formal test plans either but there would be a user acceptance test used by the business to make the production deployment decision.

My first thought was to hire a testing firm to come in and execute a traditional testing phase, but I do not have the budget for it so needed to be more creative.  

As I am currently in the requirements definition phase, I plan to use this time with the technical and business community to document not only the requirements but also the use cases in a fashion typical of the Rational Unified Process (RUP).  Not only will this enable me to confirm that I have requirements in all areas of the business, but will also provide me an informal set of test scenarios.  My technical team will use these scenarios to guide the integration testing and perform an integration test of greater than typical depth as it will include many fringe cases identified early and typically left to a more formal test phase.  Using the development team to execute on all the test cases also will flush out aspects of the system configuration that may have been overlooked.

After meeting with the business analyst, I have good confidence that she is well versed in all aspects of the business.  The issue becomes finding time for a single person to effectively test a system of this size. What I came up with is three fold strategy 1) apply the agile development approach to a commercial software implementation and provide incremental releases of functionality at the soonest possible time and build the system over time, 2) enhance unit and integration testing to include all the use cases collected during the current requirements phase, 3) engage my off shore performance testing group.

As is customary with agile custom development, the superset of functionality is divided into small testable releases.  These releases build on each other and the team get into a rhythm whereby a group is designing, a second developing the next release and a third testing what has been delivered.  The entire solution is build one releases at a time with the development and testing time periods overlapping.  The net result is a longer test cycle. The testable release concept is also applicable to commercial software as subsets of functionality are configured vice coded.

I typically start with the initial activities in the business process under development.  In this case, the authentication of users with Active Directory and all fringe cases such as invalid password, password resets, and password changes.  As this is an order capture system, the next step will be capturing orders but not sending them to the order management system, no accounting entries, or inventory transactions. This additional functionality is added in subsequent releases in the order of the business process.  The final steps in the business process and therefore releases are reverse logistics.

This approach will extend the test cycle from weeks to months and allow the BA the time to focus on smaller sets of functionality as they roll out.  The additional time comes from overlapping the development and testing cycles.  A formal system test phase can also be added where the group does exist.  The performance testing group is off shore so I can get them cheaply to cover the potential issues around performance and effectively isolate the sub systems where the performance is below expectations.
Given the integration testing is enhanced and included the end to end scenarios, I will engage the performance testing team in an incremental fashion as well thus providing the technical team a longer lead time to address any identified performance issues.

The risks in this approach are:
    1) The interfaces which the customer is building may not be available in a time frame to support the extended testing cycle
    2) The fringe test cases not identified during requirements definition may be missed until the UAT
    3) The stabilization period after the production date may be extensive and have a greater than tolerable error issue rate given the less than ideal testing
    4) I am relying on the testing expertise of a single individual and my development team where neither group are testers by profession

Below is what I plan to execute and will write a follow up post in the summer of 2010 on how it went.