A Simple Model for Forecasting System Test By John C. Goodpasture, PMP Articles / Newsletter Article Date: Sep 06, 2006 - 10:01 AM Everyone who has ever tested anything knows that it does not always work the first time it is tried. If it always worked, system test would be a simple procedure at the end of the deliverable’s development life cycle. But it’s almost never that simple, and the six sigma protocol tells us that to really reduce errors, many test and measurement repetitions are needed to ascertain root cause and correct error sources. But for most practical projects, there are only a handful of opportunities for test repetitions and measurement. Here’s a simple model that can be applied to many test situations that provides a practical, almost back-of-the-envelope, simplicity that can be employed by project managers and test deliverable managers in a variety of different project situations. After reading this article, you will know that a 40-40-20-attempt model means the test effort must be multiplied by a factor of 1.8 for planning purposes, and that test setup time must be multiplied by a factor of 3. Test Conditions We begin by thinking about test conditions. Test conditions are either functional or technical. They can be expressed as capabilities or capacities, and/or they can either be a state, or a change in state, or a functional stimulus or response. Test conditions can be tested individually or collectively. Test conditions are applied to, or exercised by, one or more scripts. A script is a step-wise procedure that is followed by a test individual or machine that proves that the deliverable can perform in the context of the test conditions. Each combination of a script and test condition, or set of test conditions, is called a scenario. Each time a scenario is run, it is called an instance of that scenario. Results attach to instances. Between instances, the test environment or system may have to be reset, rolled back, or re-initialized, and fixes may be applied to the deliverable between instances to correct problems. Instances may be grouped together into an attempt, and all the attempts constitute the test. Summarizing: A script is like a template or an outline or framework; test conditions attached to the script fill it out and provide a scenario. To test the deliverable, we run the scenario. Each running of the scenario is an instance. A group of instances run during one setup of the test environment make up the attempt. When all attempts are completed, then we have executed the test. So now what about the model? What we are going for is a simple and practical model for forecasting the effort and duration of the attempt and the test. The first idea is that an instance passes or fails when the scenario is run. The total of the passes and failures is the scope of the attempt. In this model, we do not allow partial credit for some conditions being satisfied and others not. An instance passes entirely, or it fails. Script Complexity The second idea is about the complexity of the script and the dependencies between test conditions that influence the complexity of the scenario. Script complexity can be a matter of the total number of steps, each step providing an opportunity for failure, or the logic among steps, the logic itself being a source of failure. If a script has attached more than one test condition, then dependencies among conditions may be source of failure. Dependencies generally have the effect of stretching the script duration, thereby giving rise to the opportunity for failure for those conditions or scripts that are time-sensitive. We abstract all these failure opportunities with a single estimate of complexity. To keep it simple, complexity is characterized as high, medium, or low. The exact definitions are specific to the project, but many projects adopt a 4-2-1-multiplier sequence on effort, corresponding to high-medium-low. The test manager applies judgment and experience to assign a complexity to each scenario. Pass rate is the last concept in the model. Pass rate is simply the total passes divided by the total instances in the attempt. Pass rate corresponds inversely to complexity: common sense tells us that the more complex scenario is less likely to pass the first time. In our model, the pass rate is cumulative: the number of passing instances in the first attempt add to the passing instances the next attempt and so on until all scenarios are passed. An Example Let’s take a numerical example to illustrate the model. Suppose we have a scope of 10 medium complexity scenarios. If everything worked perfectly, we could run one instance of each scenario and expect: 10 instances, 10 passes, 0 failures. Just to be sure things are stable, we might run each scenario 10 times, hoping for a result of 100 instances, 100 passes, 0 failures. But from experience we expect that medium complexity scenarios will not all work perfectly. We estimate a pass rate by applying lessons-learned and other observations and measures from similar projects or industry metrics. For this example, we might estimate that a medium complexity scenario will correspond to a 40-40-20-attempt model. What does that mean? It means that we estimate that 40% of the scenarios will pass in the first try of instances, another 40% of the scenarios will pass on a second round of instances, and the remaining 20% of scenarios pass the third time: 40% + 40% + 20% = 100% of scenarios passed. Applying the model, we forecast the results as follows: on the first attempt we run 10 instances, one for each scenario and expect 4 instances to pass and 6 to fail. On the second attempt, we test the 6 failed scenarios with 6 instances. Our pass rate model is cumulative, so we forecast another 4 scenario passes corresponding to 40% of scenarios successfully passed on the second round, and 2 failures. On the third round, the 2 remaining scenarios pass with 1 instance each, and 0 fail. The total instances in the 40-40-20-pass model are: 10 + 6 + 2 = 18. At this point, all scenarios have passed. The test is complete. The test required three attempts, and the attempts collectively consisted of 18 instances. The 18 instances verified the test conditions on 10 scenarios. The 40-40-20-model has an effort multiplier effect of 1.8 times the effort and duration of one instance. Our complexity was estimated as medium. Recall that a medium complexity requires two units of effort, where one unit is the effort required to run a low complexity scenario and four units are required to run a high complexity scenario; 4-2-1, high-medium-low. We have not yet said what the value is of one unit of effort, but if it were 5 hours, then the medium scenario should take 10 hours. Our effort forecast for the test is then 180 hours corresponding to 18 instances of a medium complexity scenario. Of course, we can restructure the model to fit any repetitive test situation. Any three-attempt series means two reset or rollback efforts are needed, one between the first and second attempt and the other between the second and third attempt. A total setup effort is then the original setup plus the two setups between the attempts, for a total of three setup efforts. The 3x reset effort multiplier effect is independent of the pass rate. But if we change the pass rate, the multiplier effect changes for the effort that goes into the running of the scenarios. For example, a more pessimistic sequence of high complexity scenarios might be 10-30-60. In this case, the multiplier is 2.5. A low complexity sequence might be 60-20-10. The multiplier is 1.5. Summarizing: the attempt model is a shorthand means to quickly assess the effort and duration of a test. The number of attempts multiplies the effort to reset or rollback the test environment. The pass rate multiplies the execution effort according to complexity. Simple monikers like 40-40-20 mean multiple the execution effort by 1.8 and the setup effort by 3. About the Author John C. Goodpasture is Founder of Square Peg Consulting. He can be reached at www.sqpegconsulting.com. This article comes from allPM.com http://allpm.com/ The URL for this story is: http://allpm.com/modules.php?op=modload&name=News&file=article&sid=1601