So, how long do you need for testing?
How many times have you answered this question?
We are all aware of the Pareto rule, 80% of the outcome is dictated by 20% of the effort! or as Benjamin Franklin would say “If you fail to plan, you are planning to fail”
Let me share some of the wisdom that I have applied to creating test estimates.
In creating the testing estimates for software development projects there are many approaches for small and large projects. Top down, bottom up, expert knowledge. Below is an example of how I approach creating test estimates that is as accurate as possible.
One of the first things I, as a test manager want to understand is how long will it take us to create and execute the test cases for a new feature or some other change to an application that we are responsible for testing. This activity will also help inform test planning. This isn’t an ‘agile’ approach, and has more in common with waterfall, but it’s one that works and is usually accurate.
For new projects with no previous history, estimation will require a thorough analysis of each new component being built, the business processes, integration points, APIs built or used, UI design and other supporting documentation to build a picture of the overall project and the phases that it will be built and delivered to test.
Very often this info isn’t available when you need it, so assumptions are used and made clear to the project team at the outset.
For the example described below we’ll assume that the change is to an existing feature in an already live app.
WHAT?
First step is understanding what it is that needs to be tested.
This is the job of the QA team to investigate and find out these impacts, some of it may be understood based on experience the team has of delivering similar projects. Known as ‘expert knowledge’ or will have to be gathered from others who have that expert knowledge (BA’s, Dev, Marketing etc) and then incorporated into the test case design phase.
Either way, this is the most important part of the testing phases.
Analysis of each change and the impact those changes have on areas of the application that may not be explicitly mentioned in the requirements.
Next Step, Identify the number of high-level test cases and scenarios.
What new tests do we need to execute in order to ensure that we have coverage of all the new changes and requirements?
Once those are defined, look at the regression tests.
What existing functionality could potentially be broken when this new change is implemented?
When you have this number of test cases, you need prioritize them based on complexity.
High, Medium and Low.
In this case ‘complexity’ is a measure of how long it takes to prepare and execute a test case.
For Example;
High Test Complexity:
- 1 Hour Test Data
- 2 Hour test preparation.
- 1 Hour Test Execution.
Medium Test Complexity:
- .5 Hour Test Data
- 1 Hours test preparation.
- 5 Hour Test Execution.
At this point, you can sum the number of tests and calculate time for test data, test prep and test execution and convert into hours or days effort.
Contingency:
Nothing ever goes exactly to plan so we need to include that in the estimates.
This is the extra time we build into the task so we can absorb any unforeseen events and delays.
I usually add 20% to the overall test case prep and execution time. You can refine this approach as you become more familiar with your app and the level of quality delivered to testing.
Failure Rate/ Defect Retest:
Again, this is a form of contingency.
Tests fail, and when they do, they need to be retested (including regression tests) after the cause of the failure is resolved.
Adding 20% of the Test Execution total time as Failure Rate/ Defect Retest time.
Now you know, how much time is required to create the tests, set up the test data and run the tests.
HOW?
Next step is, how will we test it? This is usually handled in the test plan, but the estimate needs to include any time related to setting up test environments and managing and other non-test activities that are required.
Testing is typically carried out in a QA environment that may be similar but different to the live environment. For example. The QA environment may not be connected to all associated systems that the live environment interacts with. Financials, Data Warehouse etc. This needs to be understood and planned for.
The QA environment usually will not have the same level or type of data that is required to execute a test.
Creating test data can be time consuming and needs to be planned for, it also may need to be recreated in order to repeat a test. For example, Running a batch process on a month’s data. Most likely that month of data will now be stamped with a processing date and can’t be reused.
How do we handle that? Roll back? Use a new Months data? That all needs to be discussed and planned for upfront.
But it all takes time and needs to be included in the testing estimate.
WHO?
Now we know how many hours/days effort is needed. We need to understand what resources we have to deliver the project.
People are the main resource for any project whether its manual testing, creating automation scripts or performance tests, so we need to find out how many people we need to deliver it all, or most likely how much we can do with the resources we have.
I usually estimate a 6-hour day per person for purely testing activities, this excludes meetings, emails and conversation with colleagues all of which are valid and need to be considered.
At the end of this process, we have estimated what is in scope for testing, how long it will take to create and run the tests. How much time will be required for defect management and retest and how long it will take to complete a full cycle of testing with the resources we have.
This information needs to be fed to the project stakeholders so they can include the estimate for testing in the project life cycle.
Unrealistic delivery schedules, poor planning and poor reporting are key contributors to software project failures. Planning a test estimate is key to communicating the requirements needed to mitigate risks for the business. Monitoring a plan is impossible if there is no plan in place.
I hope you found this article of value, please like and share. I’d like to hear what approach delivers the best results for you. If you would like to know more about our services, check us out at www.celticqa.com
