Regression Testing, Automated vs Manual

 

There are many misconceptions surrounding the interactions between the development, manual testing and automated testing groups including where automation resources fit into the development cycle and the timing surrounding completion of the deliverables they provide. Frequently the automation team is put into the same slots and their deliverables are expected at the same milestone or cycle checkpoints as the manual testing team. Applying the lens used to view manual testing to the automation team is inappropriate and it creates expectations that are not maintainable in the long term by an automation team with a reasonable or cost-effective headcount. Here we will detail the optimum workflow and expectations that process/product owners and project managers should have when utilizing automated testing as outlined in this strategy document.

A User Story’s Journey

Regardless of development methodology or project management approach being utilized the vehicle used to convey business requirements to development teams is a user story. A given user story goes through various states and checks on its journey from being defined to passing QA and becoming production ready. This journeys length, and the associated effort to ensure quality can vary dramatically between workflows that exclude and include automated regression testing.

To focus on this variance in effort let’s quantify it in resource hours and then attach a dollar amount to those hours.

Consider a user story to add additional useful information that is accessed via a profile page once a given user logs into a web application. That story might look something like the one outlined below.

User Story 5482 – User Profile Page – Display Address Information

Description – As a user, I would view the address associated with my account from my profile page.

Note: This story is only for the displaying of address information. The ability to modify address information is outlined in User Story 5483.

Requirements

    1. Address information should be displayed in the following fields
      1. Address Line 1 (Line 2 as needed to conform to USPS line length specifications)
      2. Unit/Apartment/Suite number
      3. State
      4. City
      5. Zip code
    1. All display elements should be styled using the same standards as the existing elements on the profile page.

Acceptance Criteria

    1. Address information should be retrieved from the Address table according to the following
      1. Address Line 1 -> Address.LineOne
      2. Address Line 2 -> Address.LineTwo
      3. Unit/Apartment/Suite number -> Address.UnitAptSuite
      4. State -> Address.State
      5. City -> Address.City
      6. Zip code -> Address.ZipCode

After the planning meeting, this story has the following tasks created and associated hour estimates.

Task 1 – Development – Update user profile view to render address information, 2 hours

Task 2 – Development – Update user profile view model to contain address information, 1 hour

Task 3 – Development – Update controller to populate address information, 2 hours

Task 4 – Test – Create/Update user profile page test case, 2 hours

Task 5 – Test – Manual test execution, 1 hour

Task 6 – Test – Execute user profile page regression, 4 hours

Next, let’s focus on the time to complete looks like in workflows with and without automated regression suites.

Without Automated Regression

In a purely manual testing approach, the effort in hours for story 5482 can be calculated with the following formula.

DevTime + QATime = TimeToComplete

OR

(2+1+2) + (2+1+4) = 12 total hours to complete the story

The assumption in this calculation is that all tests pass on the first attempt and there is only a single build to perform regression testing on prior to release. This means that the story follows the green path in Fig. 1 and answers yes to both questions about tests passing.

 

So how do failures impact the equation when tests do no pass?

First, consider a failure in tests that are not part of the regression suite. This occurs when the ‘No’ path is taken from the first decision block in the chain and triggers the following actions.

  1. The story is returned to the responsible developer as failed with detailed steps on how to reproduce the failure
  2. The developer determines the root cause of the failure and implements the fix (Addition of time to complete the story, let’s say two hours) and delivers the remediated code for a second round of testing (Adding an additional hour for manual test execution).

The formula now looks like this:

(DevTime + QATime) + (DevRemediationTime + QARetestTime) = TimeToComplete

OR

(2+1+2 + 2+1+4) + (2 + 1) = 15 total hours to complete the story

The discovery of this issue has added three additional hours (two from the developer and one from QA) to our completion timeline. Depending on the allocation of the two resources directly involved this could impact the ability of the team to deliver on other items committed to. The above formulas do not take into account any meetings or time needed from other resources on the business or perhaps operations teams needed to troubleshoot the issue or clarify requirements.

This unknown or hard to define quantity can be designated as ‘X’ or OtherTimeNeeded, changing the formula to:

(DevTime + QATime) + (DevRemediationTime + QARetestTime) + OtherTimeNeeded = TimeToComplete

OR

(2+1+2 + 2+1+4) + (2 + 1) + X = (15 + X) total hours to complete the story

In the above scenario we see that the discovery of one issue, assumed to be relatively simple, added three quantifiable hours to our time to complete and the factor X representing the unquantifiable hours. Additionally, the issue was found while executing non-regression testing.

Now consider a scenario where new feature passed testing and an issue was found during regression. How would time to complete be impacted?

Let’s start with our original formula:

DevTime + QATime = TimeToComplete

In this scenario our failure occurs in the final decision block in Fig.1, meaning the time investment in this story incorporates all the hours estimated and looks like:

(2+1+2) + (2+1+4) = 12 total hours to complete the story

Since the failed test is part of the regression suite any changes made that were incorporated into the build being tested could be the root cause. This adds a level of complexity that would require an additional abstract variable be added to our formula, let’s call that RootCauseAnalysis. In addition to root cause analysis, this also triggers the below actions.

  1. Developer remediation of the issue (For the sake of this exercise we will assume it was not acceptable to allow this defect into production)
  2. Additional manual test execution for the story that introduced the defect to ensure function matches design after the fix
  3. Additional manual execution of the regression suite to confirm the fix corrected the defect and that no other issues were introduced

The three actions above, as well as root cause analysis, will need to be added to our base formula.

We started with: DevTime + QATime = TimeToComplete

Let’s add our new variables and put in some hours for our concrete example:

(DevTime + QATime) + (RootCauseAnalysis + DevRemediationTime + QARetestTime) + AdditionalManualRegression = TimeToComplete

OR

(5+7) + (3+3+1) + 4 = 23 total hours to complete

As you can see in the example above finding defects in regression can substantially impact time to complete in environments that rely on purely manual testing, specifically manual regression. How can we mitigate this impact and risk? The answer is to convert our manual regression into an automated regression.

With Automated Regression

Let’s take another look at our formula for story 5482 in a purely manual testing approach. Specifically, the test task ‘Task 6 – Test – Execute user profile page regression, 4 hours’

DevTime + QATime = TimeToComplete

OR

(2+1+2) + (2+1+4) = 12 total hours to complete the story

 

Examine Fig. 2.

Note that the green, or manual testing, the path stops after the execution of tests that target functionality being developed in the current sprint and the regression task is now automated. Another leg has also been added to show automated regression is also being executed as part of the CI/CD pipelines. How does this impact our formula used to calculate time to complete? Since the regression tasks no longer require manual effort we can remove it from our formula.

The abstract does not reflect this change, the hourly breakdown does.

DevTime + QATime = TimeToComplete

OR

(2+1+2) + (2+1) = 8 total hours to complete the story

Off the top, the introduction of automated regression has reduced the time to complete story 5482 by 25%. This is not a complete view of the benefits of automated regression though. In addition to now having four resource hours freed up we have also executed regression twice, once during the build and once at the end of the cycle. Essentially we have added two rounds (8 hours based on the task in our story) of regression in our current sprint at no cost to capacity.

Let’s consider the scenario discussed earlier where a failure occurs in tests that are not part of the regression suite. This occurs when the ‘No’ path is taken from the first decision block in the chain and triggers the following actions.

 

  1. The story is returned to the responsible developer as failed with detailed steps on how to reproduce the failure
  2. The developer determines the root cause of the failure and implements the fix (Addition of time to complete the story, let’s say two hours) and delivers the remediated code for a second round of testing (Adding an additional hour for manual test execution).

The formula , in a manual environment, looks like this:

(DevTime + QATime) + (DevRemediationTime + QARetestTime) = TimeToComplete

OR

(2+1+2 + 2+1+4) + (2 + 1) = 15 total hours to complete the story

The formula, in an automated environment, looks like this:

(DevTime + QATime) + (DevRemediationTime + QARetestTime) = TimeToComplete

OR

(2+1+2 + 2+1) + (2 + 1) = 11 total hours to complete the story

The difference between the two formulas is that regression no longer adds time to the QATime variable. This shows that our net gain is still +4 resource hours. Consider the flow outlined in Fig. 2 that this issue took us down. During the process there were at least two builds, meaning our regression was executed twice by the pipeline, and once to validate our cycle development prior to release. This means the application was regressed three times and instead of adding 12 resource hours in effort, we got them for free.

 

Finally, consider the above scenario where a defect is found during regression.

The formula, in a manual environment, looks like this:

(DevTime + QATime) + (DevRemediationTime + QARetestTime) = TimeToComplete

OR

(2+1+2 + 2+1+4) + (2 + 1) = 15 total hours to complete the story

The formula, in an automated environment, looks like this:

(DevTime + QATime) + (DevRemediationTime + QARetestTime) = TimeToComplete

OR

(2+1+2 + 2+1) + (2 + 1) = 11 total hours to complete the story

Triggering the below actions.

  1. Developer root cause analysis
  2. Developer remediation of the issue
  3. Additional manual test execution for the story that introduced the defect to ensure function matches design after the fix
  4. Additional manual execution of the regression suite to confirm the fix corrected the defect and that no other issues were introduced

The actions above are then added to the formula.

We started with: DevTime + QATime = TimeToComplete

The formula , in a manual environment, looks like this:

(DevTime + QATime) + (RootCauseAnalysis + DevRemediationTime + QARetestTime) + AdditionalManualRegression = TimeToComplete

OR

(5+7) + (3+3+1) + 4 = 23 total hours to complete

 

The formula, in an automated environment, looks like this:

(DevTime) + (RootCauseAnalysis + DevRemediationTime )+ QATime = TimeToComplete

OR

5 + (3+3)+1 = 12 total hours to complete

So, what happened to our formula? Aren’t there a bunch of variables missing? Yes, there are. In the purely manual environment, we ran regression at the end of our cycle, meaning all the resource hours for our story have been burned down when we discover the defect. This means that all the effort related to correcting the defect are added to the total because there is no task overlap. In a workflow incorporating automated regression, the test run that discovers the defect happens immediately after the developer completes their tasks, much earlier in the development cycle. This means QATime is not expended so there is no need for the variable QARetestTime and QA time should be placed after the sum of root cause analysis and DevRemediationTime. Finally, our regression test is automated so there is no need for the AdditionalManualRegression variable. Using our example hours this works out to a net gain of 11 hours of resource capacity now available for additional tasks. Following this example our regression tests have been executed three times, the initial run that found the defect, the run to build the remediation and the pre-deployment run, regular priced at 12 hours and free for us because we utilize automated regression.

If you are thinking of implementing testing automation, Intersys have diverse solutions and experts to help you to go through the strategy, plan and successful implementation of the automation components and development, contact us!

Share this:

Leave a Reply

Your email address will not be published. Required fields are marked *