Regression strategy in one go

“If we would have regression executed before hand we would have caught this defect…” — this conclusion you might hear very often during root cause analysis of the defects especially production escapes. And it’s fair statement. According to the Six Sigma the desired outcome of “zero defects” mindset would be “to produce long-term defect levels below 3.4 defects per million opportunities (DPMO). The 3.4 dpmo is based on a “shift” of ± 1.5 sigma”.

Concept “zero defects” exists but technically is almost never possible. In real project situations very often the concept evolves to “zero business harming defects” and it aims to eliminate blocker / critical defects and reduce costs associated with waste. That emphasizes the importance of getting things done right from the beginning.

And one of the approaches that QA engineers can use as a powerful tool in achieving high standard quality is consistency of Regression strategy execution. Here is one slide strategy that will keep it simple for you:

Regression strategy in one slide

First of all any Regression testing strategy should be justified Why we need to spend time and resources:

Why Regression is important

• to overall improve user experience — better quality better impression target enduser groups will have which increases number of returning customers;
• increases confidence in the delivered quality — over time new code implementation or existing code updates can lead to “code decay” where unrelated changes negatively impact the behavior of the system components. Regression testing helps maintain the integrity of the software and prevents its degradation;
• reduces cost of defect fixes — regression testing helps identify defects that may have been introduced due to code changes (e.g. new features, bug fixes etc.). By retesting the software, developers can catch and fix issues before they reach end-users which save the cost by avoiding business impact in production;
• decreases test cycle — as most common approach is to have regression suite to be automated and run over CI/CD. This reduce time and resources needed for the regression cycle and provide opportunity to get feedback on quality faster;
• decreases probability of missing the hidden requirements — regression testing may encounter discrepancies between the current software behavior with the expected behavior based on existing requirements. It can indicate that hidden requirements exist that were not addressed in the initial specifications. Or regression testing might reveal unexpected changes in the system behavior, which might again be indications of hidden requirements that were not explicitly specified but were inherent in the original system.

In order to achieve “Whys” here is key triggers When Regression execution should be considered :

When Regression needs to be executed

• new features or peace of functionally is added to the existing code;
• rework on configurations were done in the new build;
• defects fixes were patched to existing version of the app;
• existing features were reworked due to change of the requirements;
• system tuning occurred based on performance testing recommendation report to improve the metrics;
• test bed software versions were upgraded to newest (e.g. server software, desktop software, VM software etc.);
• hardware of the physical machines were upgraded.

Based on What is impacted the following techniques can be applied to select effective Regression suite:

Techniques that can be used to select What is Regression scope

• “Selective, priority based” regression testing fits well within Agile development practices, enabling faster and more frequent releases without compromising quality. By selecting and prioritizing test cases, the regression testing effort is optimized, making efficient use of testing resources, time, and effort. It’s important to note that selective, priority-based regression testing is not meant to replace comprehensive testing altogether. It complements other testing approaches and should be used in conjunction with other testing techniques
• Risk based regression testing is prioritizing high-risk areas ensures that critical defects are identified and addressed early in the development cycle. The first step is to identify potential risks associated with the changes being made to the software (e.g. changes to critical functionality, integrations, performance, security, and areas prone to defects). Assess the potential impact of each identified risk on the overall software application — consider factors like the likelihood of failure, the severity of consequences, and the number of users affected. Prioritize the test cases based on the identified risks. Test cases associated with high-risk areas are given higher priority, while those linked to lower-risk areas may be deprioritized or excluded from regression testing.
• Partial regression testing is a software testing approach that focuses on testing specific areas of a software application that are directly affected by recent code changes or updates. Instead of retesting the entire system, partial regression testing targets only the relevant modules or functionalities that have been modified, along with their immediate dependencies.
• Progressive regression testing involves gradually building and expanding the regression test suite as the software evolves. It is a continuous and incremental process that aims to validate both new features and existing functionalities with each development iteration.
• Corrective regression testing is conducted to validate and verify that changes around defects have been successfully fixed. It’s important to note that corrective regression testing should not replace regular regression testing. While corrective regression testing focuses on verifying the fixes for specific issues, regular regression testing ensures that all functionalities of the software continue to work as intended with each update or change.
• Complete regression testing, also known as full regression testing or exhaustive regression testing, is a software testing approach in which the entire software application is retested from scratch after each change or update. It involves executing all existing test cases, including those related to new features, existing functionalities, and any interdependent components. Despite its advantages, complete regression testing can be time-consuming and resource-intensive, especially for large and complex software applications.
• Retest All regression testing approach is similar to complete regression testing however, the primary focus in retest all regression is on retesting the specific areas that were affected by the recent changes rather than retesting the entire application.

Effective Regression testing strategy is very much rely on How you will approach it. So here is key steps to make your strategy neat.

Step 1: Analyze impact of changes in business rules, architecture solution, code , environment, test scope etc. Anything that becomes different from the previous iteration needs to be accessed.

Step 2: Plan how and when you will perform cycle of regression testing.

• Ideally regression suite should be automated and triggered automatically with every new build to upper environment (for example run nightly on schedule or run on demand via CI/CD pipeline). It can be also manual execution cycle for non-automated test cases. So make sure based on type of execution you plan for required tools and resources. Here some help in making decision on what automation strategy

Test automation pyramid

Building a solid and stable technology ground to resolve business needs is very important. That’s why technology facing test cases usually are more preferable for test automation as they more isolated, they run faster, and less costly — required less time and resources to develop and maintain the scripts. This needs to get early feedback on the question “are we building the product right?”. More deep into technology more faster ROI. At the same time business facing test cases need to be automated at some % as well so we can get early feedback on question “are we building the right product?”. This approach is helping preventing bugs and lower done fixes cost as we can catch and resolve way before production.

• Determine which test cases need to be executed for regression testing in the current project phase (see techniques described above).
• Entry and Exit criteria should be clearly defined so you know when to start and when to call it as Done. Here some example:

Regression testing entry and exit criteria example

Step 3: Execute planned regression suite will required certain steps:

• Prepare the test environment (e.g. test data, tools, software, and hardware) to mimic the production environment as close as possible. Ensure that the environment is stable and ready for testing.
• Execute the test cases tagged to the regression test suite. This may involve both automated and manual testing, depending on the selected approach and the nature of the test cases.
• Record and report any defects or issues encountered during regression testing. Defects should be documented with sufficient details for developers to reproduce and address them.
• After developers resolve reported defects, document root cause analysis and retest the fixes to ensure they have been implemented correctly without impact to existing functionalities. Continue this iterative process until the software meets the exit criteria.
• Evaluate whether the regression testing exit criteria have been met. If all exit criteria are satisfied, consider the regression testing process complete.

Step 4: Report regression testing results and any lessons learned during the testing effort. Prepare test summary reports to communicate the regression testing outcomes to stakeholders.

Step 5: Reflect on the regression phase outcomes and make sure continuous improvement is established — review the effectiveness of the regression testing strategy and identify areas for improvement. Use the lessons learned to enhance future regression testing efforts.

To summarize the importance of effective regression testing strategy — it’s essential to keep a focus on efficient use of resources, timely feedback, and thorough validation of the software application. The strategy should align with the overall testing goals and project needs. Effective regression testing will help the application remains high quality, stable, reliable, and close to “zero defects” with each change or update.

Continue reading how to create a robust testing strategy