Regressions and how to avoid them
Software bugs are a part and parcel of software development process. No software will ever be bug free, it's just we try to build the feature in such a way that users will not run into potential issues in the most common cases.
Software bugs are introduced all the times with every new feature built, every patch or even with simple configuration changes. As changes are rolled out, engineering teams retest the application to ensure everything is working as expected. This process of retesting older functionality with new changes is called Regression testing - the idea being whatever was working before should not break with the new changes.
The regression testing methodologies vary widely depending on the team size, velocity of the changes, maturity of the team and processes. But regressions still happen all the time and are a concern for most teams.
How can regressions be avoided?
There are fundamentally two reasons for regressions to occur:
Assumptions - People assume all the time that certain code change is so simple that it will not break anything. That's assumption is the main reason for not testing certain functionality and so a regression would happen. The fundamental principle of quality engineering is - Never assume. Never assume that certain thing will work, always test!
Lack of Automation - It's practically impossible for even a large team to retest the whole application however long the release cycle is without automation. Automation is the only sure way to retest all the critical functionality before new changes are rolled out to production.
Regression Test Suite
A fully automated regression suite is a must have for any engineering team to avoid high severity regressions. This regression test suite must cover all critical flows that are of high importance to the customers and no changes shall be rolled out to production without running these tests. Regression tests must be executed against development, staging and pre-prod environment for any change including a simple configuration change to the environment.
In theory, a fully automated regression test suite guarantees that a code change will not cause a high severity issue (P0, P1 or P2) in all the important flows. Having a regression suite gives better confidence to the engineering team to rollout changes and not having to worry if a deployment would ruin their weekend or an upcoming holiday.
A static test suite is useless
Having a regression test suite is great. But a regression test suite cannot be static. For example, if you have 10 tests for v1 of your product, you cannot have the same 10 tests in your test suite while you are deploying v2.
A regression suite is always dynamic, has to be maintainable and always increasing in size with time.
While it's common for large organizations to spend time and resources for building their own dedicated in-house test infrastructure - smaller teams or teams that need to quickly get to market will not have enough resources to have a dedicated test infrastructure.
Such teams can use readily available monitoring platforms that can continuously run API tests or end-to-end functional tests against their dev, staging or even production environments. This is to ensure that all the required flows are working as expected all the time and get alerted when something is broken.
Monitoring platform such as DevRaven allows you to run end-to-end functional tests, monitor workflows, run integration tests and also simulate real-world user behavior during test execution. This document walks through an example Workflow monitor setup process.
Features such as multi-environment monitoring allows you to run the same tests on all your environments continuously. If there is a new change that breaks certain functionality, those get caught in your dev or staging environment before they affect your customers in production.
While regression issues are frustrating and a major concern, by employing proper testing and monitoring strategies with a good test suite most(if not all) high severity regression issues can be avoided. If you cannot invest time for test automation, maintenance of regression/monitoring suite and always spend time on new features, regressions are bound to happen.
Full disclosure: I am the founder of monitoring platform, DevRaven. DevRaven is an easy to use monitoring platform for running end-to-end synthetic tests or API tests. New tests can be added with simple to use no-code editor or ready to use recipes. No coding required!
Sign up for your free account at devraven.io. No credit card required to sign-up and use the product for up to 12500 monitoring checks per month for free.