Does every change need to go through a thorough quality assurance process?
It’s unlikely that a cookie-cutter QA process takes into consideration Deployment Confidence. If it takes 30 minutes for a developer to merge and deploy changes to the staging environment for testing, then a QA tech can access those changes and test for another 15 or more minutes, you’re looking at about three-quarters of an hour worth of work.
Before anything even reached staging, there should have been a code review, which perhaps took another 15 minutes of a different developer’s time (assuming that the code review was perfect).
Perhaps the Product Manager already reviewed the changes in her local environment, consuming another 15 minutes of time.
All and all, we’re looking at a total of 1.25 hours of work spent on quality assurance in this case. Over an hour of quality assurance, all for one change.
What is the added value of the 1.25 hours of Quality Assurance? Did we stop to consider the Deployment Confidence for this work item? If the risk of breakage was low, the 1.25 hours spent on QA was probably overkill. The work item only needed a code review and perhaps a spot check by the Product Manager to make sure that the software behaved correctly. We could have avoided the staging deployment, in this case, avoided the use of a QA tech, and we could have delivered value to production faster.
Standard QA Process
- All team members should rebuild their environments on a regular basis, usually when pulling in new changes from the remote repository.
- Developers should test their code to the best of their ability before they submit their work for a peer’s code review.
- Someone else on the team performs code review: Code review means you read the code carefully and you pull the code into your local environment to test that it is working. Bonus points if, as a developer during code review, you have the time to test the specific feature change to ensure its behavior is appropriate.
- Review each pull request for a measure of Deployment Confidence: This isn’t an exact science, but a rough risk estimation is possible:
- Low Deployment Confidence: the app needs thorough testing in a staging environment before deployment to production. This staging deployment should mimic, as much as possible, the deployment to production: including running migrations or other “launch-checklist” type items. The goal here is to surface any issues that may are likely to be encountered during the production deployment.
- High Deployment Confidence: Code review and local testing is enough time spent before production deployment.
- Production rollbacks must be quick and painless. If you can’t easily revert changes in the production environment, spend time on this first. Production stability is paramount, and if production is fragile, then breakages will occur no matter the complexity of changes made.
It’s a better use of time to improve production stability than it is to add extra layers of process and time spent to deploy each change to production.