Continuous Integration is designed to ensure quicker delivery of changes to production. But we still need to ensure these changes meet perform to expectation. How can we deliver performance assurance in the fast moving world of CI ?
There are 4 key ingredients:
- Automation
- Risk-driven approach
- Performance engineering strategy
- Smart test analysis
1. Automation
In a CI environment, speed and accuracy are some of the biggest challenges. You need to keep up with the increased level of change, but ensure that you are not creating noise. Automation of test execution and test analysis can give you this edge.
2. Risk-driven approach
A risk-driven approach ensures that you only concentrate on the sources of performance risk and can help you to keep up with the velocity of change. Not all changes need to be tested. This goes hand in hand with automation as automating the testing process should free up the time to spend on assessing risk. There are 6 key sources of performance risk for which each change should be assessed and appropriate mitigation actions designed:
3. Performance engineering strategy
Define a strategy for testing in continuous integration. This could mean mitigating medium/low risks through gradual rollouts, monitoring in production, modelling or production testing. This video describes the strategy used at Arcadia
4. Smart test analysis
Pattern matching is a technique the should speed up the traditionally lengthiest part of the performance engineering process; analysis. By using these techniques and defining what “good” looks like, we can automate the process of finding “bad”:
The next step is then to take that one step further with smart analysis, and using machine learning to gradually perfect the pattern matching process, reducing the frequency of false positive results.
