This blog explores how to load test to very large user volumes through:
- Automation of Load Injection
- Test Automation
- Automation of Test Analysis
Our client was preparing for their mobile app for a massive annual peak in traffic.
Confidence was low - during the previous two years the service had buckled under peak demand and crashed. The client was determined to avoid any repeat and asked us to load test the live service in order to prove it was ready.
The focus of the test was a back-end API service, responsible for serving data to the mobile app.
By analysing the demand patterns from previous peaks, we ascertained our testing targets of 500,000 virtual users and 15,000 transactions per second (TPS).
We used a suite of software, called Cloud Load Testing (CLT), to deliver:
- Automation of Load Injection
- Test Automation
- Automation of Test Analysis
1. Automation of Load Injection
Performance testing with 500,000 users requires a large number of load injectors. Our CLT suite provides us with a mechanism to create numerous load injectors, and, for the purposes of controlling costs, to destroy them easily as well.
We use Amazon Web Services (AWS) to host our load injectors. This enables us to globally distribute our load (which is important to ensure we don’t overload the content delivery network that is in front of the service we test).
2. Test Automation
Test automation is required to
- Upload test data and scripts
- Collect results
- Schedule tests
- Execute tests
We've developed automation to run in parallel across multiple load injectors. Through a process of improvement we have reduced the time to perform some tasks from 75 minutes down to 5 minutes.
3. Automation of Test Analysis
In terms of time saving, we have found the most value is in the automation of the analysis of performance test results.
Load testing to 500,000 users creates enormous data sets which need to be parsed and mined in order to find performance issues.
We automated the processing of multiple results files. With reference to the load ramp profile, we use a series of algorithms to detect performance anti-patterns in the data sets. This allows us to identify and prioritise test defects.
By automating the test analysis we have been able to quickly find issues and shorten the feedback cycle to the client.
Summary
Load testing to 500,000 users requires automation to manage the large amounts of data involved and numerous cloud load injectors. The 3 key elements of automation are:
- Automation of Load Injection
- Test Automation
- Automation of Test Analysis
To learn more about performance testing and how it can benefit you - download our performance testing primer here.
