Insights

Author: Danny Quilton

Shifting Right is all about learning lessons from production that we cannot learn from testing. Here are four of those areas we can learn from:

• We don’t know how user behaviour will change when our code goes live.
• Scalability of batch workloads
• It's impossible to test all browser-platform-device combinations
• Testing workloads and models are often inaccurate

We don’t know how user behaviour will change when our code goes live

Serverless architectures charge for what you use. For example, with AWS Lambda, you will be charged for the number of times your code is triggered. How does the demand profile per user change? How has the capacity consumed per unit demand changed? 

One client reduced cloud cost by £1.5M p.a. by moving a logging workload from a high to a lower performance storage service. An asynchronous logging service did not require gold-standard performance. 

Scalability of batch workloads 

Many organisations have business-critical overnight processing that must complete by a deadline. If we have a processing window of, say, 12-hours, this is laborious and slow to performance test. 

When compared to the data we can gather from production, those 12 hours only show a small fraction of the picture. For example, with 90 days of production observations, you will have a clear picture of the performance. With this amount of data, what could you learn about scalability? 

It's impossible to test all browser-platform-device combinations 

In B2C, for example, there are at least 63,000 possible browser-platform-device combinations. It’s not feasible to test them all. Many more insights and performance lessons can be found from production observations. 

Testing workloads and models are often inaccurate 

Performance testing often only tests a few workload models with set rules. In production, the workload fluctuates from one minute to the next, and day to day. 

What can we learn by shifting right? Should our testing scenarios be re-calibrated since they are no longer accurate? 

Over the next couple of weeks, we’ll be talking about part 2 of lessons learnt when shifting right.