Performance Testing and DevOps

Introduction

Why is DevOps Important?

DevOps aims to remove the barriers which prevent organisations from delivering and updating systems at speed. Today not only do we want to make changes to enable users to do things faster, but the rate of change itself is getting faster. This means the time from idea to delivery must be reduced to a minimum, one of the factors in the drive for continuous delivery.

In the past, a major barrier to faster evolution was the organisational structure around the software lifecycle. Architects would pass their designs on to the developers, who wrote code to meet the business requirements. Testers (functional and non-functional) would then have their say before releasing the code into the responsibility of operations teams. Issues were then passed back to the developers before repeating the cycle.

A long cycle length was accepted in order to reduce risks and improve quality. DevOps, however, combines the processes, practices and tooling required to deliver speed without compromising quality.

By automating test activities, practicing continuous integration and putting processes in place to provide quick feedback, DevOps enables continuous delivery of features and improvements aligned with business objectives. A team of DevOps engineers takes full ownership of a product or system, removing barriers and ensuring delivery focus.

Why Does Performance Matter?

While speed of delivery has become ever more important, the same is true for the performance of the systems which are delivered, as shown by our research. How can you be confident that code you release will perform as required by the business?

Historically, of course, the answer involved dedicated teams, expensive toolsets and a lot of time! For today’s DevOps teams a new way of working is needed, combining processes, practices and tooling to ensure that change is delivered seamlessly and without introducing performance risk. Continuous delivery requires continuous performance.

The fact is that performance, and to some extent security, are quite different quality measures from functional quality. If something is broken functionally it will usually be quickly evident – as soon as the faulty product is released, the error can be picked up and feedback used to correct the problem.

However, performance issues, bottlenecks and stability defects may remain unnoticed for weeks or months, until the next annual peak or until data has grown to a certain level. And by then, there are hundreds – or thousands – of individual releases to unpick.

How Can You Ensure Performance in DevOps?

Having the right strategy to ensure DevOps performance is critical to delivering what the business requires – swift delivery of fast, efficient systems. Here we’ll describe 6 key components which make up any successful performance strategy, and how these areas need to be transformed to successfully implement a DevOps approach:

Performance Engineering Process
Tooling
Environments
Data
Reporting
Roles and Responsibilities

Performance Testing Process

The first thing to get right when implementing DevOps performance is to ensure that the right process is in place. Often teams start by looking at tooling (perhaps because that’s more exciting!) but, without the right process, tools will be chosen wrongly and poorly implemented.

A good DevOps process consists of 6 activities:

A combination of these activities will be carried out for any change delivered by the DevOps team.

Risk Assessment

Objective

A good process starts with understanding the performance risk of any change. Taking ownership of this risk as a DevOps team. Planning steps to mitigate it.

Approach

Performance risk assessment takes place at the same time as maturing, planning, estimating. As soon as the team starts thinking about the items in the backlog, they start thinking about risks to performance.

Having this performance mindset is critical to making sure that the end product performs – while all the team should have the same performance focus, it can help to appoint a Performance Champion to lead on it; more on that later. Performance NFRs need to be reviewed and updated where needed – for help on setting NFRs see our NFR Template.

It’s helpful to classify the risks, as well as the requirements, across the 7 key pillars of software performance.

Performance Design and Build

Objective

Use team mindset to eliminate risks early. On one hand using knowledge of performance behaviours and antipatterns to improve implementation. On the other using understanding of change and implementation to improve smart test design, scripting and data.

Approach

There’s no need to wait for code to be checked in to start improving performance.

The first step in a performance design is to review the implementation for performance anti-patterns – behaviours which will impact performance. At the same time the team will start updating scripts, data, mocks etc. to prepare for performance testing.

These alterations need to be ready before changes are checked in so that tests are ready to run automatically without delaying delivery.

At the end of this phase of the process everything is ready for the DevOps engineer to check in their code and deploy it… well, almost.

Performance Unit Testing

Objective

Reduce risk of build failures during continuous integration.

Approach

An important stage in the process which often gets overlooked is assessing performance at a unit or profiling level, before the code is actually checked in and deployed.

This is down to the individual developer, using profiling tools (often available in developer’s IDEs, or sometimes specialised tools may be required) or timed unit tests, to provide rough measures of time and resource taken by different high-risk code areas.

When this is complete, the developer will be as confident as possible on the performance of the code and is ready to start automated performance testing.

Performance Component Test

Objective

Test for a subset of key pillars of performance in a component environment.

Approach

There are two levels at which tests may be run at this stage in the process:

Service component test – in a CI environment for performance, focussing on the performance of an individual service.

Product performance component test – in a Team Performance Test Environment, focussing on performance of the entire product delivered by the team.

For either type of test, mocking of any interfaces out of scope is crucial to ensure the correct focus.

Everything from setup of the environment, through to analysis of the platform, will need to be fully automated using appropriate scripts and tooling to achieve full continuous delivery.

For more detail on what needs to be considered for a successful performance test, see our Performance Testing Primer.

At the end of this step, the team will have a full view of the performance of the product or components for which they are responsible.

Performance Integration Test

Objective

Test for all key pillars of performance in an integration performance test environment, when and where an integration performance risk is present.

Approach

An integration performance risk means that there is some kind of interaction between the products developed by multiple DevOps (or other) teams. A single team (often separate from the individual product teams) will be responsible for this risk.

This team will carry out the same steps as in the Performance Component Testing, but in a fully integrated environment without any mocking present (except potentially 3rd party systems).

This is particularly important in large organisations with dozens of product teams producing component parts of the final system but needs to be considered with any number of teams.

Once this is completed, the system changes are ready for release to production. But the process isn’t finished yet…

Production Performance Assessment

Objective

To take ownership of performance where it really matters! To improve the Performance Engineering process by completing the feedback loop. To respond quickly and proactively to any production performance incidents. To mitigate the risk of environmental, operational or usage differences between test and production, and to manage any other residual performance risk.

Approach

There are three types of production performance assessment which should be implemented where appropriate, depending on the Performance Risk Assessment:

Performance health check – using dashboards, alerting etc. to compare current performance to past baselines and NFRs.

Production performance model validation – validating the model used for constructing performance tests.

Production performance testing – repeating performance testing in production.

Production performance assessment is never completed, but is an ongoing activity, continuing to track and own any risks while delivering the performance the business asked for, as new changes continue to make their way through the process, are deployed, and utilised.

Tooling

Automation is crucial to achieving the goals of DevOps. A number of different types of tool will need to be used together to deliver the process. Here we’ve assumed that certain pre-requisites will be in place – e.g. that developers have the correct IDE, source control and an automation server supporting a working pipeline. We’re focussing on tooling which requires special attention when ensuring performance in DevOps.

Performance Test Tool

The first thing you need is tooling to create and run a performance test. This will need to be embedded into a delivery pipeline to fully automate test execution.

Server-side performance – You will need a tool which will generate load against your target system. This may be in the form of http requests, as in the case of a web application, or whatever other type of protocol your system uses. You may need to generate requests that simulate full user actions, or which target individual services or endpoints with specified data sets. The tooling must support testing of services and APIs as well as user interfaces.

Client-side performance – In the case of web applications, it is increasingly important to also measure time on the client, in this case a browser. By having a tool which will capture rendering, script and download times, you can measure the time to actually deliver content to the user, which is crucial. A client could also be an app running on a mobile or any other type of user device.

Performance Test Support

Before and during test execution, there are a number of other areas in which tooling is crucial to eliminate manual effort and to capture the test results.

Risk Mitigation Planning – Having a tool to capture your test plan, and any other test actions that are needed, ensures consistency and repeatability of the testing. It should be able to track the risks which have been raised and their status (open, mitigated, accepted, defect occurred, etc.)

Service/HTTP mocking – As mentioned earlier for most tests, there may be a need to mock out external interfaces. There are a number of specific requirements that need to be considered in a mocking tool for performance use, due to the load it will be placed under and the need to replicate time as well as data.

Infrastructure/server monitoring – Without monitoring performance of the underlying technology defects will go unnoticed and root cause analysis will become very difficult. This may be done using built in system utilities or 3rd party toolsets.

Whether the technology used is physical infrastructure, cloud based or completely serverless, monitoring the key service metrics is critical. Monitoring should include servers, networks and storage as well as any other components. This needs to be in place both for test and production.

Application monitoring – Application Performance Monitoring (APM) tools are valuable sources of data about the performance of the application under test. Using an APM or log-parsing solution means quicker turnaround times when issues are detected. Being able to create dashboards for production monitoring is also important.

There may be other specific types of tool in this category depending on your sector – for instance in ecommerce you could use RUM (real user monitoring) and conversion tracking to monitor real user experiences. Application Monitoring also needs to be in place both for test and production.

Performance Test Analysis and Reporting

After the test is complete comes the difficult part: what are the results and what needs to be done about it? Again, to achieve full Continuous Delivery this needs to be automated into a build pipeline.

Test results analysis – This is perhaps the hardest part of the process to automate but getting it right is crucial. False positives cause delays and slow down delivery. And false negatives – not spotting performance defects – are perhaps even worse!

The analysis should include checking against known thresholds, assessing against past baselines and identifying known performance antipatterns. Getting this right is critical to successfully achieving continuous delivery – without a reliable means of signing off performance automatically, changes cannot be automatically deployed to production.

Test status reporting – More on this under the ‘Reporting’ section below but having a tool which will allow you to track change status, risks, defects and tasks, summarised on easy-to-read dashboards is important.

Defect tracking – Ideally this should link into your test status reporting as defined above.

For each of these tooling categories it is important to select a tool which will:

Integrate with the other tools selected
Fit with the organisational model and process
Match the technology being built and the skillsets of the engineering teams

Environments

In a cloud first DevOps organisation, environments become as much a part of the solution being developed as the software code. You will need to plan how environments will be set up and automated into the development pipeline.

First, work out what environments will be required and what confidence level each environment is designed to give you. The diagram below shows a fairly standard approach where there are 3 levels of test environments for performance:

CI – here this refers to an environment used for individual service component testing
PTE – standard ‘full’ Performance Test Environment for the team, containing all services which the team delivers
IPT – an Integrated Performance Test environment providing coverage of integration level risks across teams and platforms

CI Environment

The CI performance stack will be spun up as required on build to enable a performance component test of a single service in isolation.

This will typically include the following components:

Load injector - provisioned with the selected performance test tool for directing load at the service under test
Service container - hosting the specific service being built
Mocking framework - containing mocks for all nearest neighbour services (or alternatively services may be designed to utilise a central mocking host)

Team Performance Test Environment

The team performance test environment (PTE) contains all the services owned by a single DevOps team. The PTE will be used for running performance component test on build for the entire product delivered by the associated team.

The PTE will include:

One or more load injectors (depending on team and expected volumes)
Service containers for all services delivered by the team
Any other components owned by the team or necessary for service communication such as service buses
Mocking framework for external services only (services owned by other teams)

Integration Performance Test Environment

An environment will be required to conduct representative performance tests across the full solution, including all products built by the various teams, and any external dependencies. This will be a single Integration Performance Test Environment (IPT). An IPT should be managed in the same way as the production environment so that releases do not hit the environment until they are ready for production.

Depending on the type of Continuous Delivery (CD) being used and the level of risk acceptable, it may be deployed to in parallel with production, to allow integrated tests to proceed without delaying functionality release.

The environment will contain the following components:

Multiple independent load injectors
Full deployment of services delivered across all teams
External legacy components
Mocking where required for 3rd party components

For each environment, it is important to ensure that it is designed efficiently to deliver performance while keeping costs under control. Non-production environments account for a significant proportion of cloud cost in most organisations. If performance environments are not sized and maintained appropriately, they can bring test costs higher than actual production hosting!

Data

The reason data deserves special attention in a DevOps world is that it is crucial for getting meaningful results and needs to be automated in such a way that tests are representative and repeatable. Some tools exist which may help with managing this, but the hardest part is ensuring that the data being produced is correct and meets all the criteria.

There are three types of data that need to be considered for any performance testing:

Bulk Data

Bulk data is relevant where there is a data repository of some kind, this could be a traditional or NoSQL database.

The following factors need to be considered:

Is there the right volume of data to simulate not only current but future production scenarios?
Is data of the right type and range to make any interactions meaningful?
Have any new tables or stores been considered and prepopulated to expected volumes?

Reference Data

Reference data is created in data stores or elsewhere for the purposes of referencing in a test script. This is the most complex data type to manage and requires special consideration when automating data creation, as the data needs to be available both to the script and in the target system. This may involve automating extracts of data from the target system prior to script execution.

Parameter Data

Parameter data is referenced in the script without any specific requirement to create it anywhere else as part of the test scenario.

Again, ensuring that the range of data is understood and replicated is key to achieving meaningful test results. Incorrect parameter data may lead to unrepresentative use of cache or buffering or may ignore situations which are particularly heavy on performance.

Reporting

There are two types of reporting to consider

Tracking risks, findings and issues through the DevOps performance process
Monitoring production performance to ensure a good feedback loop – critical for successful DevOps adoption

Tracking

There are several types of information which need to be stored and reported on during the DevOps change lifecycle:

Performance risks for user stories
Performance defects against user stories
Tasks for investigation of test findings
Performance signoff status of user stories

Managing these through a single task/defect/risk tracking tool such as Jira is ideal as it provides one point of access for all this information. This means that tracking dashboards can be produced which combine this data into a single view for the product, for example as below:

It’s particularly important that performance signoff is communicated clearly and effectively, as it needs to both make sense to the whole team and product owners or other sources of requirements within the business.

Production dashboards

Tracking performance in production is at the heart of maintaining a short feedback loop and delivering software quickly. There will always be some remaining risks which need to be identified in production and without the right data and reports, production performance assessment becomes impossible.

There are 2 high-level types of dashboards which should be created to track performance in the live system:

Team Performance Dashboard

The team performance dashboard should be aligned to a subset of selected metrics from your Service Metric Framework
These should be metrics for which there have been NFRs defined in order to make the dashboard meaningful

Integrated Performance Dashboard

The integrated dashboard should focus on elements which are within the remit of the performance integration function, including:

End-to-end User level metrics (response time, concurrency, throughput)
High-level view of underlying shared platforms
High-level view of any system failures or degradation which may cause resiliency issues

Roles & Responsibilities

Team performance accountability

A DevOps team has ultimate accountability for the performance quality of the product they deliver and operate. This resides with the team as a unit and all team members contribute to fulfilling the responsibilities that go with it
The DevOps team owns all performance risk inherent in their developed product. They are responsible for ensuring that the risk is reduced to the desired level and that residual risk is managed throughout the lifecycle
The DevOps team will also be responsible for collaborating with Performance Integration on any mitigating and managing risks which sit across the enterprise solution rather than purely within their product
Having the right skills in the team is crucial to successful delivery. This includes both technical skills around understanding performance anti-patterns, creating test scripts and performing analysis, as well as business skills to understand how the system will be used and the impact of performance on users, budgets and business events

Performance Champion

Without the necessary performance skills in place a DevOps team will fail to take full ownership of performance. A Performance Champion is a member of a DevOps team who has a performance focussed skillset, understands performance anti-patterns, and is able to take accountability for coordinating the team to deliver a solution with the required performance.

The task of a Performance Champion should be to mentor and share aims, attitudes and knowledge regarding performance for assimilation by the rest of the team. The role of Performance Champion is typically important in a new DevOps team and becomes less important as time progresses (if they're carrying out their role correctly!).

Performance Integration

In an integrated system where there are multiple product teams contributing towards a combined whole, there will be a need for a defined performance integration function if any of the following are true:

Shared underlying infrastructure
Calls between product services
Calls into legacy systems
Services from multiple product teams being rendered in the same page

These introduce risks which are not owned by any one product team. Performance integration is accountable for owning these risks and mitigating them to agreed levels, by coordinating between product teams. This role may sit with a specific engineering team, a platform team, or a dedicated performance function.

Performance Integration must also be responsible for managing NFRs at a business level and how these break down between different product teams, where there are calls between product services or multiple services rendered in a single page.

Conclusion

Now you know what is needed to provide confidence in DevOps performance from your team. How do you implement this strategy, to enable fast delivery of fast, efficient solutions?

Maintaining a performance process and artefacts in a DevOps organisation needs to be low effort and highly automated. There will be activities that need to be carried out constantly by the DevOps team, but these should become their normal way of working. Performance will be embedded in their delivery. The most difficult parts of the process are firstly to build a strategy which will work for you and your teams and, secondly, to undertake the transformation to embed it.

This requires technical skills around scripting and automation, as well as business skills in creating and embedding processes, plus interpersonal skills for creating a culture of performance in the organisation.

It also requires performance skills to design the right tests, environments, and data, to build automated signoff analysis and to create dashboards and reports that tell you what you need to know immediately.

These are not all part of most developers' or DevOps engineers’ skillsets, and it is non-trivial to build them up. A transformation process can take 6-months to a year in large organisations with many teams but needs to be prioritised to start delivering benefits immediately.

Once in place, the business benefits of embedding performance in DevOps should be quickly apparent. Code will be delivered faster, while still satisfying non-functional requirements. There will be fewer incidents in production, enabling the team to focus more on development and further increase velocity. The costs of environments, both in production and development, will reduce as efficiency is built into the system.

Capacitas brings clients a structured approach to performance in DevOps, with skilled consultants focussed on performance, and the technical tooling and automation to enable fast performance signoff in continuous delivery. Our focus in DevOps is on delivering the processes, skills and tools needed to enable teams to become autonomous units, owning the performance of the solutions they deliver, and ensuring that not only do they deliver quickly but also meet the performance needs of the business and users.

For more information and to discuss how this could work for you, get in touch.

Get in touch

Key Takeaways

Define your process first, not the tooling!
A DevOps team has ultimate accountability for the performance quality of the product they deliver
The DevOps teams should be responsible for collaborating with a Performance Integration team on managing risks which sit across the enterprise solution rather than purely within their product.
Define who is accountable for Integrated performance (intra-team)
Tooling should go beyond test execution; deploy Test analysis, APM, RUM, Defect-tracking, Risk mitigation planning and Cloud & infrastructure monitoring tools
Cloud performance environments must be sized and maintained appropriately otherwise that can exceed production hosting costs
The single most important and difficult component to automate is performance test analysis
Performance is not simply response time and throughput, it’s way more complicated than this, there are 7 pillars of performance
The Ops of DevOps – deliver production performance assessment on release
Deploy production performance dashboards to ensure a good feedback loop – this is critical for successful DevOps adoption
Upskill your developers with specific performance and analysis skillsets
This will deliver business benefits: ship code faster, fewer incidents in production, lower cost of development and production environments

Thomas Barns

Principal Consultant

About the Author

Thomas Barns is Risk Modelling and Performance Engineering Service Lead at Capacitas, responsible for service definition and ensuring consistent best practice across projects.

Over the past 10 years he has worked on large projects providing capacity and performance expertise to clients and owned the roadmap for developing Capacitas’ technical software solutions. During this time, he has seen a big shift in how software engineering is undertaken and viewed by the business and has built on this to introduce more effective and efficient performance risk management processes. This has meant shifting focus away from large scale system testing to a full lifecycle approach, alongside research and development in automated data analysis.

Thomas has recently been defining and governing Performance Engineering processes and standards for a multi-million pound multi-vendor programme of work at a FTSE 100 company, and helping clients define performance approaches for DevOps.

Introduction

Why is DevOps Important?

Why Does Performance Matter?

How Can You Ensure Performance in DevOps?

Performance Testing Process

Risk Assessment

Performance Design and Build

Performance Unit Testing

Performance Component Test

Performance Integration Test

Production Performance Assessment

Tooling

Performance Test Tool

Performance Test Support

Performance Test Analysis and Reporting

Environments

CI Environment

Team Performance Test Environment

Integration Performance Test Environment

Data

Bulk Data

Reference Data

Parameter Data

Reporting

Tracking

Production dashboards

Team Performance Dashboard

Integrated Performance Dashboard

Roles & Responsibilities

Team performance accountability

Performance Champion

Performance Integration

Conclusion

Key Takeaways

Thomas Barns

About the Author

Bring us Your IT Challenges

Navigation

Find Us

Contact Us