UKHSA - DevSecOps

CASE STUDY

Large scale transformation with DevSecOps

Client: UKHSA
Service: DevSecOps

The journey to DevSecOps

In October 2020, during the COVID-19 pandemic, business IT systems consultancy, Capacitas engaged with NHS Test & Trace (later to merge with Public Health England and the Joint Biosecurity Council to form the UKHSA).

Capacitas offered advice on managing batch systems scaling at speed which was well-received.

This resulted in the hyperscale-cloud consultancy undertaking a successful project to build and fully automate a performance testing capability to support the scaling of healthcare systems processing COVID-19 test results - and bringing these into the UKHSA to automate responses to future pandemics.

The work on the project enabled UKHSA to achieve significant successes, both in terms of driving operational efficiencies and cutting costs. Working with Capacitas, UKHSA successfully scaled systems from 300,000 to six million COVID-19 tests per day. The agency was also able to reduce test cycle times from two weeks to two hours on average, and test environment costs by over £2 million in total. In addition, the agency managed to cut defect leakage to zero by automating analysis using machine learning techniques.

The success of the work has empowered other teams within the agency to also ‘shift left’ and take ownership of their own non-functional testing. Moreover, the trust built between the two organisations has led to them working together to evolve UKHSA’s DevSecOps capabilities moving forwards. 

Scoping the DevSecOps challenge

The focus has shifted from performance testing to how the team could roll out the very same principles in developing faster, cheaper, and better solutions across the whole of DevSecOps. UKHSA had formed a central DevSecOps style operating model, enshrined within a specific department within the organisation. This involved bringing together multiple disparate practices within the agencies forming UKHSA, all of which had been working in different ways - effectively in operational silos.

There was little, if any, consistency in the way these different groups operated. In some cases, it was taking over a month to get new delivery changes out into ‘live production’. UKHSA’s CIO was committed to reducing this lag time.

Delivering a solution

Capacitas became involved in building a maturity model and maturity assessment across 60 key teams assessing the tooling being used. The objective was to find a way of delivering DevSecOps consistency across the organisation, raising the capability levels of the internal developers while reducing costs.

Capacitas was able to draw on its own expertise to build a high-level view of how the operating model should work and the toolchains that the new DevSecOps team should be standardising on.

The decisions taken were based on what technologies would meet UKHSA’s needs as well as what is standard in the industry, so that it would be easy for UKHSA to bring in new staff to manage the technologies and maintain costs effectively.

Capacitas also focused strongly on delivering cultural change across the development team to build a more collaborative and less siloed mindset. It identified where good practice was being followed as well as the areas lacking in resources to carry out tasks constructively.

This all fed into the target operating model Capacitas built for DevSecOps, the tooling selection, and supported the automation of measurements throughout DevSecOps.

Capacitas rolled out the DORA metrics framework (created by Google’s DevSecOps research team) which recommends four key metrics organisations should be monitoring across all their teams around speed and efficiency of delivery. The four metrics (deployment frequency, mean lead time for changes, mean time to recover, and change failure rate) are critical in standardising and collecting metrics consistently as they demonstrate much of the value coming from DevSecOps whilst allowing delivery teams to visualise the areas of strength as well as those that require more attention and development.

	Cost Saving Driving transformation within existing budgets Consolidating tooling Optimising cloud Cutting down on vendor teams
	Delivery Speed Automation and ML Reworking processes that added delays Building the culture of product ownership
	Quality and Security 89% reduction in incidents E2E security tooling in pipeline Quality automated and responsibility pushed to product teams

 

Achieving Results

Delivery teams from third parties had been attempting to report on the DORA metrics for some time but had not been able to automate the process. There was no standard stipulating how the metrics should be collated and where the boundaries were for collecting them.

Drawing on its in-depth expertise of DevSecOps, and more specifically, of DORA, Capacitas started to generate automated ways to capture these metrics for UKHSA. Once they had been attained, the consultancy was able to pinpoint where bottlenecks were occurring and assess their root cause.

This procedure helped shape the development of reports which have, in turn, driven the tuning and adjustment of processes and pipelines across the delivery teams at UKHSA.  With these changes made, improvements have been seen month on month across the delivery teams that were onboarded on the journey.

This included one of the teams improving delivery efficiency by 85% over the course of three months by simply tweaking processes based on statistics from these DORA metrics.

DevSecOps transformation

With these metrics understood, Capacitas was able to work with the teams to introduce test automation and change workflows to improve DevSecOps performance and reduce delivery times.

In parallel, the consultancy also focused on implementing cost-cutting measures for UKHSA, working together with multiple teams across the government agency to reveal where there were inefficiencies which they could then resolve.

Benefits achieved

With Capacitas’ help, the DevSecOps function has improved the quality of the UKHSA technology by reducing incidents affecting users while also focusing on efficiency and rolling out cloud best practices to reduce spend across the organisation. Overall, delivery speed has improved by 60% and production incidents reduced by 89%.

Critically, the Capacitas project has also resulted in significant cost savings. ‘Shift left’ optimisations across the organisation have saved more than £1 million while also increasing delivery throughput. The Capacitas initiative has also enabled UKHSA to reduce cloud spend by more than £2 million.

Today, the DevSecOps operating model supports standardised tooling, technologies and processes that enable fast and efficient software delivery and automation.

Other key benefits achieved with the help of Capacitas include:

• A standardised Maturity Assessment Framework defined by the DevSecOps function helps assess the DevSecOps maturity of individual delivery teams and provides support to teams to align with UKHSA and industry best practices

• A common test blueprint also runs a standardised assessment for each delivery team to assess the standard of quality testing and frameworks within delivery teams and helps provide guidelines and support to teams that have lower scores
• The journey to the automated collection of industry-standard performance metrics – DORA Metric – has resulted in a standardised process and automation across the portfolios within UKHSA which ensures engineering best practices are followed and highlights areas of excellence and pain points to the delivery team
• The roll-out of a central DevSecOps approach to Development Lifecycle and DORA Metrics reporting has now enabled teams to conduct interventions and improve delivery throughput and quality

	Capacitas are experts in cloud excellence and our consultants holds deep industry experience. We have a unique and holistic data analytics approach that spans through cloud cost, performance, and scalability vectors to understand and address customers’ specific business and technology challenges. This is not possible with traditional approaches that fail to consider interdependencies across all three cloud vectors.
	Capacitas goes beyond traditional cloud optimisation approaches. We ensure that benefits across the three key cloud vectors are maintained by helping our clients navigate ongoing business and technology changes. We go right up through the entire tech stack into the application thereby typically getting x 4 cost and scaling opportunities!
	At Capacitas, we also focus on transferring cloud knowledge to customers’ engineering teams in a programmatic way, and to a far greater extent than traditional cloud optimisation partners. This ensures that scalability, performance, and cost benefits are maintained for the long term.