DevOps is an approach to enterprise software development that aims to remove the barriers preventing organisations from delivering and updating systems at speed. It combines the processes, practices and tooling required to deliver speed without compromising quality.
Taking this a step further is DevSecOps; a set of practices that encourages agile relationships between development and IT operations teams. Organisations, especially when working with a monolithic application, can find a disconnect opens up between these teams which pass ownership between them, undermining efficiency, and performance.
DevSecOps bridges this gap, combining the two functions, integrating ownership at each stage of the development cycle. It fully integrates security as a shared responsibility and encompasses culture, automation, and design. Amid unceasing demand for new applications and updates it helps organisations achieve faster development with higher quality and fewer incidents – and at reduced cost. It boosts efficiency and results in higher levels of governance.
But this evolving methodology requires expertise to handle the increased requirements and greater complexity of software as all organisations face a global shortage of developer skills. And with no one-size-fits-all solution to choose from when it comes to DevSecOps, every organisation is doing it differently, and few are getting what they want from it.
As an organisation, Capacitas have been involved in helping multiple clients through their transition into the world of DevSecOps. Along with way, we see plenty of good practices adopted and applied, while also some less desirable ones that threaten to limit the positive impact and benefits that DevSecOps can bring to product teams.
In this article, I’ll be covering three of the most common mistakes made when transitioning to DevSecOps models, and – most importantly – how to avoid repeating the same mistakes…
Mistake #1: Lack of knowledge sharing and best practice
In an organisation full of numerous product teams, separate teams will have their own ideas and ways of implementing DevSecOps, and will keep the intelligence of their implementation to themselves.
Rather than having the single-minded attitude of “knowledge is power”, “sharing is caring” is very much the order of the day. It is the ideal attitude for helping all product teams move in one common direction. There will undoubtedly be different interpretations and implementations to aspects of DevSecOps (e.g. how CI/CD pipelines build product team-specific code), but these should be different for team-specific aspects.
Solution: Establish company-wide Communities of Practice
Building Communities of Practice across the organisation fosters the culture of knowledge sharing across product teams and typically focuses on (but not exclusively):
Skillset – the technical and softer skills required from product team personnel to operate effectively and optimally in a DevSecOps environment
Automation – ensuring consistencies in QA and CI/CD pipeline frameworks, in terms of the physical similarities, and the knowledge about them community-wide, as well as identifying common tips for adopting automation packs into the product teams’ landscapes
Observability – which metrics to monitor, which observability tool is best to use, what useful metrics should be pulled together into a common shareable dashboard, interpreting results and how to act on those results
Quality – testing approaches, quality of code being written by developers, code review practices, standards around code scanning for best practices and for security compliance
Team ownership – ownership and governance of guidelines of best practices for DevSecOps across the community, ownership of DORA metric tracking and actioning
Architecture – ensuring cohesion in a direction of travel, what cloud provider is agreed to be used, how should resources be set up, any Terraform code provided to spin up resources in an automated fashion
Requirements – is there a common approach for gathering requirements from the end users? Is there a common process in place for feeding these requirements into a commonly agreed backlog tool? Are there common stakeholders issues that Product/Business Owners must face?
There are also knowledge sharing opportunities intra-team around adoption of other important DevSecOps enabling practices, such as centralised tooling for observability and pipeline tracing, cloud platform setup and Quality Assurance test automation frameworks.
How to set up a Community of Practice.
Setting up your first Community of Practice doesn’t have to be a painful process. Here’s what we suggest:
Define scope of what the Community of Practice will cover (refer the diagram above)
Work with the product teams to help appoint Community of Practice champions and attendees.
Create an initial Knowledge Base for the community to build on
Work with Capacitas to help with kick-off and run initial launch, follow-on sessions and seminars
Mistake #2: Lack of budget / support for tooling
There is resistance from the CTOs in some organisations to spend time and money looking for off-the-shelf tools and solutions that can help enhance the DevOps landscape within an organisation. Rather, the approach is to ask product teams to produce their own solutions
Our Capacitas-commissioned DevSecOps research found that DevSecOps-embracing organisations were failing to use automation appropriately. A big part of this – from my experience – comes from lack of investment in fit-for-purpose off-the-shelf automation tools. While this has short-term financial benefits, it will put product teams at a disadvantage and in a tricky position; they will find a free-and-minimum-value product, or product teams will be asked to develop in-house automation and observability monitoring capabilities. Naturally, the different products teams choose the latter option, and you end up with multiple different nuances of the same concept!
Solution: Empower teams to select tooling
Empower the product teams to be open-minded and shop around for fit-for-purpose solutions – particularly around automation and observability monitoring – and trust their judgement on when necessary, spend on off-the-shelf solutions will provide significant benefits
There are so many off-the-shelf tools and well-supported products that should be utilised when adopting automation and observability monitoring in DevOps.
On the quality/test automation side:
SonarQube - an industry standard software tool used to measure quality metrics within a code base
SpecFlow - behaviour-driven development automated framework, based primarily in C#.NET and Gherkin
Playwright - widely considered to be a more robust, more resilience, easier to build front-end & back-end automated test framework provider than traditional Selenium,
Cypress - Node.js-based front-end test automation tool for regression testing of web applications
On the observability monitoring side:
Dynatrace - widely considered to be best-in-class provider of observability monitoring platforms
Splunk - big data platform provider, designed to simply big data collection and management
OpenTelemetry - open-source provision of APIs, SDKs and tools used to capture performance metrics, traces and event logs from cloud-native applications
DataDog - observability platform, consistency of multiple products to help build, test, monitor, debug, optimise, and secure product teams’ software
SolarWinds - provider of software to help product teams manage their networks, systems, and infrastructure
Keep in mind that there has to be a balance around costs; letting developers spend as they wish needs to be tempered, otherwise there may be significant bills racked up (I can say this, as a developer myself!). However, investment in tools that clearly provide significant benefit, will offset shorter-term investment pain with larger, longer-term efficiency gains for the product teams.
Mistake #3: Forgetting about people and processes.
Organisations who try to adopt DevSecOps often focus too much on technology, and not enough on the people and processes that would support the technology.
It is perhaps natural for organisations to assume that technology should be the sole focus when adopting DevSecOps correctly. However, you can have the best state-of-the-art technology in place but if the processes aren’t in place to fully utilise the technologies, or if the product teams and supporting stakeholders don’t fully embrace and get behind the DevSecOps methodology, DevSecOps will be implemented without having the expected benefits and optimisations.
Solution: Implement governance around people, processes, and technology.
Governance needs to be put in place when adopting DevSecOps around technology, people and processes, to ensure that all three areas are equally considered and creating the best engineering culture possible.
The ideal DevSecOps engineering culture encompasses various principles that are enabled by the three foundations already mentioned – people, process, and technology.
Principles:
User-centric data driven – building for the needs of real users; having requirements based on real user data and datapoints from existing products; actively seeking to gather more data to better inform future choices, as well as keeping the cycle of deliver ➝ measure ➝ repeat as short as possible to keep changes relevant to user needs.
Multi-disciplinary teams – taking ownership of the technical product your team is delivering; owning the quality, security, cost and changes to your product; owning the full lifecycle of your engineering product.
Problem solving – building products because you know the problems you’re solving; being flexible to an adaptive environment means that new problems are always presented which need solving; solving the next problem means the product continues to evolve in line with organisational needs.
Fast feedback – collecting data on products immediately to drive continuous improvement; monitoring not just to catch problems, but primarily to drive engineering direction; measuring both products and processes (for example, using DORA metrics) to constantly improve.
Foundations:
People – DevSecOps is founded on a culture of collaboration and shared responsibilities, i.e. ownership of product, development, security, test and operations
Process – the DevSecOps process brings end-to-end visibility and control of quality, security and cost into product teams
Technology – automation is critical to remove friction and enable fast feedback loops. A cloud-first approach increases flexibility of services, deployment and resource management and allows an infrastructure as code approach.
These principles and foundations, when combined and implemented together, (and well governed) will ultimately allow organisations to see the full benefits that the DevSecOps approach are designed to provide – improving product quality and speed, while also delivering value for money.
For more thoughts on overcoming mistakes in DevSecOps adoption take a look at our recent DevSecOps research paper "DevSecOps: Bridging the gap or widening the Divide?": https://hs.capacitas.co.uk/devsecops-report-2023
