One of the questions I often get asked is: Why is a cloud budget forecast needed? The answer is simple. Like any major expenditure, cloud budgets need to be optimised and aligned with business objectives to maintain cost predictability and prevent overspending.
However, creating a forecast isn’t as simple. Industry research highlights that accurately setting cloud budgets is often challenging, with significant consequences for businesses.
- A McKinsey & Company survey found that 75% of cloud migrations exceed their budgets, with 28% of those exceeding by more than 20%.
- In a PwC survey of UK business and tech leaders, 40% reported that budget constraints were the biggest barrier to realising measurable value from cloud technologies.
In practice, cloud budget forecast inaccuracies arise for several reasons, including:
- Inaccurate predictions of business demand growth
- Misunderstanding complex cloud pricing models
- Poor observability, resulting in insufficient data to properly size and validate cost models
- Ineffective cloud cost control policies, leading to waste and over-provisioning
- Miscommunication between DevOps and financial teams, causing inefficient resource usage
By addressing these common issues, you can improve the accuracy of cloud forecasts and prevent costly budget overruns. Integrating cloud budgeting into DevOps will significantly enhance cloud spend management, aligning it with the agility and speed that DevOps practices offer.
So, how do you do it? And how can DevOps help? By considering these 5 steps to setting the cloud budget.
Step #1 Set the cloud budget early and validate continuously
Creating a cloud budget forecast involves a complex calculation that is crucial for predicting costs. Initially, these forecasts can be inaccurate — often by 50% to 100% — until the model is thoroughly validated and refined. DevOps offers an ideal framework for this validation process, providing continuous real-time data on cost, usage, and performance from the earliest stages onwards through to load testing at peak plus volumes.
Though development environments may not operate at full scale, they present an excellent opportunity to test assumptions and usage patterns, albeit at a reduced scale. This allows teams to continuously validate external factors like the 'demand signal' capacity driver alongside internal testing.
Building a cost model is much like developing software: it requires continuous iterations of refinement. All elements—code, logic, and assumptions—should be validated throughout the DevOps cycle. Revisiting and refining the cloud cost model as the project evolves delivers greater accuracy in budget forecasts.
Step #2 Ensure your observability needs are met
You might think you have covered all your observability requirements to create and validate an accurate cloud budget forecast — but think again. Ask yourself these three key questions:
- Are you actively monitoring, capturing, and reporting data on cloud costs, usage, and performance?
- Are you doing this consistently across all development environments?
- Do you have access to the correct cost model parameters from your cloud service provider (CSP) or cloud cost advisor?
If the answer to any of these is no, it is time to address those gaps. Failing to do so will lead to poor assumptions and errors in your cloud cost model, resulting in an inaccurate cloud budget forecast.
Step #3 Anticipate vulnerabilities in the cloud budget forecast
It is important to acknowledge that no cloud cost model will ever produce a flawless forecast. It is generally acceptable for forecasts to have an error margin of up to 10%, though a more ambitious target would be an accuracy range of 2%-5%.
With this in mind it is essential that you identify where the largest forecast errors are likely to be introduced and prioritise these areas for improvement in future cycles. Aligning these improvement cycles with DevOps release cycles ensures that any investigation requiring engineering resources happens smoothly and efficiently.
Here are some common vulnerabilities to watch out for:
- Inefficient management of DevOps environments: Poor oversight of testing and production environments can result in resource wastage and overprovisioning. These inefficiencies will escalate without clear policies on how cloud capacity should be used and automation to manage resources.
- Unreliable business demand forecasts: Demand forecasts, which form the foundation of cloud resource sizing, are often error-prone and unreliable.
- Lack of cost attribution: Without processes in place to allocate cloud costs, DevOps teams may overlook the financial impact of their resource usage.
- Overlooking software efficiency: Developers often assume that cloud resources are infinite, leading to software designs that do not prioritise cost efficiency.
- Misalignment of forecast costs and usage: Check that the cloud pricing model is being applied as expected.
Step #4 Infrastructure-as-Code (IaC) templates can proliferate high costs
When done correctly, Infrastructure-as-Code (IaC) can provide significant advantages. It ensures environments are provisioned efficiently, quickly, and consistently, while meeting business objectives.
However, from a cost perspective, IaC templates can introduce risks. Poorly designed templates used across multiple environments can lead to inefficient provisioning, inflating costs and undermining any carefully planned cost models.
If you are responsible for the cloud budget, it is essential to have input into how cloud resources are deployed. Investing time in reviewing IaC practices will help you prevent costly issues down the line.
Step #5 Map business requirements to cloud resources and costs
Understanding and aligning business service requirements are crucial, as they directly influence the necessary cloud resource capacity — both for current needs and future growth. These requirements may include data retention policies, service-level agreements (SLAs) for response times, disaster recovery strategies, and more.
Each business requirement should be meticulously mapped to the corresponding cloud resources. At this stage, optimising the provisioning strategy is essential, as it significantly affects cloud cost structures and overall budget forecasts. For instance, a single resource like compute can be provisioned in numerous ways, each with a different cost profile. Assumptions regarding business needs and cloud resource utilisation should be carefully documented and validated through a structured process to improve the accuracy of cost forecasting.
Moreover, projected demand growth and other variables that impact demand patterns must be factored into this analysis. As demand increases, so will the required cloud capacity and associated costs, making it essential to account for these factors early in the planning process.
Conclusion
Accurately forecasting your cloud costs can bring significant benefits to your business – especially in terms of getting the most out of your cloud spend and looking at your longer-term cost predictability. While it may be a challenge for your business, it is not impossible as the above 5 steps set out.
If you’d like to discuss these steps or any challenges you are having with your own cloud cost forecasting, get in touch for a chat. You can reach me direct at frankwarren@capacitas.co.uk
