Non-functional requirements define the quality of a solution that is to be delivered by either an internal delivery team or a third party. This is crucial to ensure that an organisation gets a product that is of a known quality as well as the required functionality. These quality measures include Availability, Capacity, Performance, Stability and more.
The key to successful non-functional requirements is that they should be measurable, and the measurement process must be clearly defined.
However, there are many pitfalls when defining Capacity & Performance NFRs in particular. Typically these requirements are measured in terms of demand, response times for online transaction processing or job duration times for batch jobs. A mistake in these will typically expose the customer to an enormous amount of risk in terms of successful project delivery and/or cost.
1) The NFR on forecasted demand to the system is wrong.
This could be because the demand volume and mix has changed significantly since the NFRs were agreed. Or perhaps the process of converting the demand to technical transactions that use the system was incorrect, or demand was reduced to keep project within budget.
2) The NFRs uses the wrong measurement criteria for performance.
It is normal when measuring response times that outliers are removed. One approach is to use percentiles to remove these outliers. However, the incorrect use of percentile response times can end up removing not just outliers but genuine data and therefore missing significant user experience issues. In one of Capacitas’s engagements a customer’s inappropriate use of percentiles had left out a large percentage of genuine user response times rather than just the outliers.
3) The NFR definitions are not comprehensive
A very limited definition of an NFR for response times has been met. However, there are other criteria that also define the performance of a service and hence its quality. These include the stability and consistency of the service. These need to be all included when defining the NFRs for response times.
4) The NFRs can’t be measured
The NFRs ought to be built in at the design stage – Service Level Agreements and NFRs are not enforceable if they cannot be tracked and measured. In one customer engagement we found the measurement of response times as defined in the documentation was not actually possible given the chosen solution.
5) There are no limitations on how much infrastructure is required to meet the non-functional requirements.
This is equivalent to handing over a blank cheque to either your internal delivery team or third party. In one engagement we found an internal delivery team had met the NFR of building a system to a response time of under 1 second – however the initial design required 2000 servers to support the solution. Luckily, because the team were internal, they were able to be flexible, and some relatively simple tuning of the code resulted in the hardware requirements dropping by 90%.
There are many more mistakes that can be made when defining capacity and performance NFRs. How can you prevent these mistakes from being made and ensure that your IT project delivers a quality deliverable without unexpected costs? Capacitas can assist you in ensuring that you have comprehensive capacity and performance NFRs to reduce the risk of any misunderstanding.
We have employed our Risk Modelling service to help define or review NFRs for many large programmes of work.
