Organisations often use a Service Integrator (SI) to deliver major projects. The benefits of a SI is that the skills and resources which are not available within the organization can be brought in for the project duration. However, the final deliverable in terms of quality often doesn’t meet the customer’s expectations. A report by Mckinsey on why large IT projects fail recommends that “The project office should establish a few strong stage gates to ensure high-quality end products” and in addition “Top-performing projects, on the other hand, establish a clear view of the initiative’s strategic value—one that goes beyond the technical content”. Some of the key measurements of quality of the final solution are its performance, capacity, efficiency, resilience and stability. This article lists five questions you should ask your SI to ensure that the SI has these strong quality gates embedded in the project. Don’t sign on the dotted line until you are satisfied with the answers to these questions. The last thing you want is to find out that there are serious quality issues late in the project life cycle which will result in project slippage or worse project failure.
There are 5 questions you want to ask your SI (and yourself), before the project starts, to ensure quality gates exist. 
1. During the requirements stage how will you ensure the NFRs are aligned to the business objectives at the appropriate level of detail?
NFRs are often poorly defined and not fully understood by the customer. The NFRs in many cases are defined by the SI. The customer whose experience of defining NFRs may be limited at best is often asked to sign off numbers that they don’t fully understand. These numbers are contractually binding. In many cases the NFRS have too much detail in wrong areas and not enough detail in the right areas, leading to them not being aligned to the project’s strategic goal. The best way to avoid this situation is to get some independent expertise to ensure that the NFRS are aligned to the business objectives as early as possible. See Capacitas’s article on top 5 NFR mistakes.
2. At the design stage how will you identify the risks in the solution before it’s built, and will there be a plan to address these risks?
Systems often have quality issues after they are built. However, identifying and resolving defects after the system is built is more expensive and time consuming than resolving them during the design. This increases the risk of project slippage or, worse still, project failure. Research suggests that 34% of defects are introduced during the design phase. Using appropriate skills and processes during the design stage these defects can be identified and the appropriate risk mitigation steps taken. This might involve optimizing the design to remove the defect and/or putting in place appropriate risk mitigation steps to manage the defect, e.g. having specific tests to understand its criticality. At Capacitas, we use risk modelling to identify defects and risks during the design stage and deliver recommendations on design optimisations. A detailed risk mitigation plan is provided and is fed into the plan.
3. While you are building the solution how will you continuously measure the quality of the solution?
There are many activities that can be carried out during the early build stages to identify risks. At Capacitas performance engineering is carried out throughout the project lifecycle. Activities include code profiling, unit tests etc. However, the challenge is understanding the data that is produced by these activities. Without a defined analysis process to identify risks then the data is open to misinterpretation. This could lead to unnecessary activities being carried out resulting in project slippage. Worse still critical defects might be missed leading to project slippage later in its lifecycle. At Capacitas we employ our unique analysis methodology and our own internal benchmarks to triage these defects appropriately.
4. When the system is built and integrated, how will this quality be measured and how will you know whether results are good or bad?
The NFRs should be based on measurements made in live (although in many cases we find that they are based on test system measurements). However, for a new system you don’t have the ability make measurements in live. Measurements made in test systems are very different to those made in live. Your SI needs to have a process in place to bridge the gap between what you observe in test and what you expect to happen in live. At Capacitas we use a unique analysis methodology, combined with modelling and our own internal benchmarks to ensure that the solution delivered will meet the NFRs in live - not just in the test environment.
5. When the system is live, how will you measure the quality of the solution and will there be a warranty period during the next planned peak period?
The system is live. However, what is the process of measuring the system against the NFRs and when do you decide that the system has been successfully delivered? Typically, projects are rolled out during periods of low demand. The system is usually not tested in anger until many months later by which time the original project team (& knowledge of the system) have long since disappeared over the horizon, leaving you responsible for resolving the production incident. Here at Capacitas, we understand that capacity and performance is for life not just a point-in-time in a test environment. In recognition of that Capacitas carries out our unique post go-live validation service that is part of the software performance engineering service. This service is in addition to the demand & supply modelling and operational analytics services which should be carried out on an on-going basis for your services when they are operational.
