The challenge of integrating performance testing into agile development methodology is not inconsiderable and often involves a steep learning curve. Performance testing pressures are amplified by constraints such as time, tools, software stability, that operate in the agile sprint development window which is usually confined. This blog suggests 5 ways to add value to agile performance testing to help you achieve your performance testing goals. A later blog on this same topic will extend the coverage.
1. A little planning goes a long way
This point is a quite obvious one but is none-the-less fundamental. If absent from your process it will undoubtedly result in failure. The whole ethos of Agile is to complete development and testing of business requirements in a very short time-frame. You are going to have to react pretty quickly to tackle the testing tasks in hand. It goes without saying that you need to be prepared!
Preparation for a sprint happens before the sprint window opens and planning session commences. This is what I term pre-planning and entails understanding the emerging feature prior to the onset of sprint. Understanding the characteristics of the feature backlog at an early point helps by allowing you to determine,
a) If you are going to performance test it,
b) How to execute the performance testing,
c) Establish dependencies (E.g. Stubs, test harnesses, data, etc.),
d) Define and agree the acceptance criteria for sign-off,
e) Establish needs for any specialist skills or tools,
Armed with this knowledge then you will be empowered to plan effectively.
Further to this, in effective development teams I see a tendency to design complex feature in advance of sprint. This process mitigates a large amount of risk in terms of code quality, and provides the performance assurance analyst an opportunity to model the impact of the solution and propose refinements to avoid performance defects.
Lastly on this subject, it is vital that you become an active participant of Sprint planning meetings. You must ensure your voice is heard and this is an opportunity to raise your requirements.
2. Risks Assessment drives testing scope
Any performance testing assignment needs to be scoped appropriately based on solid performance risk likelihood and impact criteria. This is never more important that in Agile. You could be spending a lot of time preparing test scripts and scenarios against a feature change that is not going to have a significant impact. For example, a minor content change on a statically generated web page will not usually generate massive performance headaches. On the other hand, if the feature implicates significant business logic changes or possibly architectural changes then you would be well advised to get under the covers. Also, it is much better to make your own judgement on the potential impact. Developers have a tendency to overstate ‘good’ performance as they prototype, and they build in different environments with no background load factors.
Common sense suggests that there should be well structured and understood performance risk criteria which, when applied, will narrow down the really important features to focus testing on. In my experience this is the only way to operate in agile. Formalisation of the risks assessment process is essential to ensure that the development team and the business buy-in to the process.
3. Regression testing (“Test the loved ones repeatedly”).
Most applications have a well-established pattern of usage and certain transactions will always predominate. Invariably the main user journeys (and hence code paths) do not vary from sprint to sprint. A good example of this is the ‘booking funnel’ used on an e-commerce website. It is prudent to develop a standard regression test strategy that exercises the main user journey in a consistent and representative way, and be prepared to simulate this pattern as a regression scenario. As code and database changes emerge the same regression scenario should be re-executed. This approach holds several advantages:
- This is the code path that the end user performs most so it is vital that performance is good.
- Feature change does not necessarily entail additional script development tasks and may be covered by the regression scenario.
- Changes in performance can be easily identified as development changes are introduced.
- The regression scenario can usually be executed from Sprint Day 1. Some poor performing changes can be isolated early.
- Provides baseline performance for the release thus defining an acceptable level of performance.
This topic will be discussed in more depth in a later blog.
4. ‘Transaction cost analysis’
This is a great analysis opportunity that falls out of the regression scenario and adds tremendous capability to the agile test strategy. Transaction cost is the calculation of the average computer resources (Processor, Memory, Database, Network, etc.) consumed at each component of the architecture for each transaction. The power of the analysis cannot be understated as it provides an absolute performance measurement for comparing atomic performance across releases and deployments, and naturally exposes the impact of change.
The graph below is a real-world example of a regression benchmark executing a series of ‘Single Transaction’ tests to isolate transaction costs. The results demonstrate how the analysis of CPU service times can highlight variances in transactional performance for difference code ‘Package’ deployments. In this example,
- The CPU service times of Viewbooking and STS transactions increased dramatically and were the subject of defect investigations, both of which resulted in favourable optimisations.
- The performance of the Purchase transaction was significantly improved as a consequence of refactoring work.
- The performance of the other transactions was stable.
Another additional benefit is that transaction costs can be used as input for capacity and performance modelling exercises allowing predictions of scaled performance and capacity requirements to be made.
This topic will be discussed in more depth in a later blog.
5. Underlying symptoms of poor performance
Whilst there is a tendency of performance testers to focus on the performance experienced at the load Injector (the virtual user), this does not necessarily give the complete picture. The results if favourable may lead one to believe the outcome is successful when in actual fact the opposite is true. There are circumstances that exacerbate the need for deeper analysis. For example, many architectures use asynchronous services such as messaging which may be un-performant even though the effect is not directly appreciated by the end user. In any case best practice strongly advises that performance of applications, operating systems, databases, networks should be studied and assessed as part of analysis. Whilst agile time constraints are challenging it is necessary to complete this task to avoid issues down the line. Areas that should be examined typically include,
- Effects of component queuing
- Transaction throughput and latency
- Concurrency in application and database components
- Any attritional impact such as memory leakage, cache pollution, etc.
- Locking and serialisation on resources (logical and physical)
- Unstable performance (E.g. periodic spiking activity)
And of course,
- Computer resource consumption (E.g. Processor % utilisation, Memory usage and paging, Database performance, etc.)
With many modern load generation tools it is much easier to monitor the performance of application, system, network, and database performance from the load controller. This really simplifies the analysis process as it provides a ‘single view’ of performance adding great convenience to the results capture process. Agile performance testers should be looking to leverage their load generation tools to best effect.
A method of performance triage should be adopted by your agile performance strategy. This triage process will allow you to evaluate the underlying root cause of poor performance by following a standard proven procedure.
If you would like to learn more about our Modelling and Performance testing solutions, please click below, to see our latest webinar.
This topic will be discussed in more depth in a later blog.
