In my previous blog I touched upon several areas where the value of Agile performance testing could be improved. This blog continues with the same theme, considering 6 additional ideas to improve the testing.
6. Augmentation of Test Analysis
Analysis in performance testing can be time consuming. This can lead to frustration amongst the Agile development team.
Load generation tool providers usually provide a means of capturing performance metrics from the end points, and graphing from a single analysis view. These allow an opportunity to visually correlate a broad range of different metrics alongside load testing metrics. Reports invariably can be scheduled with pre-defined reporting templates. In general, these analysis facilities are slick and the outputs can easily be shared. In short, this is a good feature.
However, there are limitations that necessitate supplementary approaches to be taken,
- Servers being monitored often require firewalls to be opened up from the load controller.
- Load tools do not provide a means of statistically testing a large numbers of metrics to establish meaningful relationships and assist with root-cause investigation.
- Load tools do not provide a way of creating derived metrics such as ‘CPU Service Time’. See Part 1 of this blog for 'Transaction costs analyses'.
- Key performance measures will need to be extracted from source data for purposes of performance and capacity modelling.
- Performance resource monitoring may incur additional licencing costs for the load generation tool
Whilst load tools are getting more sophisticated at results analysis and comparison, they remain some way off meeting requirements for a large range of uses. Load tools were never originally built as sophisticated analysis tools so this is not too much of a surprise.
Building in the automation of results analysis to extend the capability of the load tool is a challenging but valuable task. It is possible to build this analysis layer at low cost with MS Excel VBA macros to provide an efficient and effective supplementary analysis layer.
The main advantages of this approach are as follows:
- A consistent method for comparing result sets which is ideal for regression testing.
- Performance defects that may have otherwise been hidden are now visible.
- A high degree of automation is achievable.
- Transactions parametric that can be used as input in performance models.
- Ability to tap into more comprehensive ranges of analysis methods to rapidly investigate a large range of performance metrics.
7. Minimising ‘lost-time’
Testing ‘downtime’ is invariably unavoidable and frustrating. However, time-wastage during sprint can be disruptive and potentially harmful to delivery schedules. In this respect I characterise avoidable time wastage as ‘lost-time’. Examples of ‘lost-time’ include the following:
- Poor planning resulting in late dependency resolution. For example, late notice to developers to provide test stubs.
- Additional test repetitions as a consequence of invalid test set-up.
- Additional test repetitions as a consequence of external contamination of performance tests.
- Additional test repetitions as a consequence of weak performance analysis. It is sometimes seen as necessary to eliminate possible causes by for example testing different code levels and functions. A more thorough analysis of the original performance test result tests may be just as fruitful.
- Procedural failures such as code deployment, data refresh, etc.
- Poor communications between development teams and performance testers thus resulting in incorrect test and script design.
- Poor information exchange on defects resulting in misdirected investigative effort.
In reality the list of potential ‘lost time’ causes is extensive. In many cases lost-time is avoidable.
Before addressing these points, it is recommended that you understand where time is being lost. The recommendation of this blog is that before taking action you should carefully record and subsequently analyse where your ‘lost-time’ and ‘down-time’ is leaking. Another benefit of logging time is that you have a real record that can be used to influence where testing improvement initiatives should be directed.
8. Streaming environments to increase performance testing throughput.
No matter how efficient the performance tester it is unlikely that you will fully utilise a single performance environment. Indeed with multiple Sprint teams it is likely that there will be contention for the environment.
A more pragmatic albeit costly solution is to increase the performance testing environments (PTE), tools, and resources to support parallel testing streams. An increasingly popular model in Agile is to have cut-down environments linked with individual development teams. This presents performance testers an opportunity to work effectively with a small group of developers at a much earlier point in Sprint, on a targeted set of features. The model below shows 3 cut down PTE’s and 3 performance testers each allied to discrete development teams. This model works well as it shows the impact of change relative to a baseline established in the given cut-down PTE, and allows the Performance Tester to closely collaborate with the developers influencing their approach to the solution. The feedback loop with the developers is tight and efficient.
The performance testing process eventually scales up into highly scaled PTE environment (PTE “Prime”). At this stage in Sprint a number of performance testing defects have been isolated in PTE1/2/3 and resolved by development teams. The “Prime” environment should be as like-live as possible. The objective of the larger scale performance testing environment is to identify performance bottlenecks that manifest under higher load on a more realistic target architecture.
9. Application configurability
Building the ability to disable/enable application feature into software will dramatically improve your chances of being able to make progress on feature performance testing and facilitate testing from early stages of development. The simple scenario below amplifies this point.
Consider a four step test script of a web based process with feature changes impacting Steps 2 & 4 (highlighted in blue):
During the execution of the script Step 2 fails as the feature change is not stable. Steps 3 and 4 are unable to Complete. Application configuration has been set to ‘enabled’ to allow the feature to be tested.
By disabling the feature on Step 2 it is now possible to make progress in the execution to allow the feature change in Step 4 to be tested. Whilst this is not an ideal situation it does facilitate progress in testing as it allows the unique performance traits of the feature change in Step 4 to be compared relative to baseline. In short, application configurability permits greater control and flexibility in all areas of testing, and upholds a key tenet of Agile/Sprint to make progress ‘where you can’
In this simple example, once the issues related to Step 2 have been resolved the script will need to be re-tested. In practice the use of configuration can be more elaborate. For example, there may be a dependency between Step 2 and Step 4 with Feature ON, therefore it is not possible to test Step 4 in isolation.
10. Enhancing test analysis with Modelling
Performance and Capacity modelling is often perceived as a ‘dark art’ and as a consequence is often avoided. However this view undermines the value that can be realised,
- Models are complimentary to performance testing and add value to the conclusions drawn.
- Models can predict future infrastructure usage based on forecasted business usage, and the parametric data gathered from testing.
- Performance models do not provide ‘whole solution’ answers but are scoped to provide targeted predictions for particular question.
- Simple performance models yield tremendous value and are not difficult to build providing certain principles are adhered to (I.e. basic Queuing theory, operational laws, etc.)
- Unlike performance testing, performance and capacity models provide much greater flexibility for answering different questions.
- Models can be constructed and provide answers early in the project lifecycle before the onset of testing.
Consider a real situation where capacity forecasts are produced as a product of testing,
a) An existing capacity model uses as input the measured CPU Service time transaction costs for the main transactions.
b) At the end of each Sprint cycle the capacity model is rerun using the new CPU service times obtained for the release.
c) Capacity of production infrastructure is forecasted before Sprint closure.
d) If CPU usage is predicted to be excessive then efficiency issues are resolved before the changes are released.
The modelling process outlined above is quick, lightweight, and reliable. In this example, the method of capacity assessment has been operating successfully for many years.
11. Automate everything
Whilst the notion of automation is not a new idea, it is worth re-iterating. The benefit of automating standard performance testing procedures is realised by improved productivity. Listed below are some examples of where automation can help:
- Uploading and manipulating test logs
- Analysing test results
- Deploying software and database components
- Restarting services and stubs
- Clearing and cleaning the environment
- Disabling components in the testing fabric
- Disabling/Enabling software features
- Capturing errors records
…and so on (examples of scripts that perform these functions are available upon request).
If you would like to learn more about our Modelling and Performance testing solutions, please click below, to see our latest webinar.
