An infinite number of monkeys...

So you've built a complex system for a large client who has thousands of customers. The service has to be highly available, highly reliable, and at the same time needs to handle all of those users.

You've stuck to all of your favourite 'best coding practices' and it's going to be hosted on a really powerful platform... It'll probably be fine, right?

We have just launched the new and improved South Staffs Water website. The site allows customers to log in and manage their account online, submit meter readings, view their bills and securely make payments. It also provides help and information in the form of news articles, chat and an FAQ section

There were two main areas of concern around the performance of the site. First of all the marketing/information section occasionally has to cope with sudden spikes of traffic in the event of a water supply incident. It could be anything from 20 people in a street trying to find out why their water is a funny colour, to 1000 people in a village trying to find out why their water has gone off entirely.

Our other focus was on the number and complexity of requests made by logged in users in the My Account section. We had to integrate with a number of challenging legacy systems while at the same time keeping the experience snappy for the user. A lot of data was requested up front and cached to reduce the number of calls and some clever circuit breaker features were added to ensure that the system wouldn't fail entirely if one of the applications began to buckle under high load.

The tool we picked was JMeter. It's a widely used industry standard performance testing tool. It's open source and available for free. JMeter allows you to replicate the requests made by a browser and then repeat and upscale them. It also includes further benefits like adding assertions to make sure your responses are correct as well as quick, and the ability to easily pipe in test data like usernames and passwords for multiple test accounts.

Combining JMeter with flood.io allowed us to create a much higher number of distributed requests than we could ever create from our studio. Flood is a cloud based load injector that allows you to upload your JMeter scripts and they are replayed using Amazon's cloud infrastructure around the world to generate the load.

There were no great surprises in the end. Several rounds of testing helped show that all of our defensive coding had paid off and backed up our suspicions that some of those third party services might struggle to keep up under high load. The only real surprise was that everything ran better than expected, which leads me to the ultimate goal of any project - happy clients!

No amount of performance testing is a substitute for real users. So the live site is being constantly monitored and improved. You can't predict exactly how people are going to use your service. And even if you could, you couldn't replicate it exactly due to too many unknowns and external factors. What performance testing (or indeed any type of testing) does give us is more knowledge about our product. And when you increase the amount of knowledge, you increase the level of confidence.

We built the site with these factors in mind and as always learn't plenty along the way. So when the next challenge comes along we can apply these skills and techniques in order to provide the best solution for our client and their end users.