Meaningful performance tests

The performance test had to reproduce the application’s real processes, not just generate traffic. Requests for individual paths, including the critical path, had to be built from retrieved and processed data, taking dependencies between systems into account.

A static set of requests sent by a load generator did not reproduce those dependencies. The measurement also had to describe the application rather than the limits of the route test traffic took to reach it.

I designed the tests around three principles:

• Scenarios built from data — requests prepared from retrieved and processed data, taking dependencies between systems into account.
• Separating scenario preparation from load generation — a separate application prepares the data and requests, and the load generator executes the test itself.
• Measurement free of access-path limits — tests run where the result describes the application, not the route test traffic takes to reach it.

I designed and built an application that integrates with the systems needed to prepare the data, builds requests for different test scenarios and passes them to a load generator. I ran the tests where the measurement did not include the limits of the access path.

The tests revealed classes of anomalies that a static set of requests would not show: lost requests and response timeouts under a certain load, and events left unprocessed or processed incorrectly under heavy volume.

Value: a reliable picture of how the application behaves under load, including specific classes of failures under heavy request and event volume.