Why Is Measuring Application Response Time via Scripts a Bad Thing?

When the web based applications emerged, enterprises realized that now that public users from far away places were going to use these applications, that existing script oriented QA tools for Win32 needed to evolve. Evolve they did, into products like Topaz from Mercury, and services from Keynote that measured the response time to a URL and then to various clicks and actions on the pages underneath that URL. The vast majority of the systems management and APM industry has now run off with this concept as the source of the end user response time data in their APM products. This has the following flaws :

• Applications change constantly, and therefore so must the scripts. It is hard to get the scripts working in the first place (in a meaningfully comprehensive way), and daunting to keep them up to date. The average Fortune 1000 enterprise has over 1000 custom developed applications. The budget and manpower simply do not exist to create and maintain the scripts for all of these ever evolving and numerous applications.
• It is very hard to script all of the (sometimes dumb) things that users do. There are numerous permutations of code paths through an application. Some are valid (intended by the designer) and some are not. You need to know when users (for whatever reason) take invalid paths through an application. It is impossible to script every possible sequence that a user might take through an application, and you want to know when a user takes in invalid one (might point to a training or design issue).
• N instances of a synthetic agent cannot simulate hundreds or thousands of users. Just like you cannot script all of the things that one user might do, you cannot simulate the interactions of your production user base as they concurrently hit the resources of the applications system with a few synthetic agents. Scripts are fine for making sure that the application works in QA, they simply underwhelm the problem of figuring when real users are having real problems.

Bernd Harzog
CEO
APMExperts.com
www.apmexperts.com
bernd.harzog@apmexperts.com