What Exactly Should We Measure?

From my other posts it should be clear that I do not think that the cover the waterfront approach used by frameworks is workable, nor do I think that monitoring every element of the application system gets us to an ability to accurately measure end user experience and assure, on an on-going basis, a good user experience. So, what should we measure, and how?

The answer starts with the following assertions:

1. Every action taken by every user of the applications system is important
2. Some actions taken by some users are more important than others

In order to approach the problem of end user experience measurement from the above two perspectives, we must be able to measure the following:

1. Who is the user? This means understanding the importance of each user relative to all other users.
2. What transaction is the user attempting to execute? This means knowing which transactions are critical, important, convenient, and irrelevant.
3. When an important user is attempting to execute a critical transaction, what is the experience of that user from the start to the end of the transaction. This includes the response time for each step, the total response time for the entire transaction, and all errors and messages that the user sees during the entire transaction.
4. When a user is executing multiple transactions from a technical perspective, but one from the users perspective, the entire interaction needs to be captured. For example, if a user is about to buy some shares of a stock (one transaction), but pauses before committing the transaction to do one more piece of research (a lookup that is technically a second transaction), both need to be looked at together to really understand the user's experience. If the lookup fails with a HTTP 404 error, and the user aborts the transaction, you will never know why unless you looked at both together.
5. Not only are individual transactions important, but their rate and performance relative to periods of time also matter. Each transaction has a normal rate for a time of day and day of week. If that rate and performance change relative to the norm, that needs to be captured and noted. Understanding that "something is different" is critical to starting the process of understanding why.
6. Understanding long duration transactions. When someone buys something from an online store, is the transaction complete when the user completes the checkout process, and the vendor charges the credit card successfully? Maybe it is from the perspective of the online e-commerce system, but from the perspective of the user, the transaction is only complete when the product is actually delivered, and maybe really only complete when the product successfully meets the need for which it was purchased.

In summary, measuring end user experience holistically, is much harder than capturing some response times here and there. As organizations start to do this, I predict that it is going to expose tremendous weaknesses in how the underlying systems are actually being managed and monitored. This will be a very bad thing for the incumbent infrastructure monitoring vendors who are charging customers a lot of money for products, that at the end of the day, are measuring the wrong thing (infrastructure availability). And, it will be a wonderful thing for new vendors more in tune to the emerging needs of customers who embrace true measurement of end user experience.

Bernd Harzog
CEO
End-User-Experience.com