In-Depth
Seeing Your Application Through Your End Users’ Eyes
End- user experience management is becoming an important component for ensuring the success of Web-based applications.
by John Spirko
Does this happen at your company: e-mail and phone calls pour into the help desk, saying, "The Web expense form is down," but when the help desk group contacts the Operations group, they are told, "We have all green lights on our monitoring console. The system’s not down. It’s running fine."
This is a scenario many of us see every day. There is a disconnect between IT, business, and end users when it comes to keeping business applications running. Well-intentioned IT professionals spend inordinate amounts of time troubleshooting ill-defined problems reported by end users. Business-oriented staff and end users try their best but don’t have the technical skills to clearly describe the symptoms they are seeing at their end. It’s simply a difference of perspective.
How can IT professionals, in effect, see through the eyes of their end users? A new application management best practice is emerging, called end user or customer experience management, that promises to provide the next best thing, short of remotely hacking into a user’s visual cortex.
IT analyst firm Forrester Research defines end user experience management as tools that maximize the quality for end users accessing IT infrastructure and applications. The practice emerged as IT organizations found that monitoring solutions and other application/service management practices were not addressing an important element of a successful enterprise application—the end-user experience.
The Perspective Gap
As the importance of applications has grown within enterprises, so, too, has the number and type of management initiatives and practices. These initiatives fall under the umbrella of application or service management, and several are shown in Figure 1.
IT and business people are both major stakeholders in the success of enterprise applications. Each stakeholder has his own management practices, developed in the context of his role in providing the application service. IT initiatives are focused on ensuring that the system is operating and on providing technical support to end users, while business initiatives manage the usage of the service (who’s using it, what are they doing, etc.).
These management practices all have the common goal of improving the business service in some way, and each does well at addressing one particular aspect of the system. On the surface, this division appears to be efficient—each stakeholder is rightly focused on managing the aspects of the service that he is responsible for.
However, this has created a perspective gap. IT staff are generally not aware of or concerned with practices such as chargeback reporting or user auditing. The same goes for business staff regarding IT’s practices, such as performance and system monitoring or change management.
Worse, this perspective gap has exacerbated a large type of problem, one which neither side is able to tackle individually because it mixes business and IT roles and responsibilities. At their core, end user experience issues are issues where end users are experiencing problems with an application, yet there is no simple malfunction or problem that accounts for it. These problems usually cannot be resolved using today’s application management practices.
Does it matter? Absolutely. Whether it’s an external customer-facing application or an internal application for employees, end users are the ultimate arbiter of success and failure. External customers may vote with their feet and go elsewhere, but even "captive" employee users will work hard to avoid using an application they abhor, to the detriment of the company. In short, these problems can be key to the success or failure of business applications.
Capturing the End-User Experience
We do not need to throw out our existing practices and tools; databases, system monitoring and so on are obviously necessary and valuable. On the business side, practices such as Web analytics and chargeback reporting fill important needs. We really only need to concentrate on bridging the gap, finding ways to gather information on—and manage what is happening to—end users.
At a high level, we need to capture and log what users are doing, their interactions, and responses from the application. We need to capture this in an automated fashion, and be able to review and trace any particular session. Only then will operations staff be freed from interviewing users to determine what they were doing based on vague descriptions.
There are two approaches to take this. You can either watch end users (and there are several ways to do this) or you can run simulated end-user activities and capture how the application handles each. The tradeoffs include where you want to (or can) capture user interaction, and when you want to manage this process.
Passive Solutions
Passive monitoring in this context means watching all users as they enter and exit the application. It is one of the most common methods of gathering end-user activity data. There are three ways to collect data: via network, desktop agents, or server agents.
Network Passive picks up packets from the network and groups them into useful categories. The advantage is that there’s no overhead on either end; no agent is installed. However, this approach takes up a port on the network and only supports specific protocols.
Desktop Passive requires an agent on all user desktops, which sends metrics back to a management server for analysis. Among its advantages: it handles all types of client-server communications and has low server overhead. However, it is only practical for internal applications, and, of course, there’s the need to install an agent on all desktops.
Server Passive requires an agent on the Web, plus application and/or database tiers, that work together to collect and send to a management server. Its advantages are that it covers all server statistics, and specialized agents can track any type of application. However, it imposes some degree of overhead on the server, requires management of server agents, and provides less perspective on end-user activity.
Active Solutions
With Active solutions are simulations that use synthetic transactions. This approach is used to measure user experience and availability by emulating end users. There are two techniques used for this approach.
With active service, an outside vendor is contracted to send or playback transactions against the application. The vendor then periodically reports on service performance. Advantages include the ability to playback from multiple locations around the world, and the fact that no special software is needed. However, only certain types of transactions are available and commit transactions are discouraged.
In the active software technique, you implement a synthetic transaction solution and have control of both the recording and playback of the synthetic transactions. Its advantages include a more controlled environment (the same transactions make for easier comparison) and the ability to test the application in a variety of ways and loads before putting it into production. However, it works best for internal applications and requires that you maintain the scripts as the application changes.
Real World Applications
We can illustrate how these theoretical approaches can help by applying them to some example scenarios.
A large U.S.-based company needed to address performance issues that surfaced during new application rollouts and new marketing campaigns. Problems continued to surface despite pre-production testing and tuning and were not being caught early in production. The IT department was tasked with improving this situation because it was hurting the effectiveness and success of the applications.
The solution in this case called for using passive end-user monitoring technology over the network. Passive monitors were able to pick up metrics on actual end-user transactions and group them intelligently for analysis, including the critical end-to-end times. This solution made sense because the source of the slowdowns and problems did not seem isolated to one particular component or tier.
Another example: a mid-sized international company had an e-Commerce portal that generated 90% of its revenue. Several problems had them completely stumped. The marketing team could only suspect that they were losing business and were not sure how to get the visibility they needed to fix it. They tried several solutions over three months, including instrumenting Web pages and analyzing Web logs.
The solution in this case involved combining their Web analytics tool with an active user experience management solution. The analytics tool narrowed down the issues to a type of session that exhibited the problem. Active end user transaction synthesis was then used to play back different amounts of user interaction and trace precisely through the transaction, identifying the source of the issues.
Conclusion
End-user experience management is becoming an important component for ensuring the success of Web-based applications. It augments established practices on both the IT and business sides of the house, including application performance management, system monitoring, Web analytics, charge-back reporting and change management. It bridges the gap between business and IT initiatives, providing the first holistic cross-discipline view of an application, from that all-important perspective, that of the end user.
- - -
John Spirko is a product manager at Quest Software. He has almost a decade of industry experience as a software engineer, developer, DBA and system consultant, and speaks regularly on end-user management topics. You can contact him at john.spirko@quest.com.