Just in time for the holiday shopping season, APMdigest asked experts from across the industry – including consultants, analysts, organizations, users and the leading vendors – for their opinions on the best way to measure eCommerce performance, in terms of applications, networks and infrastructure. Part 3, the final installment, covers the customer journey.
The best way an IT team can measure performance is by running synthetic load tests. For eCommerce applications, studies have shown that users expect response times on the order of 3 seconds or so. Otherwise, they may go to a competitor. First, you want to determine where the bulk of your users are located because you don't want to test as if all of your users are located near your web servers. Second, you want to run your load tests to determine that your application can respond within this response time for a single user only. If not, you need to make the necessary changes. Third, you ramp up the number of users with your load testing software to determine how many users can be supported at one time and still have your application respond within that time frame. Lastly, you want to ramp up users even more to determine at what point does your application respond time, not only goes well beyond 3 seconds, but your application stops responding due to lack of resources. At that point, you want to make the necessary changes to your application and/or increase its available resources to support your users.
Senior Consultant and Founder of RootPerformance
In the ever-growing ecommerce market, having a high conversion rate and low churn is a great thing. But, the technology behind your site can hinder those goals. Make sure you are appropriately load testing your web site and apps to ensure they are performing optimally, and customers aren't deserting your site due to a disappointing digital experience.
Implementation Engineer, Apica
TRACK EACH USER TRANSACTION
The common denominator for measuring ecommerce performance is the transaction. There is an indisputable correlation between metrics for application, network and infrastructure performance, user experience, and business outcome. Without accurate transaction data, insights into conversion rate, subscriber growth, average order size, cart abandonment rate, existing/new customer, percent mobile, gross margin and a raft of other metrics are not possible.
Technology Analyst and Founder of TechTonics Advisors
Make sure every user transaction is recorded for success and latency, if you can add $ value to it then great. Monitor the volume of the transactions along time and how much eventually went through successfully. This is the KPI the business is mostly interested in.
Solution Architect, Correlsense
TRACK THE ENTIRE CUSTOMER JOURNEY
Measuring ecommerce comes down to knowing your user journeys from the inside-out and outside-in. If you can drill down into slower transactions on your site and view associated URLs from the stack trace level, you'll pinpoint the issues that are most critical.
Implementation Engineer, Apica
MEASURE CART ACTIVITIES
If I were to choose one KPI to closely monitor during peak time for IT teams, it would be measuring service calls associated with cart activities against SLA. An ecommerce system can predominantly fail in three ways during peak times: page load time, cart activities and OMS related activities (i.e. successfully completing a purchase). Each of these components is crucial to customer experience and revenue generation. While all three are equally important, measuring cart activities is pivotal and serves as a precursor to OMS issues downstream, capturing customers interactions that are more indicative of revenue generation.
VP of Marketing, Skava, an Infosys company
MEASURE SITE PERFORMANCE ACCORDING TO USER LOCATION
As an ecommerce business, chances are your various customers around the globe are experiencing dramatically different levels of site performance (speed, availability and reachability), based on their geographic distance from your datacenter(s). Measuring ecommerce transactional performance accurately depends on taking measurements from points as close as possible to customers. This is the only way you can determine if additional steps — like using a CDN, or measuring how your CDN is performing — are needed to improve site performance in particular regions. In past years we've also seen an increase in micro-outages — shorter, locally contained outages — during periods of peak traffic. So a high frequency monitoring approach — measuring site performance from different locations, around the clock at close intervals — can be most effective in proactively detecting, identifying and fixing these issues.
CEO and Founder, Catchpoint
Measuring and improving the quality of the user experience is key to building a lasting ecommerce brand. Application owners have to account for the performance cost of new features otherwise they incur performance debt with every release and the user experience becomes intolerable over time. Some ecommerce teams have implemented a "user experience performance budget" to control the feature creep. Along with that it is critical to understand the user experience for each important customer segment. For example, on average a shopping site may be meeting the performance budget but for users building large shopping lists the performance may be unacceptable. If an application owner is only concerned with the aggregate they may be missing an opportunity to serve a highly valued customer better.
Sr. Product Manager, Riverbed
Frictionless consumption of an application is the end goal on any revenue-generating ecommerce-related application. Accordingly, one of the best methods to measure ecommerce performance is user satisfaction. Unfortunately for IT organizations, user satisfaction is quite subjective and is often surrounded by soft metrics. However, by applying the fundamental skills of monitoring with discipline, IT can quantify user satisfaction via application performance management success, which is highly correlated with end user satisfaction. First and foremost, administrators should discover and establish a baseline of the application's performance and health from key performance metrics such as availability and end-user response times. To handle application scalability, set thresholds for acceptable end-user and factor in tolerance levels for any variance in performance. You should also establish alerts for incidents and anomalies to ensure conformity to acceptable levels of performance. To handle the availability aspect of the application, establish redundancy and failover for application services to mitigate the risks of any down incidents. This will ensure the application's continuity, which means more time to generate revenue for the company.
Head Geek, SolarWinds
Only real user monitoring (RUM) can provide the proper analytics for application performance. There is a lot of variability within browser types, versions, and platforms along with the network and application server variations. Because of this, artificial probes cannot properly simulate the real user experience and deliver reliable analytics. Performance for ecommerce applications should be validated, not tested, through the measurement of real-user requests and responses to the application. Different control points in the application delivery infrastructure are positioned to request and collect metrics about the real user performance and experience. The application delivery controller (ADC) is well positioned to be a key point for monitoring and managing the user experience related to the ecommerce application performance. A management system that can collect and analyze metrics from the end-user, network infrastructure, and application server is going to provide the most realistic and valid information for application performance.
Director of Application Delivery Solutions, Radware
It's critical for IT teams to measure and analyze the user's journey using RUM beacons (vs. synthetic or just network/line monitoring) to understand engagement data like scroll/click/hover and how that correlates to their flow through a session. IT teams historically would focus on basic KPIs like time to first byte or page load time, but without the context of how those metrics impact and vary based on where they are collected in the visitor flow it's too easy to miss the bigger picture of what's really impacting the user experience and business outcomes.
Senior Director of Product and Industry Marketing, Akamai Technologies