Apple App Store Turns 11: Benchmarks Mobile Teams Should Adopt to Improve Stability and Experiences
July 17, 2019

Eric Futoran
Embrace

Share this

The 11th anniversary of the Apple App Store frames a momentous time period in how we interact with each other and the services upon which we have come to rely. Even so, we continue to have our in-app mobile experiences marred by poor performance and instability. Apple has done little to help, and other tools provide little to no visibility and benchmarks on which to prioritize our efforts outside of crashes.


11 years ago, Apple opened their App Store. While other stores existed, none was tied to what would become our primary medium of interacting with the world - the iPhone.

Previously, all of us were using browsers to interact, which had its limitations, namely being tied to a stationary or bulky computer.

With the advent of the App Store and the fact of now having a hardware device in our pockets, the opportunity for higher fidelity experiences emerged. First, games and social experiences released, similar to those on a browser. While not an innovative jump from browser-based games and social networks, people readily adopted them because these apps were what we were already using, but more importantly, now readily available to us at all times of the day. Then those experiences improved rapidly to take advantage of the capabilities of the phone itself. Games improved, social experiences improved, and the way we interacted with the world changed.

Humans are drawn to the lowest friction experiences. Conversations became text and chat, marketing became push notifications and SMS, purchasing became instantaneous, and short form video clips became streaming augmented always-on experiences. Even food ordering, hailing a cab, and checking in at work changed because it was just so much easier to conduct on a phone and “seemingly” a better experience.

The behavioral change of where we spent our time happened hyper fast, but the tools available to developers to build those streamlined experiences did not. Initially, the focus was on acquiring users and developing features. Stability was an afterthought. Fittingly, Apple released a single metric, crashes, and only limited capabilities to solve these crashes. Even today most developers don't know that Apple provides a crash reporting tool, and to be honest, those that do, dismiss it as unusable.

Crashes became the de facto and only benchmark of app quality. If an app has 99% crash-free sessions, it's relatively good. Apple did push another benchmark, ratings, but user ratings are very easily gamed through prompts and most users with unstable or poor experiences walk without rating the app.

We all have issues with apps. It's not uncommon to find someone swearing at their phone or attempting to restart it to solve a problem. (Reminds me a lot of conducting a silent prayer and restarting an old desktop to solve a problem.) To help developers adapt and improve all of our experiences, below are some of the new benchmarks that apps should follow to actually provide a stable experience. In fact, Apple has been rumored to use or will soon use many of these benchmarks to determine app rankings in the store, while Google has already started to do so. Please note that only monitoring tools track these benchmarks and less than 1% of apps use a dedicated monitoring tool — although adoption is growing rapidly.


How to Read these Benchmarks: These benchmarks indicate the percentage of clean sessions for each app category by error type. The overall app health is based on a combination of all these metrics, not included here for simplicity. For example, Force Terminations are defined as when a user leaves an app and immediately terminates the app from memory while the app is still running. CPU Pegging is when the processor is fully maxed out and the app is unable to run.

Key Findings:

■ Crash session scores across all verticals are over 99%, which is expected since it has been the primary focus of developers to date.

■ Social Media apps have the worst Force Termination score, which may indicate a larger issue that is going unreported, such as a broken registration or frozen startup.

■ Shopping apps also resulted in a poor Force Termination score, which could be attributed to users frustrated with a feature not working in the app, like a purchase flow failing or a frozen item page.

■ Entertainment and Education apps perform the worst across several categories; an indication of instability, complexity of app, and immaturity of app type.

Now that we are on the 11th anniversary, it's time for us as app developers and teams to shift to the new benchmarks to provide our users with the experiences that they deserve, and Apple as a user-focused high-fidelity brand that users expect.

Methodology: Embrace created a data set from top 100 mobile iOS apps. User experiences were measured based on actual impact to a user, including freezes and abrupt closes.  The latter includes but is not limited to crashes. All data was collected for the time period between June 1 and 30, 2019.

Eric Futoran is CEO of Embrace
Share this

The Latest

August 21, 2019

For the first time, a majority of companies are putting mission critical apps in the cloud, according to the latest report by Cloud Foundry Foundation ...

August 20, 2019

The cloud continues to transform IT in every industry. But in order to migrate to the cloud, embrace these new technologies and truly evolve their business, organizations need an underlying network that can support digital transformation ...

August 19, 2019

One common infrastructure challenge arises with virtual private networks (VPNs). VPNs have long been relied upon to deliver the network connectivity and security enterprises required at a price they could afford. Organizations still routinely turn to them to provide internal and trusted third-parties with "secure" remote access to isolated networks. However, with the rise in mobile, IoT, multi- and hybrid-cloud, as well as edge computing, traditional enterprise perimeters are extending and becoming blurred ...

August 15, 2019

The configuration management database (CMDB), along with its more federated companion, the configuration management system (CMS), has been bathed in a deluge of negative opinions from all fronts — industry experts, vendors, and IT professionals. But from what recent EMA research on analytics, ITSM performance and other areas is indicating, those negative views seem to be missing out on a real undercurrent of truth — that CMDB/CMS alignments, whatever their defects, strongly skew to success in terms of overall IT progressiveness and effectiveness ...

August 14, 2019

The on-demand economy has transformed the way we move around, eat, learn, travel and connect at a massive scale. However, with disruption and big aspirations comes big, complex challenges. To take these challenges head-on, on-demand economy companies are finding new ways to deliver their services and products to an audience with ever-increasing expectations, and that's what we'll look at in this blog ...