Apple App Store Turns 11: Benchmarks Mobile Teams Should Adopt to Improve Stability and Experiences
July 17, 2019

Eric Futoran
Embrace

Share this

The 11th anniversary of the Apple App Store frames a momentous time period in how we interact with each other and the services upon which we have come to rely. Even so, we continue to have our in-app mobile experiences marred by poor performance and instability. Apple has done little to help, and other tools provide little to no visibility and benchmarks on which to prioritize our efforts outside of crashes.


11 years ago, Apple opened their App Store. While other stores existed, none was tied to what would become our primary medium of interacting with the world - the iPhone.

Previously, all of us were using browsers to interact, which had its limitations, namely being tied to a stationary or bulky computer.

With the advent of the App Store and the fact of now having a hardware device in our pockets, the opportunity for higher fidelity experiences emerged. First, games and social experiences released, similar to those on a browser. While not an innovative jump from browser-based games and social networks, people readily adopted them because these apps were what we were already using, but more importantly, now readily available to us at all times of the day. Then those experiences improved rapidly to take advantage of the capabilities of the phone itself. Games improved, social experiences improved, and the way we interacted with the world changed.

Humans are drawn to the lowest friction experiences. Conversations became text and chat, marketing became push notifications and SMS, purchasing became instantaneous, and short form video clips became streaming augmented always-on experiences. Even food ordering, hailing a cab, and checking in at work changed because it was just so much easier to conduct on a phone and “seemingly” a better experience.

The behavioral change of where we spent our time happened hyper fast, but the tools available to developers to build those streamlined experiences did not. Initially, the focus was on acquiring users and developing features. Stability was an afterthought. Fittingly, Apple released a single metric, crashes, and only limited capabilities to solve these crashes. Even today most developers don't know that Apple provides a crash reporting tool, and to be honest, those that do, dismiss it as unusable.

Crashes became the de facto and only benchmark of app quality. If an app has 99% crash-free sessions, it's relatively good. Apple did push another benchmark, ratings, but user ratings are very easily gamed through prompts and most users with unstable or poor experiences walk without rating the app.

We all have issues with apps. It's not uncommon to find someone swearing at their phone or attempting to restart it to solve a problem. (Reminds me a lot of conducting a silent prayer and restarting an old desktop to solve a problem.) To help developers adapt and improve all of our experiences, below are some of the new benchmarks that apps should follow to actually provide a stable experience. In fact, Apple has been rumored to use or will soon use many of these benchmarks to determine app rankings in the store, while Google has already started to do so. Please note that only monitoring tools track these benchmarks and less than 1% of apps use a dedicated monitoring tool — although adoption is growing rapidly.


How to Read these Benchmarks: These benchmarks indicate the percentage of clean sessions for each app category by error type. The overall app health is based on a combination of all these metrics, not included here for simplicity. For example, Force Terminations are defined as when a user leaves an app and immediately terminates the app from memory while the app is still running. CPU Pegging is when the processor is fully maxed out and the app is unable to run.

Key Findings:

■ Crash session scores across all verticals are over 99%, which is expected since it has been the primary focus of developers to date.

■ Social Media apps have the worst Force Termination score, which may indicate a larger issue that is going unreported, such as a broken registration or frozen startup.

■ Shopping apps also resulted in a poor Force Termination score, which could be attributed to users frustrated with a feature not working in the app, like a purchase flow failing or a frozen item page.

■ Entertainment and Education apps perform the worst across several categories; an indication of instability, complexity of app, and immaturity of app type.

Now that we are on the 11th anniversary, it's time for us as app developers and teams to shift to the new benchmarks to provide our users with the experiences that they deserve, and Apple as a user-focused high-fidelity brand that users expect.

Methodology: Embrace created a data set from top 100 mobile iOS apps. User experiences were measured based on actual impact to a user, including freezes and abrupt closes.  The latter includes but is not limited to crashes. All data was collected for the time period between June 1 and 30, 2019.

Eric Futoran is CEO of Embrace
Share this

The Latest

January 23, 2020

EMA is about to embark on some new research entitled Data-Driven Automation: A Vision for the Modern CIO. We're trying to piece a puzzle together that so far we don't believe anyone to date has fully done — seek out where and how IT is moving toward integrated strategies for automation in context with real-world objectives and obstacles. We'll be looking at four use cases, each of will no doubt tell its own story ...

January 22, 2020

Many pitfalls await CIOs on the journey to the cloud. In fact, a majority of companies have been only partially successful, while some are outright failing. To learn more about this migration, Business Performance Innovation (BPI) Network surveyed IT and business executives and conducted in-depth interviews ...

January 21, 2020

The online retail industry has yet to have a Black Friday/Cyber Monday weekend unscathed by web performance (speed and availability) problems. Luckily, performance during 2019's hyper-critical online holiday shopping weekend was better than in years past, as we did not see any systemic, lengthy outages. While no website went completely down, several retailers did experience significant problems. Why have online retailers yet to figure out how to be crash-free during this all-important peak traffic period? We've identified several reasons for this ...

January 16, 2020

Gartner highlighted the trends that infrastructure and operations (I&O) leaders must start preparing for to support digital infrastructure in 2020 ...

January 15, 2020

Edge computing usage is starting to increase. The obvious follow-up question is, "So, what can I do with edge computing?" I'm glad you asked. There are lots of things you can do ...