Perfecto Mobile released the Mobile Test Coverage Index, a data-driven benchmark to help organizations determine the appropriate mix of mobile devices, operating systems and form factors to test their apps and responsive websites and ensure optimal app quality for end users.

Today’s mobile market is extremely fragmented and constantly changing, challenging organizations with device mix obstacles, future releases, and the pressure to deliver flawless customer engagement. Currently, most enterprises have no standard approach for testing mobile apps and responsive websites, and the business impact that non-tested or poorly tested apps have is devastating to a brand. To help begin this testing process to ensure continuous quality and develop a comprehensive mobile strategy, enterprises need to define a set of mobile devices to test apps and responsive websites against, a strategy that best represents their target market and end users. This should be part of a larger definition of the user profile, such as the environment that the app is operating in (network conditions, GPS location, etc.). To answer the vital question “what devices should I begin testing on,” the industry benchmark report provides crucial information and insight into market trends and the correct device mix to ensure coverage.

Perfecto Mobile’s Quarterly Mobile Test Coverage Index aggregates 360,000 hours of exclusive Perfecto Mobile customer usage data, market data, and future device release timelines to create an extensive list of the 32 most relevant devices. It serves as a recommended starting point for organizations beginning the test coverage process.

“The first question both business and technical executives alike ask when building a mobile testing strategy relates to coverage. A relevant device and environment mix offers strong test coverage, getting the organization much closer to the true end user experience,” said Roi Carmel, SVP of Products and Strategy, Perfecto Mobile. “It is challenging to achieve desired test coverage for mobile end-user profiles in today’s rapidly expanding ecosystem, especially since a mobile device’s shelf-life is around 9 months. Our goal with the quarterly Mobile Test Coverage Index is simple: Help enterprises better understand their customers and automate app testing to deliver a high-quality digital experience.”

The report provides an index for reaching 30 percent, 50 percent and 80 percent test coverage in North America, France, Germany, Italy, Spain and the UK. The device selections address five key sectors: device model, screen size, GA date, hardware system and recommended operating systems. The number of devices in each test bracket may vary from quarter to quarter to reflect the changes in the mobile market. The report will be continuously updated to meet these market changes.

Organizations can use the index as a benchmark to take advantage of the three levels of mobile market coverage and scale as their needs grow.

Key takeaways from the research include:

- 30 percent of the market can be covered with 10-16 devices, based on market adoption, market device leaders, reference devices and device characteristics.

- 50 percent market coverage can be covered with 17-25 devices. This level of test coverage yields an increased number of devices from manufacturers such as Sony, LG, Samsung, Apple, HTC and more.

- 80 percent market coverage can be achieved with an index of 26-32 devices. This is the final group of devices to test on that will guarantee the most recommended coverage.

- While Apple pushes its users to upgrade to the latest iOS version, there are earlier iOS 8 versions that should be included in test plans. Additionally, iOS 7 still accounts for 17 percent of total users, and should be included at the same level of iOS 8 test plans.

- Including the most widely adopted Android versions KitKat and Jelly Bean is important to an Android test plan, Lollipop proves to be very important as well. Lollipop, the latest OS from Google, is running on the Google Nexus devices (considered reference devices).

This report analyzed and aggregated a wide range of public, private and proprietary data from 4,000+ device profiles and 360,000+ user hours across U.S. and EU markets, which includes Spain, UK, France, Italy and Germany.