The Amazon S3 Outage - When the Internet's Hard Drive Fails …
March 06, 2017

Denis Goodwin
SmartBear

Share this

Last week, SmartBear observed a sudden and protracted 5X increase in web page timeout errors associated with the failure of Amazon's S3 cloud-based storage service. Looking a bit more closely at our data, we dug up a few more interesting angles on the impact of the failure.

Event Timeline

The issue hit suddenly – we saw an immediate spike in errors at 12:35 p.m. EST, and by 12:45 p.m. EST, error rates were 5X normal. For some specific types of timeout errors, the spike was more than 10X normal. At 3:30 p.m. EST, the error rate began dropping and by 3:50 p.m. EST, rates had returned to normal.


Web vs. API

The issue hit web pages hard, while API monitors were not noticeably impacted by this outage. Web pages and web apps often utilize content storage hosted by cloud services such as Amazon S3.

Common failure scenarios on Tuesday included page elements failing to load, which could cause either the whole web page to time out or specific content on a page might not render. Depending on the design of a given page, this partial content failure could be relatively minor or it could render a critical web journey non-functional. File uploads and downloads that rely on S3 storage endpoints were particularly hard hit.

In order to get a complete picture of application health, it's necessary to monitor your real user's journey through the application. The monitored user journeys that depended heavily on content hosted in S3 failed. Those that didn't have that dependency continued functioning. I personally experienced this with Slack – I was able to use the app, however files could not be uploaded presumably because these files are stored by Slack using S3 as the storage mechanism.

While far less pronounced than the spike in errors, some response time degradation was observed in API monitors that continued running successfully. Given that the issue affected Amazon's storage services rather than their hosting services for applications, this makes sense.

Geographic Impact

The issue was more acutely felt in the United States, but we observed impacts all over the globe. The spike in page errors was seen on websites dependent on Amazon S3, many of which are U.S.-hosted websites that are likely monitored from U.S. locations. Unsurprisingly, error counts spiked by as much as 25X in some U.S. monitoring locations. While not as significant as the U.S. locations, timeout and page error increases were also observed from Canada, Europe and Asia.

Takeaways

Much of the web is built on the backs of cloud providers. Most of the time, these cloud services provide a great user experience. Amazon will learn from the root cause of this issue and likely emerge from this outage more resilient than ever. It's impossible to control all aspects of these shared services – but here are three steps to take that are in your control.

1. Identify your business critical applications

2. Proactively monitor user journeys on these applications

3. Don't rely on your third party provider to tell you when it is down

It is key to utilize independent monitoring services to ensure your applications are up, functioning correctly and fast. Furthermore, missing content can be catastrophic or merely inconvenient to a critical user journey – it's important that your monitoring tool can be configured to know the difference.

Denis Goodwin is Director of Product Management, AlertSite, SmartBear Software.

Share this

The Latest

September 19, 2017

IT professionals tend to go above and beyond the scope of their core responsibilities as the changing business landscape demands more of their attention, both inside and outside of the office, according to the Little-Known Facts survey conducted by SolarWinds in honor of IT Professionals Day ...

September 18, 2017

Digital video consumption is viral and, according to a new study released by IBM and International Broadcasting Convention (IBC), more than half of the 21,000 consumers surveyed are using mobiles every day to watch streaming videos, and that number is expected to grow 45 percent in the next three years ...

September 15, 2017

No technology that touches more than one IT stakeholder, no matter how good and how transformative, can deliver its potential without attention to leadership, process considerations and dialog. In this blog, I'd like to share effective strategies for AIA adoption ...

September 14, 2017

Enterprise IT environments are becoming more heterogeneous and complex, with fragmentation permeating cloud infrastructure, tooling and culture, according to a survey recently conducted by IOD Cloud Technologies Research in partnership with Cloudify ...

September 12, 2017

One area that enables enterprises to reduce complexity and streamline operations is their virtual desktop infrastructure (VDI). Virtualization is a linchpin of digital transformation and effectively optimizing an enterprise's VDI is essential to moving forward with digital technologies. Delivering the best possible VDI performance means taking a fresh look at what "desktop" means today. The endpoint, or desktop, now can be a physical thin client, a software-defined thin client, a traditional laptop, a phone or tablet. To reduce operational waste and achieve better performance across the desktop environment, consider these five actions ...

September 11, 2017

In incident management, we often overlook the simple things in favor of trying to do too much, too soon. Why not make sure we've done the fundamentals properly? ...

September 08, 2017
For our Advanced IT Analytics (AIA) Buyer's Guide, we interviewed more than 20 deployments to help us better assess vendor strengths and limitations. So given the abundance of riches to work with, I've decided to illustrate several of the more prominent AIA benefit categories with actual real-world comments ...
September 07, 2017

The Input/Output Operations per Second (I/O) capabilities of modern computer systems are truly a modern wonder. Yet no matter how powerful the processors, no matter how many cores, how perfectly formed the bus architecture, or how many flash modules are added, somehow it never seems to be enough ...

September 06, 2017

By taking advantage of performance monitoring, IT and business decision makers can gain better visibility into their cloud and application performance. Dedicated performance monitoring has become essential for providing visibility into all areas of application performance and keeping the business running optimally ...

September 01, 2017

There may be no more critical emerging technology for IT organizations in the digital age than advanced IT analytics (AIA) — most commonly called “operational analytics.” It is with this in mind that EMA is launching what we believe is the first ever buyer's guide for AIA adoption ...