Don't Be the Next Instapaper
February 21, 2017

Michelle McLean
ScaleArc

Share this

Instapaper, a "read later" tool for saving web pages to read on other devices or offline, suffered an extensive outage 2 weeks ago. The site was unavailable for a day and a half, and even after restoring service, the company had to explain that its archives would be impacted for another full week. Ultimately, it was able to restore the archives sooner, but the outage garnered extensive press and social media coverage.

The cause of the outage was that an indexing file Instapaper relies on for reaching all stored links exceeded the max file size supported on the older instance of Amazon Web Services the site was first built on. You can read if you want more details .

While Instapaper hit a unique problem — a file size limitation — its experience speaks to a much larger problem: scaling a database is difficult, and never quick. That basic fact explains why outages like the one Instapaper suffered are surprisingly common.

Engineering a scaled database — and then performing the application changes needed to take advantage of that scaled out database — is tough coding work indeed. We encounter companies with full control of their source code who are petrified to make the changes needed to scale database capacity. Perhaps it's an ecommerce app, and it's too close to Black Friday. Or maybe it's just a case of attrition: the folks who really understand that code base are long gone, and the current engineers don't dare mess with the interworkings of the app.

These kinds of meltdowns are common during surge events, like the one ESPN suffered with the launch of Fantasy Football or the one Macy's suffered last Black Friday. Sometimes customers can see these events coming (e.g., they're expecting a major traffic surge on Black Friday) and sometimes they simply don't (e.g., their product gets a nod from a celebrity and all of a sudden they're swamped).

When a traffic surge takes down your site, it usually means the data tier was already fragile. Scaling the web infrastructure is pretty easy, as is scaling internet capacity. But scaling the data tier itself is where the challenges lie.

The Instapaper crisis also illustrates how the cloud alone doesn't solve the challenge of scaling the data tier. While elasticity is a hallmark of cloud services, the physics around having an application talk to multiple instances of a database remains a challenge. We've seen some customers suffer from an inflated sense of confidence that running in the cloud takes away these difficulties.

Don't wait for disaster to strike. Whether you're running on prem or in the cloud, keep a close eye on all metrics that reveal how "hot" your systems are running. Ensure your disaster recovery plan is robust — and recently tested. Better yet, don't rely on disaster recovery. Instead, run in active/active mode, where you've got multiple instances of all critical systems running in different locales, with the systems able to take on the full load if one portion fails.

Take steps now to scale your data tier and avoid these kinds of catastrophic outages. Those "Here's why we failed" engineering blog entries are no fun to write.

Michelle McLean is VP of Marketing at ScaleArc.

Share this

The Latest

June 15, 2018

When it comes to their own companies, 50% of IT stakeholders think they are leaders and will disrupt, while 50% feel they are behind and will be disrupted by the competition in 2018, according to a new survey of IT stakeholders from Alfresco Software and Dimensional Research. The report, Digital Disruption: Disrupt or Be Disrupted, is a wake-up call for the C-suite ...

June 14, 2018

If you are like most IT professionals, which I am sure you are, you are dealing with a lot issues. Typical issues include ...

June 13, 2018

The importance of artificial intelligence and machine learning for customer insight, product support, operational efficiency, and capacity planning are well-established, however, the benefits of monitoring data in those use cases is still evolving. Three main factors obscuring the benefits of data monitoring are the infinite volume of data, its diversity, and inconsistency ...

June 11, 2018

Imagine this: after a fantastic night's sleep, you walk into the office ready to attack the day. You sit down at your desk ready to go, and your computer starts acting up. You call the help desk, but all IT can do is create a ticket for you and transfer it to another team to help you as soon as possible ...

June 08, 2018

As many IT workers develop greater technology skills and apply them to advance their careers, many digital workers in non-IT departments believe their CIO is out of touch with their technology needs. A Gartner, Inc. survey found that less than 50 percent of workers (both IT and non-IT) believe their CIOs are aware of digital technology problems that affect them ...

June 07, 2018

CIOs of 73% of organizations say the need for speed in digital innovation is putting customer experience at risk, according to an independent global survey of 800 CIOs commissioned by Dynatrace ...

June 05, 2018

Digital Transformation requires more than just the latest technology, it's a mindset that iterative change is on the way and should be embraced. This also requires us to factor in the people and process parts of the equation and find ways to measure the end-user-experience (EUE). One way to do this is to sponsor an Application Performance Monitoring (APM) initiative that can provide visibility to the business, help communicate the progress, and highlight the impacts to the organization. ...

June 01, 2018

While DevSecOps, much like DevOps itself, is more about changing IT culture than employing certain types of technology, some tools can be an important support. To find out what the right tools are, DEVOPSdigest asked experts from across the IT industry for their opinions on the top tools to support DevSecOps ...

May 31, 2018

Availability has cemented itself as one of the most essential elements of any successful data center. However today, what many organizations are beginning to realize — sometimes the hard way — is that traditional methodologies and technologies for high availability (HA) have limits. What's needed instead is a new approach that enables the dynamic transfer of workloads in IT environments based on optimizing the particular job at hand ...

May 30, 2018

Self-driving cars, integrated toys, smart home appliances, and even critical infrastructure have all become part of the ecosystem of Internet of Things (IoT) devices, which begs the concerning question, "How will network administrators process all the data generated?" ...