The world is awash with application performance monitoring products. From log aggregators and alerting systems to metric wranglers and dashboard tools. Got lots of data? No problem. Just plug in the latest gizmo to better understand the health and performance of software applications. Got silo'd teams? No problem. Apply some analytics wizardry and watch folks magically start working together – a veritable DevOps love fest, right?
To be considered best-in-class, monitoring systems should naturally be part of the team doing awesome stuff to bring out the awesomeness in others. With that in mind, how do you select the best monitoring team players? It's pretty simple, approach it the same way you would go about hiring people – screening and interviewing them.
So, imagine for a moment that application performance monitoring products are job candidates. What strategies and questions would you ask to sort the wheat from the chaff? Here are five good ones:
1. What do you consider to be your major strengths?
Be on the lookout for excessive claims about monitoring superpowers. If a candidate claims they can detect every anomaly or pattern across complex distributed systems, then consider striking them from the list. On the other hand, if and by way of example, the interviewee exhibits honesty by describing the data management and dimensionality challenges involved with correlating events from code to container then it's an indication of domain expertise – a rare but valuable characteristic.
2. Describe a time you worked as part of a team
Again a great question to sort out team players from the glory seekers. Strike a red line through the name if the candidate claims they've accurately pinpointed the root-cause of every problem, but only have partial insight across the application technology stack. s. But, it's onward to interview stage 2 if the product has analytical smarts which actually assist teams of engineers (not loners) become better at what they do. That doesn't mean doing the work for them, it means creating the circumstances that help them become even more valuable - like collecting problem evidence and sharing observations to fix and improve.
3. Describe your engineering background
Candidates waxing on about how they've evolved from mega years of lab development plus all the qualifications (patents) they've received is all fine and dandy, but not nearly enough. Look for products that have experience of working in complex environments – from microservices to big-iron; designed by folks with similar street credibility and enterprise toughness. Give special credit to products that have evolved as a result of observing real engineers – not specialists, but those working in real-world situations, trying to solve white-knuckle problems and make sense out of crazy-weird anomalies – each and every day.
4. Why did you apply for the position?
A simple question which stumps many candidates. If the answer is to troubleshoot more complex problems and issues, then prepare to hit the fail buzzer. But if the response is more considered – such as, aligning operational pain to business and customer pain, or giving on-call teams the confidence in themselves not to go into meltdown mode over every alert, then it's time start prepping the "your hired" paperwork. But seriously, any half decent monitoring product should help kill alert fatigue not create more noise.
5. Where do you see yourself in five years?
Ok, that question makes me wince because I can't figure out what I'm going to be doing tomorrow. A good monitoring candidate will be just as honest, stating that futureproofing systems is going to get even more complex. They'll explain how microservices and containers are great for fast delivery, but tough suckers to manage. They'll discuss how modern platforms empower teams to do really cool-stuff, but introduce many more unknowns. They'll be brutally open about the task ahead but not shy away from it, because, and here's the rub – they thrive in chaos, extremes and uncertainty.
In this chaotic world, organizations should always seek out the toughest tools that help the toughest teams sort out all the really tough stuff. And yes, finding the best ones requires asking some tough questions.
And when in doubt, a free trial is always another great way to find the tool that's the best fit.