Universal Monitoring Crimes and What to Do About Them - Part 2
May 23, 2018

Leon Adato
SolarWinds

Share this

To help your organization increase data center efficiency and get the most benefit out of your monitoring solutions, here are the remaining universal monitoring crimes and what you can do about them:

Start with Universal Monitoring Crimes and What to Do About Them - Part 1

4. Flapping or sawtoothing alerts

When an alert repeatedly triggers (a device that keeps rebooting itself or processes keep deleting/creating temporary page files so that one moment it's over threshold, the next it's below, for example), that condition is known as flapping or sawtoothing.

What to do about it: These types of alerts have several possible resolutions based on what is supported by your monitoring solution and which best fits the specific situation:

■ GOOD: Suppress events within a window. Ignoring duplicated events within a certain period of time is often all you need to avoid meaningless duplicates.

■ ALSO GOOD: As mentioned previously, add a time delay to allow for self-resolution, avoid false-positives, and eliminate other potential issues that don't necessarily require a remediation response.

■ BETTER: Leverage "Reset" logic. Wait for a reset event before triggering a new alert of the same kind. Avoid making the reset logic merely the reverse of the trigger (if the alert is > 90%, the reset might be 90% for 15 minutes, but it won't reset until it's

■ BEST: Two-way communication with a ticket or alert management system. This is where the monitoring system communicates with the ticket and/or alert tracking system, so you can never cut the same alert for the same device until a human has actively corrected the original problem and closed the ticket.

5. No lab, test, or QA environments for your monitoring system

If your monitoring system is watching and alerting on mission-critical systems within the enterprise, then it is mission critical itself. But despite the fact that many organizations set up a proof-of-concept environment when evaluating monitoring solutions, once the production system is selected and rolled out, they fail to have any type of lab, test, or QA system that is maintained on an ongoing basis to help ensure the system is maintained.

What to do about it: Duh. Implement test, dev, and/or QA installations that serve to ensure your monitoring system has the oversight necessary for a mission-critical application.

■ TEST: An (often temporary) environment where patches and upgrades can be tested before attempting them in production.

■ DEV: An environment that mirrors production in terms of software, but where monitors for new equipment, applications, reports, or alerts can be set up and tested before rolling those solutions to production. And as mentioned earlier, this is the perfect place to also monitor your production monitoring environment.

■ QA: An environment that mirrors the previous version of production, so that if issues are found in production, they can be double-checked to confirm whether the problem was introduced in the last revision.

Note that I'm not implying you necessarily must have all three, but it's worth considering the value of at least one. Because "none" is a really bad choice.

Final thoughts

The rate of technical change in the data center today is rapidly accelerating and traditional data center systems have undergone considerable evolution in a very short period of time. As complexity continues to grow alongside the expectation that an organization's IT department should become ever-more "agile" and continue to deliver a quality end-user experience 24/7 (meaning no glitches, outages, application performance problems, etc.), it's important that IT professionals give monitoring the priority it deserves as a foundational IT discipline.

By understanding and addressing these top universal monitoring crimes, you can ensure your organization receives the benefit of sophisticated, tuned monitoring systems while also enabling a more proactive data center strategy now and in the future.

Leon Adato is a Head Geek at SolarWinds
Share this

The Latest

March 04, 2021

User experience is a big deal. For public-facing interfaces, the friction of a bad customer experience can send potential business to your competitors. For IT services delivered within your organization, bad UX is one of the main drivers of shadow IT ...

March 03, 2021

When we talk about accelerated digital transformation, a lot of it is embodied in the move to cloud computing. However, the "journey to cloud" will not be uniform across organizations and industries, says Sendur Sellakumar, Splunk's CPO and SVP of Cloud. The uncertainty of the pandemic means that in 2020, many organizations tried to rein in spending to get some last value out of existing infrastructure investments. Yet some things you can't skimp on ...

March 02, 2021

The Model T automobile was introduced in 1908 ... Within a few years, competitors arrived on the scene including relic names such as Overland, Maxwell, and names that survived like Buick and Dodge. So, what does this have to do with the hybrid cloud market? From a business perspective — a lot ...

March 01, 2021

DevOps Institute announced the launch of the 2021 SRE Survey in collaboration with Catchpoint and VMware Tanzu. The survey will result in a more in-depth understanding of how SRE teams are organized, how they are measured, and a deep dive into specific automation needs within SRE teams ...

February 25, 2021

Organizations use data to fuel their operations, make smart business decisions, improve customer relationships, and much more. Because so much value can be extracted from data its influence is generally positive, but it can also be detrimental to a business experiencing a serious disruption such as a cyberattack, insider threat, or storage platform-specific hack or bug ...

February 24, 2021

Previously siloed IT teams and technologies are converging as enterprises accelerate their modernization efforts in reaction to COVID-19, according to a study by LogicMonitor ...

February 23, 2021

You surf the internet, don't you? While all of us are at home due to Covid lock-down and accepting a new reality, the majority of the work is happening online. IT managers are looking for tools that can track the user digital experience. Executives are reading a report from Gartner or Forrester about some of the best networking monitoring solutions available on the market. Project managers are using Microsoft Teams online to communicate and ensure team members are meeting deliverables on time. Remote employees everywhere use OWA to check their office mails. No matter what, you can be quite sure that everyone is using their favorite browser and search engine for connecting online and accomplish tasks ...

February 22, 2021

With the right solutions, teams can move themselves out of the shadows of error resolution and into the light of innovation. Observability data, drawn from their systems and imbued with context from AI, lets teams automate the issues holding them back. Contextualized data and insights also give them the language to speak to the incremental, product-led approach and the direction to drive key innovations in customer experience improvement. Communicating value becomes a much easier proposition for DevOps practitioners — and they can take their seat at the company table as contributors to value ...

February 18, 2021

Prediction: Successful organizations will blur (or erase) the line between ITOps and DevOps. DevOps has to coexist with traditional IT operations ... So bring a little DevOps to every aspect of IT operations. You don't even have to call it DevOps ...

February 17, 2021

The use of unified communications and collaboration (UC&C) solutions has increased since the start of the pandemic, and this increased use has created challenges for IT teams, according to a survey commissioned by NETSCOUT SYSTEMS ...