Shoreline.io announced Shoreline’s open source solutions library, a collection of Op Packs that make it easier to diagnose and repair the most common infrastructure incidents in production cloud environments.
Launching with over 35 Op Packs freely available to the community, the solutions library addresses issues like JVM memory leaks, filling disks, rogue processes, and stuck Kubernetes pods, among others.
Published and provisioned as open source Terraform modules, each Op Pack contains everything necessary to solve a specific issue, including pre-defined metrics, alarms, actions, bots, scripts, and tests. With Shoreline’s Op Pack library, the community identifies what to monitor, what alarms to set, and what scripts to run to complete the repair. All Op Packs are completely configurable and allow cloud operations teams to decide whether to use full automation or an interactive Notebook for human-in-the-loop repair. Co-developed with Shoreline customers, the Op Packs available at launch are based on real world on-call experience at large enterprises, rapidly growing unicorns, and the largest hyperscalar production environments.
"We're all working in the same cloud environments, yet every company has to figure out on their own how to automate even commonplace issues, like filling disks or JVM memory leaks,” said Anurag Gupta, co-founder and CEO of Shoreline. “Companies can no longer afford to write their own runbooks or custom code automations from scratch. With Shoreline, every time someone in our community fixes a problem, everyone else benefits.”
The following Op Pack solutions are immediately available, and free to Shoreline customers. The solutions library will continue to grow each month as new Op Packs are added by the Shoreline community. With each additional Op Pack in use by a customer, time is freed up for engineers to focus on innovation, rather than repetitive, mundane tasks that are better handled through automation. Op Packs available at launch include:
■ Streamline Kubernetes Operations
- Kubernetes node retirement - Gracefully terminate nodes when marked for retirement by the cloud provider.
- Kubernetes pod out of memory (OOM) - Generate diagnostic information and restart pods that ran out of memory.
- Kubernetes pods stuck in terminating - Identify, safely drain, and restart stuck pods.
- Kubernetes pods restarting too often - Detect pod restart loops and capture diagnostics to identify the root cause.
- IP exhaustion - Clear away failed jobs or pods that are consuming too many IP addresses.
- Stuck Argo workflows - Argo makes declaratively managing workflows easy, but it can leave behind many stale pods after workflow execution that should be deleted.
■ Reduce Toil (on both VMs or Kubernetes)
- Disk resize / disk clean - Disk full incidents can lead to wide-spread outages and data loss that can damage customer experiences and lose revenue.
- Networking issues - Network related issues are often hard to diagnose, and can lead to a very bad experience for customers.
- Intermittent JVM issues - Capture diagnostic information for intermittent issues that are hard to reproduce and debug.
- Server drift - Restore uniformity when configuration files, databases, and data sources on your VMs and containers differ.
- Config drift - Ensure observed state matches desired state on your system configuration, e.g. Kubernetes yaml, Cloud config, etc.
- Memory exhaustion - Running out of memory rapidly degrades customer experience and must be pre-empted.
- Disk failures in kern.log - Detect when a disk has errors or has entirely failed by inspecting the OS’s kern.log. Automatically capture these events and kick off fixes such as recycling the VM.
- Network failures in kern.log - Detect when a network interface has errors or has entirely failed by inspecting the OS’s kern.log. Automatically capture these events and initiate fixes such as recycling the VM.
- Endpoints unreachable - Determine when there are no endpoints behind your Kubernetes service or these endpoints have become unreachable.
- Elastic sharding replica management - Determine when your elastic search clusters have too few replicas per shard, and automatically kick off healing.
- Log processing at the edge - Analyze log files on the box to identify issues that cause production incidents, and eliminate costs of centralized logging.
- Kafka data Processing Lag - Restart slow/broken consumers when systems are falling behind in processing messages through a queue.
- Kafka topic management - When the length of your Kafka topic is too long, applications may begin to break.
- Processes consuming too many resources - Determine if the system is using too much memory or CPU at the process level.
- Restart CoreDNS service - CoreDNS, the default Kubernetes DNS service, can degrade in performance with too many calls causing massive latency.
■ Avoid Major Outages
- Certificate rotation - Sooner or later every company gets bitten by expired certificates and when they do, it can cause a catastrophic outage.
- DNS lag - Trigger rolling restarts of the DNS servers when they are responding slowly and causing widespread system issues.
Companies around the world rely on Shoreline’s incident automation platform to resolve common incidents in production, broaden the team that can safely repair incidents, and perform live site debugging of new incidents. Pairing this Op Pack solutions content with the Shoreline platform accelerates time to value and increases ROI for Shoreline customers.
The well-known "No free lunch" theorem is something you’ve probably heard about if you’re familiar with machine learning in general. This article’s objective is to present the theorem as simply as possible while emphasizing the importance of comprehending its consequences in order to develop an AIOPS strategy ...
IT operations is a metrics-driven function and teams should keep score as a core practice. Services and sub-services break, alerts of varying quality come in, incidents are created, and services get fixed. Analytics can help IT teams improve these operations ...
Big Data makes it possible to bring data from all the monitoring and reporting tools together, both for more effective analysis and a simplified single-pane view for the user. IT teams gain a holistic picture of system performance. Doing this makes sense because the system's components interact, and issues in one area affect another ...
IT engineers and executives are responsible for system reliability and availability. The volume of data can make it hard to be proactive and fix issues quickly. With over a decade of experience in the field, I know the importance of IT operations analytics and how it can help identify incidents and enable agile responses ...
For businesses with vast and distributed computing infrastructures, one of the main objectives of IT and network operations is to locate the cause of a service condition that is having an impact. The more human resources are put into the task of gathering, processing, and finally visual monitoring the massive volumes of event and log data that serve as the main source of symptomatic indications for emerging crises, the closer the service is to the company's source of revenue ...
Our digital economy is intolerant of downtime. But consumers haven't just come to expect always-on digital apps and services. They also expect continuous innovation, new functionality and lightening fast response times. Organizations have taken note, investing heavily in teams and tools that supposedly increase uptime and free resources for innovation. But leaders have not realized this "throw money at the problem" approach to monitoring is burning through resources without much improvement in availability outcomes ...
Although 83% of businesses are concerned about a recession in 2023, B2B tech marketers can look forward to growth — 51% of organizations plan to increase IT budgets in 2023 vs. a narrow 6% that plan to reduce their spend, according to the 2023 State of IT report from Spiceworks Ziff Davis ...
Users have high expectations around applications — quick loading times, look and feel visually advanced, with feature-rich content, video streaming, and multimedia capabilities — all of these devour network bandwidth. With millions of users accessing applications and mobile apps from multiple devices, most companies today generate seemingly unmanageable volumes of data and traffic on their networks ...
In Italy, it is customary to treat wine as part of the meal ... Too often, testing is treated with the same reverence as the post-meal task of loading the dishwasher, when it should be treated like an elegant wine pairing ...
In order to properly sort through all monitoring noise and identify true problems, their causes, and to prioritize them for response by the IT team, they have created and built a revolutionary new system using a meta-cognitive model ...