In my prior blog, I tried to paint a picture of some of the surprising (and not so surprising) highlights from our research on Unifying IT for Digital War Room Performance, which is also a webinar.
Start with Opening the Gates to the Digital War Room - What is it Now, and What is it Likely to Become?
One point to reinforce is that the digital war room — physical, virtual or hybrid — is not in retreat but in fact is growing in scope to include greater participation from development and security. It's also becoming more proactive, with on average more than 30% of "major incidents" before they impacted business service performance.
The reasons for this added (not diminished) level of relevance will be examined more in depth in my webinar on April 11th (and yes, there will be replays), but generally the answer lies in the fact that improved levels of team efficiency are critical to the future of IT, and the digital war room shines a spotlight on this evolving reality.
In this blog I'm providing a few additional highlights from the insights we got on digital war room organization and processes.
A Few Organizational Insights
One of the questions we asked was directed at finding out whether war rooms, as they evolve, were becoming more organizationally defined, or more sporadic and ad-hoc. The answer was solidly in the "more formalized" category (47%) versus the group with "more ad-hoc teams and processes" (28%). Another 22% indicated that their teams were already solidly formalized and established.
Then, when we evaluated success rates to this mix, we saw that those digital war rooms becoming "more formalized and established" were far more likely to align with digital war room effectiveness than the other groups.
Well defined teams that can be brought together across all domains provide a unique advantage over fragmented, technically isolated teams
If you think about this, it does suggest a contradiction to some of the trendier thinking endorsing multiple teams and more completely decentralized ways of working. But the logic for core consistency is clear. Well defined teams that can be brought together across all domains provide a unique advantage over fragmented, more technically isolated teams when confronting the full gamut of "major incident" possibilities.
And BTW, the average head count for these teams across small, medium and large was about 15. The implication being not that all 15 stakeholders are being activated for every single incident, but there are 15 individuals assigned and available for digital war room decision making on an on-going, as-needed basis. The trend, BTW, is toward growing not shrinking levels of involvement — in large part because of the accelerating need to include development and security professionals. The overall data also showed a significant role in digital war room decision making for non-IT, or business stakeholders.
Having a single organizational owner, also helps to drive war-room efficiencies. Interestingly, "Security/compliance" was in third place for war-room ownership after "ITSM" and the "executive suite." Having senior executive involvement helped, as well. The most prevalent was ongoing "director-level" involvement, but the most effective turned out to be "CIO-level" involvement.
Processes (or Lack Thereof)
In last week's blog, we enumerated the following critical processes that help to define war-room performance:
■ Initial awareness, which is usually driven by events or some other type of automated intelligence, or complaints to the service desk.
■ Response team engagement and coordination, bringing relevant stakeholders together and providing a context for them to work together.
■ Triage and diagnostics, where problems are understood in context and then detailed requirements for remediation can be defined.
■ Remediation, where active fixes to major incidents are made, often through change and configuration management procedures.
■ Validation, in which testing is done to ensure that actions for remediation were successful, ideally from a business impact as well as a purely technical perspective.
In the non-progressive category, we discovered that, based on our data, the average response indicated only a little more than half (2.57) of these processes were defined — a surprising revelation in a rather negative way. When we mapped "success rates" to the number of processes mapped out, however, we did get a reasonable correlation:
■ 3 for the extremely successful
■ 2.5 for the successful
■ 2 for the only marginally successful
The most prevalently defined process was response team coordination — which also turned out to be the most problematic or delay-causing process. In fact, identifying process with delay or problems mapped well to the processes that were most likely to be identified, suggesting that clarifying the reality of what's going on opens the door to realizing what's wrong and how improvements can be made.
Timing is Everything
Time to assemble an effective team, on average, was about 1.5 hours, which could be damaging when a serious outage occurred
We also asked about times associated with these processes. When we asked about the time to assemble an effective team, the average was about 1.5 hours, which could, of course, be meaningfully damaging when a serious outage occurred. When asked about total time to resolution, the average was about six hours, but 20 percent took more than eleven hours. Once again, as an average, this can be concerning for incidents with major business impacts.
These are again, just a few of many highlights from our research.
Don't forget to watch the webinar for a great many more insights.
Read my third and final blog on the digital war room: The Digital War Room in Changing Times: The Impacts of DevOps, Cloud and SecOps
Interestingly, some experts say that — although convergence is happening, and sharing the data has great value — the security dashboards should not necessarily be combined with observability dashboards for ITOps, NetOps or DevOps ...
The experts have all agreed that security teams can gain great benefits from utilizing observability data. But does this mean security and observability tools should be integrated, or even combined? ...
One reason why observability and security make a good pairing is that traditional telemetry signals — metrics, logs, and traces — are helpful to maintain both performance and security ...
Observability and security — are they a match made in IT heaven, or a culture clash from IT hell? Sorry to be so dramatic, but it's actually a serious question that has gravity. The convergence of observability and security could change IT operations as we know it. And many IT authorities see this as a good thing. With input from industry experts — both analysts and vendors — this 8-part blog series to be posted over the next two weeks will explore what is driving this convergence, the challenges and advantages, and how it may transform the IT landscape ...
The journey of maturing observability practices for users entails navigating peaks and valleys. Users have clearly witnessed the maturation of their monitoring capabilities, embraced DevOps practices, and adopted cloud and cloud-native technologies. Notwithstanding that, we witness the gradual increase of the Mean Time To Recovery (MTTR) for production issues year over year ...
Optimizing existing use of cloud is the top initiative — for the seventh year in a row, reported by 62% of respondents in the Flexera 2023 State of the Cloud Report ...
Gartner highlighted four trends impacting cloud, data center and edge infrastructure in 2023, as infrastructure and operations teams pivot to support new technologies and ways of working during a year of economic uncertainty ...
Developers need a tool that can be portable and vendor agnostic, given the advent of microservices. It may be clear an issue is occurring; what may not be clear is if it's part of a distributed system or the app itself. Enter OpenTelemetry, commonly referred to as OTel, an open-source framework that provides a standardized way of collecting and exporting telemetry data (logs, metrics, and traces) from cloud-native software ...
As SLOs grow in popularity their usage is becoming more mature. For example, 82% of respondents intend to increase their use of SLOs, and 96% have mapped SLOs directly to their business operations or already have a plan to, according to The State of Service Level Objectives 2023 from Nobl9 ...
Observability has matured beyond its early adopter position and is now foundational for modern enterprises to achieve full visibility into today's complex technology environments, according to The State of Observability 2023, a report released by Splunk in collaboration with Enterprise Strategy Group ...