Multicasting in this context refers to the process of directing data streams to two or more destinations. This might look like sending the same telemetry data to both an on-premises storage system and a cloud-based observability platform concurrently. The two principal benefits of this strategy are cost savings and service redundancy.
■ Cost Savings: Depending on the use-case, storing or processing data in one location might be cheaper than another. By multicasting the data, businesses can choose the most cost-effective solution for each specific need, without being locked into one destination.
■ Service Redundancy: No system is foolproof. By sending data to multiple locations, you create a built-in backup. If one service goes down, data isn't lost and can still be accessed and analyzed from another source.
The following are 10 things to consider before multicasting you observability data:
1. Consistency of User Expectations
It's crucial that both destinations receive data reliably and consistently. If it is unclear to users what data resides in which platform, it will impede adoption and make this strategy less effective. A common heuristic is to keep all of your data in a cheaper observability platform and send the more essential data to the more feature rich expensive platform. Likewise if one platform has data integrity issues due to the fact that no one is using it outside of break glass scenarios, it will reduce the effectiveness of this strategy.
2. Data Consistency
While it's good to have a process for evaluating the correctness of your data, when you write data to two systems, not everything will always line up. This could be due to ingestion latency, differences in how each platform rolls up long term data, or even just the graphing libraries that are used. Make sure to set the right expectations with teams, that small differences are expected if both platforms are in active use.
3. Bandwidth and Network Load
Transmitting the same piece of data multiple times can put an additional load on your network. This is more of an issue if you're sending data out from a cloud environment where you have to pay the egress cost.
Additionally, some telemetry components are aggregation points that can push the limits of vertical scaling (for example carbon relay servers). Multicasting the data may not be possible directly at that point in the architecture due to limitations in how much data can traverse the NIC. It's essential to understand the impact on bandwidth and provision appropriately.
4. Cost Analysis
While multicasting can lead to savings, it's crucial to do a detailed cost analysis. Transmitting and storing data in multiple places might increase costs in certain scenarios.
5. Security and Compliance
Different storage destinations might have different security features and compliance certifications. Ensure that all destinations align with your company's security and regulatory needs.
6. Tool Integration
Not all observability tools might natively support multicasting data. Some observability vendors' agents can only send data to their product. You may need to explore a multi-agent strategy in cases like that.
7. Data Retrieval and Analysis
With data residing in multiple locations, the way your teams will need to engage with the data may differ. If you're using a popular open source telemetry dashboarding tool, then there will be at least some degree of consistency with how to engage with the data, even if the query syntax supported by each platform is different. This becomes a little more challenging if your teams are using the UI of the higher cost observability platform.
8. Data Lifecycle Management
Consider how long you need the data stored in each location. You might choose to have short-term data in one location and long-term archival in another.
9. Maintenance and Monitoring
With more destinations come more points of potential failure. Implement robust monitoring to ensure all destinations are consistently available and performing as expected. This is a good opportunity to introduce cross monitoring, where each observability stack monitors the other.
10. Migration and Scalability
As your business grows, you might need to migrate or scale your lower cost observability platform. Ensure the chosen destinations support such migrations without significant overhead.
Conclusion
Multicasting data that is collected by your observability tools offers an innovative approach to maximize both cost efficiency and system resilience. However, like all strategies, it comes with its set of considerations. By understanding and preparing for these considerations, businesses can harness the power of this approach to create observability solutions that are both robust and cost-effective.
The Latest
Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, OpenTelemetry and related technologies will evolve and impact business in 2024. Part 2 covers more on Observability ...
The Holiday Season means it is time for APMdigest's annual list of Application Performance Management (APM) predictions, covering IT performance topics. Industry experts — from analysts and consultants to the top vendors — offer thoughtful, insightful, and often controversial predictions on how APM, observability, AIOps and related technologies will evolve and impact business in 2024. Part 1 covers APM and Observability ...
To help you stay on top of the ever-evolving tech scene, Automox IT experts shake the proverbial magic eight ball and share their predictions about tech trends in the coming year. From M&A frenzies to sustainable tech and automation, these forecasts paint an exciting picture of the future ...
Incident management processes are not keeping pace with the demands of modern operations teams, failing to meet the needs of SREs as well as platform and ops teams. Results from the State of DevOps Automation and AI Survey, commissioned by Transposit, point to an incident management paradox. Despite nearly 60% of ITOps and DevOps professionals reporting they have a defined incident management process that's fully documented in one place and over 70% saying they have a level of automation that meets their needs, teams are unable to quickly resolve incidents ...
Today, in the world of enterprise technology, the challenges posed by legacy Virtual Desktop Infrastructure (VDI) systems have long been a source of concern for IT departments. In many instances, this promising solution has become an organizational burden, hindering progress, depleting resources, and taking a psychological and operational toll on employees ...
Within retail organizations across the world, IT teams will be bracing themselves for a hectic holiday season ... While this is an exciting opportunity for retailers to boost sales, it also intensifies severe risk. Any application performance slipup will cause consumers to turn their back on brands, possibly forever. Online shoppers will be completely unforgiving to any retailer who doesn't deliver a seamless digital experience ...
Black Friday is a time when consumers can cash in on some of the biggest deals retailers offer all year long ... Nearly two-thirds of consumers utilize a retailer's web and mobile app for holiday shopping, raising the stakes for competitors to provide the best online experience to retain customer loyalty. Perforce's 2023 Black Friday survey sheds light on consumers' expectations this time of year and how developers can properly prepare their applications for increased online traffic ...
This holiday shopping season, the stakes for online retailers couldn't be higher ... Even an hour or two of downtime for a digital storefront during this critical period can cost millions in lost revenue and has the potential to damage brand credibility. Savvy retailers are increasingly investing in observability to help ensure a seamless, omnichannel customer experience. Just ahead of the holiday season, New Relic released its State of Observability for Retail report, which offers insight and analysis on the adoption and business value of observability for the global retail/consumer industry ...
As organizations struggle to find and retain the talent they need to manage complex cloud implementations, many are leaning toward hybrid cloud as a solution ... While it's true that using the cloud is not a "one size fits all" proposition, it is clear that both large and small companies prefer a hybrid cloud model ...