The Evolution of Application Centric Network Visibility in Public Cloud
August 05, 2020

Nadeem Zahid
cPacket Networks

Share this

Application or network downtime is expensive, and given the growing numbers and types of high-availability and mission-critical applications, systems and networks — and our increasing reliance on them — ensuring consistent access to mission-critical applications is essential for ensuring customer loyalty and keeping employees productive. Businesses must recognize that applications availability depends on the network and implement a strategy to ensure network-aware application performance monitoring.

As most enterprises go cloud-first and cloud-smart, a key component in providing full network-aware application and security monitoring is eliminating blind spots in the public cloud. A good network visibility solution must be able to reliably monitor traffic across an organization's current and future hybrid network architecture — with physical, virtual, and cloud-native elements deployed across the data centers, branch offices and multi-cloud environments.

Unfortunately for IT teams, up until mid-2019, every major public cloud platform was a black box from the above perspective. Companies could have rich insight into network and application performance across their private data center network, as well as into and out of the cloud, but what happened inside the cloud itself was a mystery. This made application performance monitoring and security assurance difficult and porting of on-premise investigation and resolution workflows virtually impossible.

Companies worked around this lack of visibility with a variety of compromised methods, including deploying traffic forwarding agents (or container-based sensors) and using log-based monitoring. Both have limitations. Feature-constrained forwarding agents and sensors must be deployed for every instance and every tool — a costly IT management headache — or there is a risk of blind spots and inconsistent insight. Event logging must be well-planned and instrumented in advance and can only prepare for anticipated issues as snapshots in time. Neither provides the high-quality and continuous data, such as packet data, that would provide the required depth needed to troubleshoot complex application, security or user experience issues.

To solve this problem, public clouds like AWS and Google Cloud have introduced game-changing features over the last year such as VPC traffic/packet mirroring that significantly impact the ability of IT departments to monitor cloud deployments. 

Microsoft Azure had introduced a virtual TAP feature for the same purpose, but it has been put on hold for now. It’s worth a closer look to assess what it means for network and application management, and security use cases.

In mid-2019 Amazon, followed by Google Cloud, introduced traffic mirroring (packet mirroring in case of Google) functionality as part of their respective Virtual Private Cloud (VPC) offerings. Simply stated, this traffic mirroring feature duplicates network traffic to and from the client’s applications and forwards it to cloud-native performance and security monitoring tool sets for assessment. This eliminates the need to deploy ad-hoc forwarding agents or sensors in each VPC instance for every monitoring tool and reduces complexity. Compared to log data, it delivers much richer and deeper situational awareness that’s needed for network and application monitoring or security investigations. The result is simplicity, elasticity and cost savings.

Traffic or packet mirroring isn’t enough on its own, however. Just like the agent or sensor approach, it simply provides the access to raw packet data (equivalent to TAPs in the physical world) which is not quite ready to feed directly into monitoring and security tools. The complete solution is to use traffic mirroring along with cloud-based virtual packet brokering, packet capture, flow generation and analytics middleware. This adds value in a variety of ways.

In Amazon or Google Cloud, virtual/cloud packet broker can multiply the value of VPC mirrored traffic by pre-processing operations such as header stripping, filtering, deduplicating and load-balancing the traffic feeds to cloud-native tools, which saves on costs while forwarding the right data to the right tools.

In Azure, if the virtual packet broker supports an "inline mode" it can be a viable alternative to VPC traffic mirroring or agent-based mirroring features. One or more of the feeds from the packet broker can be fed to a packet-to-flow gateway tier to generate flow data such as Netflow/IPFIX if certain tools prefer flow data. A virtual/cloud packet capture tier can take a feed from the packet broker as well to record interesting data to cloud storage for later retrieval, playback and analysis. This is particularly useful for security-centric Network Detection and Response, forensics and incident response.

While most of the above value on top of cloud traffic mirroring (inline or non-inline) involves data or network intelligence delivery, more value comes from correlating and analyzing the data to spit out something more meaningful, useful and actionable. This is where the rich network analytics tier comes in. These tools consume the fine-grain metadata extracted from the above middleware and turns that into visualizations and dashboards that enable IT NetOps, SecOps, AppOps and CloudOps teams to effectively perform their jobs. The high-quality metadata can be exported to other tools such as threat detection, behavioral analytics and service monitoring solutions to enrich their effectiveness. Features such as baselining, application dependency mapping and automated alerting, coupled with artificial intelligence (AI) and machine learning (ML) capabilities add the ultimate value for today’s demanding ITOps — headed to AIOps.

In summary, a cohesive hybrid visibility suite that integrates with the new VPC traffic mirroring capabilities offered by the leading cloud providers allows organizations to use a consistent mix of tools, workflows, data and insight when managing hybrid environments (the proverbial "single pane of glass"). The ability to gather the same deep insights across both private and public infrastructure is a game changer for application and network performance monitoring and security. Black boxes shouldn’t exist in corporate networks, making fully network-aware public cloud monitoring a welcome change. This simplifies network and application performance management and speeds up mean time to resolution — ultimately enhancing end-user experience and reducing customer churn — all by de-risking IT infrastructure and operations.

Nadeem Zahid is VP of Product Management & Marketing at cPacket Networks
Share this

The Latest

September 23, 2021

The Internet played a greater role than ever in supporting enterprise productivity over the past year-plus, as newly remote workers logged onto the job via residential links that, it turns out, left much to be desired in terms of enabling work ...

September 22, 2021

The world's appetite for cloud services has increased but now, more than 18 months since the beginning of the pandemic, organizations are assessing their cloud spend and trying to better understand the IT investments that were made under pressure. This is a huge challenge in and of itself, with the added complexity of embracing hybrid work ...

September 21, 2021

After a year of unprecedented challenges and change, tech pros responding to this year’s survey, IT Pro Day 2021 survey: Bring IT On from SolarWinds, report a positive perception of their roles and say they look forward to what lies ahead ...

September 20, 2021

One of the key performance indicators for IT Ops is MTTR (Mean-Time-To-Resolution). MTTR essentially measures the length of your incident management lifecycle: from detection; through assignment, triage and investigation; to remediation and resolution. IT Ops teams strive to shorten their incident management lifecycle and lower their MTTR, to meet their SLAs and maintain healthy infrastructures and services. But that's often easier said than done, with incident triage being a key factor in that challenge ...

September 16, 2021

Achieve more with less. How many of you feel that pressure — or, even worse, hear those words — trickle down from leadership? The reality is that overworked and under-resourced IT departments will only lead to chronic errors, missed deadlines and service assurance failures. After all, we're only human. So what are overburdened IT departments to do? Reduce the human factor. In a word: automate ...

September 15, 2021

On average, data innovators release twice as many products and increase employee productivity at double the rate of organizations with less mature data strategies, according to the State of Data Innovation report from Splunk ...

September 14, 2021

While 90% of respondents believe observability is important and strategic to their business — and 94% believe it to be strategic to their role — just 26% noted mature observability practices within their business, according to the 2021 Observability Forecast ...

September 13, 2021

Let's explore a few of the most prominent app success indicators and how app engineers can shift their development strategy to better meet the needs of today's app users ...

September 09, 2021

Business enterprises aiming at digital transformation or IT companies developing new software applications face challenges in developing eye-catching, robust, fast-loading, mobile-friendly, content-rich, and user-friendly software. However, with increased pressure to reduce costs and save time, business enterprises often give a short shrift to performance testing services ...

September 08, 2021

DevOps, SRE and other operations teams use observability solutions with AIOps to ingest and normalize data to get visibility into tech stacks from a centralized system, reduce noise and understand the data's context for quicker mean time to recovery (MTTR). With AI using these processes to produce actionable insights, teams are free to spend more time innovating and providing superior service assurance. Let's explore AI's role in ingestion and normalization, and then dive into correlation and deduplication too ...