What to Look For in an Analyst Report
December 03, 2015

Jonah Kowall

Share this

The Internet is an amazing medium for anyone looking to articulate an opinion. Everyone should practice writing and expressing themselves, as it's a great tool to build throughout life. The ability to publish information is everyone's right. However, credibility is a whole different challenge. Why do we trust what is published in the New York Times, but don't trust what is published in the tabloids? It comes down to rigor in journalistic practices. In research, it's also tied to a strict methodology or process.

During my time at Gartner, I learned a great deal about the differences between analyst firms, mostly by meeting and discussing things with friends at other analyst firms. I quickly learned which firms will write a whitepaper, and which firms will not create marketing materials for software companies for pay.

In the case of Gartner, the analyst has the freedom to publish anything, if they follow the extremely rigorous research process and can defend the opinion they are creating as an analyst. The process for analysts to publish branded documents, such as a Magic Quadrant, is over 30 pages, but publicly a small subset of this is disclosed. Clients get another deeper look in this document (Gartner subscribers only).

During the publication process for any document, there is rigorous peer review, management review, and editing to handle any issues in process or the fact base. Aside from this process and methodology, the analyst speaks with hundreds of end users of a particular technology through the year on phone calls and at conferences. This allows the analyst to comprehend the reality of a market versus what vendors may care to share with an analyst.

When witnessing small analyst firms attempting to assess markets without end user perspective and without speaking to all the vendors in the research — while blatantly requesting and collecting money directly from vendors before research is even drafted — the red flags come up. I discourage any organization from participating in these blatant acts of extortion. When vendors sponsor and fund this research, it just enables the lie to persist, year after year. This is clearly a major violation of journalistic integrity. The vendors who pay continually jam this poorly crafted research down end users' throats, and avoid the questions about where it comes from. This occurs regularly.

The violations don't just stop there, but clearly there are researchers who infringe on specific research formats, whether it's the Gartner Magic Quadrant, the Forrester Wave, or the IDC MarketScape. The lawyers of the respective firms get involved in these disputes, but small single person "analyst firms" seem to do this regularly and get slapped with cease and desist letters.

Leading researchers in the public domain should be able to publicly discuss, dispute, and learn from others in public forums. You'll see top analyst firms participate in conferences, panels, LinkedIn groups, Twitter, and other public forums. If a researcher is going to build a fact-based opinion of something, they should be able to participate and defend that position. Many of these smaller analyst firms or independent researchers avoid doing so, continually hide, or many times do not even have a name associated with research. I give big kudos and credibility to the researchers and analysts who stand behind what they publish.

Jonah Kowall is CTO of Kentik
Share this

The Latest

June 21, 2024

In MEAN TIME TO INSIGHT Episode 7, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at EMA continues on the theme of network automation, asking the question: Build or Buy? ...

June 20, 2024

The total cost of downtime for Global 2000 companies is $400 billion annually — or 9% of profits — when digital environments fail unexpectedly, according to The Hidden Costs of Downtime, a new report from Splunk ...

June 18, 2024

With the rise of digital transformation and the increasing reliance on applications for business operations, the need for application performance management (APM) has become more critical ... This blog explains what APM is all about, its significance and key features ...

June 17, 2024

Generative AI (GenAI) has captured significant attention by redefining content creation and automation processes. Despite this surge in GenAI's popularity, it's crucial to highlight the continuous, vital role of machine learning (ML) in underpinning crucial business functions. This era is not about GenAI replacing ML; rather, it's about these technologies collaborating to supercharge intelligent automation across industries ...

June 13, 2024

As organizations continue to navigate their digital transformation journeys, the need for efficient, secure, and scalable data movement strategies has never been more critical ... In an era when enterprise IT landscapes are continually evolving, the strategic movement of data has become a cornerstone of maintaining agility, competitive edge, and operational efficiency ...

June 12, 2024

In May, New Relic published the State of Observability for IT and Telecommunications Report to share insights, statistics, and analysis on the adoption and business value of observability for the IT and telecommunications industries. Here are five key takeaways from the report ...

June 11, 2024
Over the past decade, the pace of technological progress has reached unprecedented levels, where fads both quickly rise and shrink in popularity. From AI and composability to augmented reality and quantum computing, the toolkit of emerging technologies is continuing to expand, creating a complex set of opportunities and challenges for businesses to address. In order to keep pace with competitors, avoiding new models and ideas is not an option. It's critical for organizations to determine whether an idea has transformative properties or is just a flash in the pan — a challenge tackled in Endava's new 2024 Emerging Tech Unpacked Report ...
June 10, 2024

The rapidly evolving nature of the industry, particularly with the recent surge in generative AI, can catch firms off-guard, leaving them scrambling to adapt to new trends without the necessary funds ... This blog will discuss effective strategies for optimizing cloud expenses to free up funds for emerging AI technologies, ensuring companies can adapt and thrive without financial strain ...

June 06, 2024

Software developers are spending more than 57% of their time being dragged into "war rooms" to solve application performance issues, rather than investing their time developing new, cutting-edge software applications as part of their organization's innovation strategy, according to a new report from Cisco ...

June 05, 2024

Generative Artificial Intelligence (GenAI) is continuing to see massive adoption and expanding use cases, despite some ongoing concerns related to bias and performance. This is clear from the results of Applause's 2024 GenAI Survey, which examined how digital quality professionals use and experience GenAI technology ... Here's what we found ...