New in 2021:
Gartner provided 2021 updates to its Peer Insights offering, including changes to its Voice of the Customer report methodology and review scoring calculations:
- Changes to Voice of the Customer (VoC) report methodology and Customer Choice distinctions place a premium on review quality. Gartner increased the review count from 10 reviews to 20 reviews (in the last 18 months) to earn inclusion in the report, however, the coveted Customer Choice distinction now includes a new element in its algorithm – the ‘Willingness to Recommend’ score. These changes begin in VoC reports published in May 2021.
- Changes to the scoring algorithm, which weighs recent reviews higher than those 13+ months old, start immediately.
- The Peer Insights website now defaults to showing all reviews, not just those submitted in the last year. Newer reviews are weighted heaviest, while those 12 months old are weighted 50% less (with a progressive 50% weighting reduction in each subsequent year).
Voice of the Customer reports get a methodology change
Gartner’s Peer Insights team hosted a January 2021 webinar to announce changes to its peer review site scoring calculations, VoC report methodology, and vendor engagement program. Webinar host and Peer Insights lead Anatoli Olkhovets focused the bulk of the presentation on updates (and upgrades) to the vendor engagement program and vendor-facing review management tools, though the meatiest updates for AR Pros came in updates to review scoring and VoC report methodology.
Gartner’s popular VoC reports, which feature the ‘Customer’s Choice’ distinction for each market’s top-rated offerings, will get a scoring overhaul starting in May 2021. To be featured as a Customer’s Choice vendor, companies:
- Must have 20+ reviews on the platform over the last 18 months.
- Must score ‘above average’ on the X-axis (User Interest and Adoption) and the Y-axis (Overall Rating)”.
Moving Right (X-axis)
There are now three equally-weighted elements to calculations for User Interest & Adoption scoring:
- Review count
- Willingness to Recommend *NEW*
- Firmographics (aka distribution of reviews across industries, geographic regions, and company size)
Gartner revised its methodology for these scores. Previously, the X-axis used to be calculated using the Product Adoption Review Coverage (PARC) Score, which focused only on review count and firmographics.
Moving Up (Y-axis)
For VoC reports, product review scores only include reviews from the last 18 months. The scores give full weight to the reviews submitted (or updated) within the last 12 months and will give 50% weighting to those published in the first 6 months of the specified 18 month evaluation period.
Willingness to Recommend:
Gartner is emphasizing scores for Willingness to Recommend, which stem from a review survey question asking reviewers if they would or would not recommend a product to a peer, mirroring the familiar NPS methodology with scores from 0 (not at all likely to recommend) to 10 (extremely likely to recommend). Previously, Gartner had used a modified Likert scale (yes, yes with reservations, I don’t know, and no) but will use the 10 point scale moving forward.
Quality over Quantity:
Gartner called out its ‘asymptotic function’ for User Interest & Adoption (X-axis) scoring, which it says will normalize scoring across each element regardless of total review count. For those meeting the minimum review count threshold (20 in the past 18 months), this ‘diminishing returns’ calculation will reward quality reviews over quantity of reviews.
Other updates of note:
- Gartner is removing its limit of 7 Customer’s Choice vendors. There will be no upper limit, though the firm says score averages will ‘naturally’ limit the number of Customer’s Choice vendor selections. However, there is a new limit of 25 vendors which can appear on the report.
Older reviews are still eligible to be updated. Reviewers are able to access their previous reviews and provide updates to text or scoring. When the review is accessed in this manner, it is removed from the site, and any changes go through the review moderation/approval process. It will then be republished with a new date, qualifying it for the 12-month cycle. Gartner says this is a very popular feature and the team is exploring ways to ‘streamline this process on the vendor side.’
Gartner also announced an update to its existing vendor engagement program, including a rebrand to ‘Customer First’ and an introduction of badge-based recognition.
Gartner said there will be no changes to the value provided to vendors participating in the Customer First program – each participating vendor will still get free access to readership analytics tools and advanced website capabilities.
Badges will be available for placement on vendor websites starting in March and will externally promote participation in the Customer First Peer Insights program. Badges will link directly to vendor profiles on the Peer Insights website.
If you’re looking for more information beyond these specific updates to Peer Insights, check out our Guide to Peer Review Sites. Included in this guide, we share an overview of the peer review sites landscape, best practices on management, as well as specific information on Gartner Peer Insights.