Take the Results Presented with a Pinch of Salt

What are we to make of Microsoft’s release of a new study into the effect of Microsoft 365 Copilot for Small to Medium businesses? The blog post on the topic appeared on October 17 and highlights some results reported by Forrester Consulting, who Microsoft commissioned “to study the potential return on investment (ROI) of Microsoft 365 Copilot for SMBs.”

As the post says, the “results of the study are eye-opening” with big claims for projected ROI of up to 353% and $1.2 million of projected benefits. Projected is the important word here because it means that the ROI and benefits are potential and not achieved, even if they make good headlines. The report of 6% increase in net revenue and 20% reduction in operating costs seem more attainable.

Doubts About Any Technology Report

Doubts surface every time that I read a report about the gains that companies can make if they would only deploy some new technology. I ask myself if the authors of the report understand the technology they’re writing about as deeply as they should. I ask if the companies covered in the report are hand-picked to make the technology look as good as it can be. I ask what direction Microsoft gave Forrester Consulting when they commissioned the report and how independent Forrester can be in what they write about. And I ask if the results gathered from the over 200 companies surveyed for the report are massaged in any way. All nagging doubts honed from years of experience as a consultant.

I’ve no doubt that Microsoft 365 Copilot can do a good job for some SMEs, especially for companies who are backed up by a partner who knows the Copilot technology and understand where the potholes are. For instance, the assertion that 50% of time can be saved by legal firms in contract reviews is believable because many contracts cover the same ground, and a Copilot agent built for the purpose can reason over a corpus of contracts when reviewing text for problems.

It’s also true that Copilot’s ability to summarize text in email, Teams chats, and documents is of great help to people returning to work after a vacation. Catching up by wading through a full inbox or hundreds of Teams chats is never fun, and Copilot absolutely can help by summarizing information and presenting what happened while people were away in a very digestible format.

No Mention of Microsoft 365 Copilot Flaws

But I worry that the report ignores the flaws we know to exist in Microsoft 365 Copilot. Some SMEs are great at organizing their information; others are not, and they succumb to the same kind of group/teams sprawl and accumulation of digital debris that happens in enterprise tenants. SMEs might not have the same training capabilities as exist in larger organizations, which can lead to bad habits like oversharing through sloppy site permissions.

As you might imagine, none of this is covered by the Forrester report. No mention is present about why an SME might need to deploy Restricted SharePoint Search (or the newer but not yet available Restricted Content Discoverability capability), or deploy sensitivity labels to protect their most confidential documents from being reused by Copilot. There’s no comment about the way that errors can creep into user documents from Copilot responses and end up by corroding the reliability of stored documents. These are real issues surrounding the introduction of generative AI to Microsoft 365.

Just a Marketing Tool to Sell Microsoft 365 Copilot Licenses to SMEs

Then I remember that the Forrester report is no more than a marketing tool designed to encourage SMEs with Microsoft 365 Business Basic, Microsoft 365 Standard, or Microsoft 365 Business Premium pay for $360/user/year Microsoft 365 Copilot subscriptions. The companies covered in the report had up top 300 employees. At list price, Copilot licenses cost $108,000 annually for 300 employees. That’s a big investment for any SME.

But someone’s got to pay for the billions of dollars Microsoft is currently investing in AI, and a large percentage of the 400-million plus Office 365 installed base comes from the SME sector. If you work for an SME and are interested in Microsoft 365 Copilot, take the time to read the report, but do so with a large pinch of salt close at hand. Investing a large chunk of change in expensive software licenses without knowing exactly how you’ll achieve an ROI has never been a good business tactic.

Support the work of the Office 365 for IT Pros team by subscribing to the Office 365 for IT Pros eBook. Your support pays for the time we need to track, analyze, and document the changing world of Microsoft 365 and Office 365.