Copilot Usage Reports Weak on Detail

Announced in message center notification MC877369 (29 August 2024, Microsoft 365 roadmap item 396562), the Microsoft Graph beta usage reports API now includes support for Copilot for Microsoft 365 tenant usage data. All tenants that use Copilot for Microsoft 365 should now have access to the usage data.

Microsoft says that the availability of this information will “facilitate the creation of customized reporting and analytics,” but the fact is that the data exposed by the API is bare-bones. On the upside, the data matches what’s available in the report section of the Microsoft 365 admin center (Figure 1).

Tenant-level summary of Copilot-enabled (licensed) users and active users.

Adoption trend (tenant summary) over time.

Last activity date for Copilot interaction in different apps for each user.

Accounts accessing the Graph data must have a Copilot for Microsoft 365 license.

User Count Summary

The user count summary report returns a count of the user accounts licensed for Copilot for Microsoft 365 (enabled users) and a count of the users with an active interaction with Copilot in each app during the reporting period (7, 30, 90, or 180 days). Unsurprisingly, when someone is enabled for Copilot in one app, they’re usually enabled for all:

Teams

Outlook (classic, new Outlook for Windows, OWA).

Excel.

PowerPoint.

Copilot Graph-grounded chat (aka Copilot Chat).

OneNote.

Loop.

$Uri = “https://graph.microsoft.com/beta/reports/getMicrosoft365CopilotUserCountSummary(period=’D90′)”
$Data = Invoke-GraphRequest -Uri $Uri -Method Get
$Data.value.adoptionByProduct

Name Value
—- —–
loopEnabledUsers 100
reportPeriod 90
oneNoteActiveUsers 3
wordEnabledUsers 100
powerPointEnabledUsers 100
microsoftTeamsActiveUsers 97
oneNoteEnabledUsers 100
excelActiveUsers 43
loopActiveUsers 2
copilotChatEnabledUsers 100
outlookEnabledUsers 100
anyAppEnabledUsers 100
anyAppActiveUsers 97
microsoftTeamsEnabledUsers 100
excelEnabledUsers 100
wordActiveUsers 61
powerPointActiveUsers 12
copilotChatActiveUsers 73
outlookActiveUsers 18

User Activity Detail

This report is the most interesting because it details the last activity date for Copilot interaction by users with each of the various Copilot-enabled apps. In addition, the last activity date for any Copilot interaction with any of the supported apps is published (lastActivityDate). An array (value) holds a separate usage report for each Copilot-enabled account.

The user principal name and display name is obfuscated if the tenant data privacy control is enabled. In the following extract, we see that the user has never used Copilot for Loop and OneNote and hasn’t used Copilot with PowerPoint since April 11, 2024:

$Uri = https://graph.microsoft.com/beta/reports/getMicrosoft365CopilotUsageUserDetail(period=’D90′)
$Data = Invoke-GraphRequest -Uri $Uri -Method Get
$Data.value[0]

Name Value
—- —–
copilotActivityUserDetailsByP… {System.Collections.Hashtable}
microsoftTeamsCopilotLastActi… 2024-09-05
outlookCopilotLastActivityDate 2024-08-29
lastActivityDate 2024-09-05
reportRefreshDate 2024-09-08
excelCopilotLastActivityDate 2024-09-05
loopCopilotLastActivityDate
oneNoteCopilotLastActivityDate
copilotChatLastActivityDate 2024-09-05
powerPointCopilotLastActivity… 2024-04-11
userPrincipalName Tony.Redmond@office365itpros.com
displayName Tony Redmond
wordCopilotLastActivityDate 2024-09-05

Adoption Trend over Time

This report returns an array called adoptionByDate with entries for each day during the reporting period (7, 30, 90, or 180 days). The purpose of the report is to track progress in Copilot adoption over time and to note if any specific action had an effect. For instance, you might run an education campaign to teach users how to generate effective results using Copilot in Excel. Over the weeks following the campaign, you’d expect to see the number of users who use Copilot in Excel to grow.

$Uri = “https://graph.microsoft.com/beta/reports/getMicrosoft365CopilotUserCountTrend(period=’D90′)”
$Data = Invoke-GraphRequest -Uri $Uri -Method Get
$Data.Value.copilotActivityUserDetailsByPeriod

reportDate 2024-06-17
excelEnabledUsers 100
wordActiveUsers 51
powerPointActiveUsers 11
copilotChatActiveUsers 66
outlookActiveUsers 15
loopEnabledUsers 100
oneNoteActiveUsers 1
wordEnabledUsers 100
powerPointEnabledUsers 100
microsoftTeamsActiveUsers 86
oneNoteEnabledUsers 1
excelActiveUsers 21
loopActiveUsers 1
copilotChatEnabledUsers 100
outlookEnabledUsers 100
anyAppEnabledUsers 100
anyAppActiveUsers 86
microsoftTeamsEnabledUsers 100

Track Copilot Activity Using Audit Records instead of Copilot Usage Reports

Although it’s nice to have Copilot usage reports included in the Graph API, the information exposed isn’t very informative in terms of how people use Copilot. Any enterprise who invests large sums of money in expensive Copilot for Microsoft 365 licenses will find a dearth of detail here in terms of understanding whether the investment is justified. In many cases, you will be better off analyzing the audit records captured for Copilot interactions to figure out what’s really going on.

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.