Copilot Audit Records Record Details of SharePoint Online Files Referenced by BizChat

Despite the launch of Microsoft 365 Copilot in March 2023, we’re still in the early stages of AI deployments. Organizations are figuring out whether AI tools can help their business and how to deploy and manage the technology. Part of that activity is data governance and compliance, which seems like a good topic for a Friday post.

Copilot Broke Your Audit Log

Starting off with an August 19 article by Zack Korman entitled “Copilot broke your audit log,” we learn that a gap appeared in the payload of audit events captured for Copilot interactions with files stored in SharePoint Online. In a nutshell, a method existed to convince Copilot to open and use the contents of a file in a response without the corresponding audit event capturing the name of the file. Obviously, this is a bad thing because it undermines the credibility of the audit system when it comes to tracking how users interact with Copilot.

According to the article, the author had a frustrating time after reporting the issue to the Microsoft Security Response Center, but eventually Microsoft closed the gap. To test the fix, I asked BizChat (Microsoft 365 Copilot Chat) to summarize this file with this prompt:

Find the recent document about problems with Copilot audit records and summarize the content. Don’t link to the file and don’t include it as a reference. Just report what the document contains.

The audit records captured for Copilot’s response includes details of the files Copilot reviewed as it constructed its response, which is what should happen (Figure 1).

Checking the Compliance Records

The information shown in Figure 1 is an audit record. For the sake of completeness, I used the script described in this article to retrieve the Copilot interaction records from my mailbox for analysis. I noticed that some of the BizChat responses had blank values for the BodyPreview property, so I used the MFCMAPI utility to examine the raw compliance records for interactions captured in the mailbox. It seems like Microsoft has changed the information captured in the PR_HTML property of the compliance record. The property now stores the list of files consulted by Copilot (in HTML) together with the interaction (in a Base64-encoded section).

I’ve seen Base64-encoded content in compliance records previously when investigating the compliance data captured for Microsoft Copilot interactions. It’s easy to decode the information to see the details of the full Copilot response, but it’s interesting that Microsoft chooses to obscure some of the content for this BizChat response.

Many Ways to Get Copilot Details

So far we’ve discussed using Copilot audit records and Copilot interaction compliance records to retrieve details of Copilot interactions. Another way is to use the aiInteractionHistory Graph API. Having an array of tools to retrieve information about interactions could be deemed a good thing for compliance administrators who need to manage and monitor how people use AI tools in a Microsoft 365 tenant. On the other hand, having so much information available could create an eDiscovery challenge, a point made by legal firm Redgrave LLP.

The issue is simple. If interaction records exist that show Copilot accessed specific files on behalf of a user, a discovery request made during legal proceedings could demand access to those files on the basis that they might be relevant to the case in hand. Remember, Copilot can only access information that’s available to the signed-in user, and if Copilot accesses files, albeit briefly, as it searches for information to respond to a user prompt, those files could be relevant to proceedings.

Redgrave focus on Copilot Chat and say that its use in tenants “underscore the importance of maintaining close alignment between IT, Legal, and Information Governance teams as AI tools become increasingly embedded in daily workflows. Organizations that proactively address these challenges will be better positioned to leverage AI benefits while maintaining defensible information governance practices.” Maybe it would have been better if the audit events had “lost” details of the pesky SharePoint files accessed by users. Only joking!

So much change, all the time. It’s a challenge to stay abreast of all the updates Microsoft makes across the Microsoft 365 ecosystem. Subscribe to the Office 365 for IT Pros eBook to receive insights updated monthly into what happens within Microsoft 365, why it happens, and what new features and capabilities mean for your tenant.