Microsoft 365 Substrate Captures Interaction Details for Microsoft Copilot

After writing about how to use the Microsoft Graph PowerShell SDK to analyze the interactions between users and Microsoft 365 Copilot in various apps, I was asked if the code reports interaction records for Microsoft Copilot. This is the free version of Copilot that appears in the Microsoft 365 app when a signed-in Entra ID user account doesn’t have a Microsoft 365 Copilot license.

The big difference between the free and paid-for version is that Microsoft 365 Copilot can use Graph queries to find email, Teams messages, and documents to ground its queries while Microsoft Copilot is limited to Microsoft’s LLMs and Bing web searches. In addition, Microsoft 365 Copilot comes with extra features, such as custom Copilot agents for SharePoint Online.

Both versions support enterprise data protection (EDP). Microsoft added support for EDP to Microsoft Copilot in August 2024 and the announcement specifically says that information about prompts and responses is retained for eDiscovery purposes.

Asking Microsoft Copilot

My first step to gather information was to ask Microsoft Copilot if it generates interaction compliance records. Figure 1 shows the negative response.

Looking Behind the Scenes

As Microsoft Copilot couldn’t answer the question, it was time to look behind the scenes. I figured that the Microsoft 365 substrate would store anything it captured for Microsoft Copilot interactions in the same hidden TeamsMessagesData folder in the user’s mailbox.

Some are curious why Microsoft selected TeamsMessagesData as the storage location for these records. It doesn’t really matter what folder is used if it’s hidden and indexed for eDiscovery, but I think Microsoft chose TeamsMessagesData because the Copilot chats are very much like regular Teams one-on-one chats. The substrate captures Teams compliance records for one-on-one chats in the same folder.

MFCMAPI is the best tool to investigate mailbox contents. After using Microsoft Copilot several times, I opened the TeamsMessagesData folder with MFCMAPI and discovered that the substrate had captured compliance records for the Copilot interactions. Figure 2 shows the record captured for the prompt shown in Figure 1.

Once I located the compliance records, it was easy to update the PowerShell script to extract and report the Microsoft Copilot interactions. The updated code is available from GitHub.

No Data Shown for Responses

I noticed that compliance records captured for Microsoft Copilot responses do not include the response in the Body and BodyPreview properties. The same is true for responses generated for Microsoft 365 Chat (BizChat) responses. Looking back through records for Microsoft 365 Chat interactions, it appears that the only output is any documents located by Copilot to form its response. In Figure 3, we see a reference to a document in a Microsoft 365 Chat response followed by some base 64 encoded text.

Inputting the encoded text into an online decoder reveals the text (Figure 4). It looks like whatever routine Microsoft uses to generate the compliance record doesn’t decode the text before it’s written into the mail item used to store the record in TeamsMessagesData.

The encoded state of the information also explains why the Activity Explorer in the AI Hub in the Purview portal can’t display Copilot’s response to a prompt (Figure 5).

Summarizing Microsoft Copilot and Compliance Records

The answer to the question is that compliance records are generated for Microsoft Copilot interactions. However, the information logged in the compliance records isn’t as easy to access as it should be. The flaw shared by Microsoft Copilot and Microsoft 365 chat suggests that some buggy code is shared by the two apps. It should be easy for Microsoft to decode responses from base64 before including clear text in compliance records.

The issue is reported, but quite when a fix will appear is anyone’s guess. Hopefully, because the problem means that compliance records aren’t as useful as they should be, the fix should appear soon.

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.