Older Files With Long Retention Periods Might Disappear Unexpectedly

Microsoft introduced retention policies and retention labels as part of its Office 365 Data Governance Framework in April 2017. These components are part of the Purview Data Lifecycle Management solution now. Capabilities have evolved a lot since 2017 in areas like special auto-label policies for “cloudy” attachments, disposition reviews, and the use of trainable classifiers, but the basics remain the same. Retention policies or retention labels track how old items are and when those items reach a threshold, a retention action happens. Often, the action is to delete the item.

Which brings me to a puzzling experience I had a few weeks ago when spreadsheets that I use to track various topics disappeared. The items that I noticed all came from my OneDrive for Business account, and when I checked there, I found the items in the first stage recycle bin. The question was what action moved the items into the recycle bin.

Old Retention Labels Might Have Simple Settings

The answer was simple. When Microsoft introduced retention labels and policies, I created a batch of labels and started to use them to manage content. At the time, things were simpler, and retention operated on creation date. Today, you can choose for retention to operate on the basis of the last modified date, which is the setting I use in more recently-created retention labels. Monitoring the last modified date is better because it means that files that remain in active use are not removed by retention processing, which is probably what you want.

The retention label applied to the disappeared files had a long retention period (Figure 1) but counted the retention period from the created date. It just so happened that all the files that ended up in the recycle bin were created in early 2015 and had therefore reached the end of the retention period. The next time Purview processed OneDrive, it duly noted the fact and took the retention action to move the files into the recycle bin.

Mystery Solved

The mystery of why my files suddenly disappeared from view was solved, but there’s a serious lesson to be learned here. It is quite likely that other Microsoft 365 tenants have retention policies or labels that operate on a “when created” rather than “last modified” basis. Checking the created date for files is very effective at clearing out old material but falls down when that material, although old, is still in active use.

The problem is that you can’t simply update a retention label to change its retention settings. You can change the retention period to increase or decrease the number of days for Purview to retain an item, but you cannot change “when created” to “last modified.” Increasing the retention period is a pragmatic way to solve the immediate problem, but it means that some files that should be removed will stay around for longer, and that’s a bad thing in an era when AI tools cheerfully consume even the oldest and most outdated information to generate responses.

A better long-term solution is to replace old retention labels with updated versions that use the last modified date. This article explains how to use the Get-MgDriveItemRetentionLabel cmdlet from the Microsoft Graph PowerShell SDK to read the retention label applied to files and the Update-MgDriveItemRetentionLabel to apply a replacement label (the same cmdlet can apply a default retention label to a folder). Another Practical365.com article looks at some of the business reasons why it might be necessary to replace retention labels on files, including a script to do the job.

Solving the Problem in Code

In my case, I took the script I wrote to report files stored in a OneDrive for Business account and ran it to identify what retention labels are applied to files in OneDrive. I then created a replacement retention label with the same retention period but with modified date as the test for retention. Finally, I copied the reporting script and modified the function that reads retention label data to look for the label (or labels) to replace and called the Update-MgDriveItemRetentionLabel cmdlet to assign the new label. You can download the script from the Office 365 for IT Pros GitHub repository.

Unfortunately, the gods of scripting (or APIs) intervened and the underlying Graph API developed a problem when updating labels. The same issue surfaced in the SharePoint PnP module. I’ve reported the problem and hope that Microsoft solve the issue soon. In the meantime, I updated the retention labels manually for several important files to make sure that Purview wouldn’t remove them. It’s disappointing that the script doesn’t work at present, but it will in the future.

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.