Create a Custom Maester Test with PowerShell and the Graph

I last wrote about the Maester tool in April 2024. At that time, Maester had just been released as a community-based framework for automated testing of a Microsoft 365 tenant against well-established frameworks like MITRE. Maester has come a long way since, and it was great to catch up with Merill Fernando and Thomas Naunheim at the TEC 2024 conference in Dallas to assess its current state.

Merill has great information-packed demos, even if they are delivered at dazzling speed. The ability to create custom Maester tests grabbed my attention this time around. Out-of-the-box, Maester comes with a set of tests based on Microsoft recommendations for Entra ID and another based on the Entra ID Security Config Advisor (EIDSCA), another community-driven project.

Regular Maester Tests

Great value can be derived from the results generated by a Maester using its default tests. You might not agree with some of the measured conditions. Last time round, my tenant failed 42 tests. The latest run failed 97. For example, my tenant failed test MS.AAD.3.7 because I don’t have a conditional access policy in place to require managed devices for authentication (Figure 1).

Insisting on managed devices is important in some contexts and less important in others. This underlines the need for Maester reports to be treated as a guideline rather than creating the absolute necessity to pass all tests. No one gets extra brownie points for achieving a perfect Maester score, and it might be the case that achieving such a state might be more painful than useful, which can sometimes be the case when seeking better tenant security. As I have noted in the past, cranking up the signin frequency interval enforced by a conditional access policy might seem like a good idea, but it’s not if forcing users to constantly reauthenticate gets in the way of real work.

In any case, the results of a Maester run creates a nice benchmark to measure a tenant against. How the tenant administrators decide to use that benchmark is entirely up to them.

Creating a Custom Maester Test

Coming back to the topic of this article, tenants can add custom tests for Maester to assess. Essentially, if you can grab some data like tenant settings or another type of value by running a Graph API request in PowerShell, a test can assess the data returned by the Graph and either pass or fail.

As an example, I wrote a test to check that the ability for end users to create new Microsoft 365 groups (and teams) is disabled. Allowing people to create new groups is a one-way ticket to team sprawl and excessive digital rot, so it seems like a good thing to test. The setting is in the Entra ID Groups policy and can be fetched and assessed as shown in the code below.

Describe “RAEntraConfig” -Tag “Privilege”, “Office 365 for IT Pros” {
It “OFFICE365.Test01: Check ‘Group creation should be blocked for non-authorized users” {

$Uri = “https://graph.microsoft.com/beta/settings”
$Result = Invoke-MgGraphRequest -Method Get -Uri $Uri

$GroupSettings = $Result.Value | Where-Object { $_.displayName -eq ‘Group.Unified’ }

$GroupCreationControl = $GroupSettings.values | Where-Object {$_.Name -eq ‘EnableGroupCreation’} | Select-Object -ExpandProperty Value
If ($GroupCreationControl -eq $false) {
$TestResult = $true
} else {
$TestResult = $false
}

If ($null -ne $TestResult) {
$TestResult | Should -Be $true -Because “User ability to create Microsoft 365 Groups is disabled.”
}

}
}

Only one test is present, but you can add multiple tests to the same file. Maester can process custom tests separately or include them in a full run. Figure 2 shows the output from the custom test. It’s functional and not as pretty or informative as Figure 1, but any administrator will know what the test measures.

If you develop a test that you think would be of interest to other organizations, you can create an issue in the Maester GitHub repository to explain the test and share the code.

Stretching and Expanding Maester

Best practice is a nebulous concept at best. In the cloud, things often change faster than the proponents of best practice can cope. Having a community-driven project like Maester available to assess your tenant is a good way to get a snapshot of how the tenant measures up against security frameworks. Being able to add your own custom Maester tests makes the tool so much better.

Learn how to exploit the data available to Microsoft 365 tenant administrators through the Office 365 for IT Pros eBook. We love figuring out how things work.