Artificial Intelligence and PowerShell for Tenant Administration – An Unlikely Couple?

I’ve been asked by a few people to comment about Lokka, the new creation of Merill Fernando, a program manager in the Microsoft Entra ID group. Lokka is a proof of concept exploring how the combination of AI Large Language Models (LLMs) and the Model Context Protocol (MCP) can bring value to Microsoft 365 administration. In this case, by generating Graph API queries in response to administrator prompts. For example, “How many user accounts belong to the marketing or sales departments.”

Merill’s a very inventive individual whose capacity to invent extends to his eye-catching tweet asking the question if Lokka is the end of PowerShell for Microsoft 365 administrators (Figure 1).

Helping Administrators with Simple Queries and Examples

Of course, the advent of a proof of concept like Lokka doesn’t mean that Microsoft 365 administrators suddenly need to lose all interest in PowerShell. AI tools can certainly be helpful in responding to queries that aren’t covered by the standard admin center GUI. They can also educate administrators by showing them how to use PowerShell to run Graph AI queries.

The Exchange Server 2007 product was the first Microsoft server to embrace PowerShell. One of the brainwaves in that product was how the Exchange Management Center (EMC) console displayed the PowerShell code it executed when it performed actions. Figure 2 shows how the EMC in Exchange Server 2007 displayed the code used to create a new mailbox.

Seeing the PowerShell code in action and being able to copy the commands for reuse helped administrators master basic PowerShell command for managing Exchange servers. Another example is how Merill’s Graph X-Ray tool gives administrators a glimpse into the Graph API requests run to perform some actions in the console.

Artificial Intelligence and PowerShell in the Microsoft 365 Admin Center

The Microsoft 365 admin center already has Copilot assistance that’s added automatically when a tenant buys some Copilot for Microsoft 365 licenses (Figure 3). The implementation is much like a Copilot Chat session where an administrator prompts Copilot for some information and receives a response containing instructions and possibly some PowerShell code. I imagine that the content used by Copilot is a restricted set of documentation, just like you can restrict a Copilot agent to reasoning over certain SharePoint and external web sites when it composes its responses.

The Importance of Training Material

There’s no doubt that we will see increasing use of AI to assist administrators with tasks as time progresses. The assistance will become more comprehensive, intelligent, and useful. However, the usefulness of any generative AI tool is bounded by the material used to create its LLMs. This means that the answers that an administrative agent can give, whether how-to instructions or PowerShell code snippets, depend on text scanned to build the LLM. If an answer exists to a question, the AI can respond. This includes incorrect answers because the LLM doesn’t know if content contained in source material is accurate. And if an answer isn’t available, the AI cannot respond without hallucinating. For example, Copilot has been known to include the names of PowerShell cmdlets that don’t exist in its responses.

The current set of AI tools we have don’t include insight or creativity. They can respond to known problems, but even so, responses are often based on whatever the most common answer is found in its source material. Those answers might be inefficient. Take the code suggested in Copilot’s response in Figure 3.

Get-MgUser | Where Department eq "Sales"

Several problems exist with the answer. First, the syntax is incorrect and won’t work because the piping to the Where-Object cmdlet is wrong (probably because Copilot absorbed an incorrect answer from some source). Second, the Department property is not retrieved by the Get-MgUser cmdlet unless explicitly requested.

Get-MgUser -Property Id, Displayname, Department | Where-Object {$_.Department -eq "Sales"}

Third, it’s always better to use a server-side filter to retrieve PowerShell objects. And in the case of user accounts, it’s also a good idea to filter out guest accounts.

Get-MgUser -Filter "Department eq 'Sales' and userType eq 'member'"

The takeaway is that generative AI can only be as good as the material used for its training. The current state of the art is such that AI can’t recognize when its output is incorrect.

PowerShell Still an Essential Tenant Management Skill

Even with the prospect of better, more complete, and more comprehensive AI tooling on the horizon, I still believe that Microsoft 365 administrators should take the time to acquire a working knowledge of PowerShell. For the foreseeable future, AI might well offer help to those who don’t even know how to start using PowerShell to manage a tenant.

Experience to date demonstrates that AI is unlikely to master the creativity that’s often needed to create something like a full-blown tenant licensing report, complete with costs anytime soon. Combining data from multiple sources to deliver a solution requires more ingenuity than running straightforward Graph requests. I await to be proven wrong that artificial intelligence and PowerShell can do more than perform straighforward, mundane tasks. In the interim, using GitHub Copilot to accelerate the development of PowerShell scripts might be the most productive way to use AI to improve Microsoft 365 automation.

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.