Using Defender XDR Portal to hunt for Kubernetes security issues
As we saw in previous article, the binary drift alert gives you information about where the activity happened like the object namespace, image, cluster, etc.
This might or might not be enough information for you to act. Say, if you want to identify “how” this drift came to be for example, did a user logged on to container and downloaded the said binary. To supplement the information provided by the alert we can then use Defender XDR portal (https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-defender-portal)
Harnessing the power of the Microsoft Ecosystem
If you are an E5 customer, your security teams most likely are very familiar with Advance Hunting on security.microsoft.com portal. Here we will extend that hunting capability to add context to your Kubernetes alerts. This is a huge time saver and cost advantage for you as you don’t need to teach your Red Team or SOC analysts Level 400 Kubernetes concepts. To jump start your Kubernetes hunting, you can leverage the developer knowledge of your Platform teams to provide most common Kubernetes actions like exec (access to a container), debug (access to node). The hunting team can then leverage these in the hunting queries in a data structure and format they already know using KQL.
Enhancing the hunt using XDR portal
Defender for Cloud sends incidents and alerts to the Defender portal (https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-security-center-defender-cloud#investigation-experience-in-the-microsoft-defender-portal).
Similarly, Microsoft Sentinel also send the data to Defender portal (https://learn.microsoft.com/en-us/azure/sentinel/microsoft-365-defender-sentinel-integration?toc=%2Fdefender-xdr%2Ftoc.json&bc=%2Fdefender-xdr%2Fbreadcrumb%2Ftoc.json&tabs=defender-portal)
Since your SOC and Red Teams are already proficient in using XDR portal, Kubernetes hunts can now easily become part of their playbook.
By looking at the Alerts and Incidents in the XDR portal you can see the birds eye view of what, who, and where. This will equip you to further narrow down your search in Hunting Queries.
Fig. Attack Story for Binary Drift
The Evidence and Response tab shows all the relevant evidence. Most likely, you will create your hunting queries using these fields.
Fig. Kubernetes objects related to the incident
This integration allows us to run Advance Hunting queries using CloudAuditEvents table that has Defender for Cloud data.
The query below looks for exec in a Pod named ubuntu (assumption here being that this pod is also running a container ubuntu where the drift happened and alert generated)
CloudAuditEvents
| where DataSource == “Azure Kubernetes Service”
| where OperationName == “create”
| where RawEventData.ObjectRef.resource == “pods” and RawEventData.ResponseStatus.code == 101
| where RawEventData.ObjectRef.subresource == “exec”
| where RawEventData.ResponseStatus.code == 101
| extend RequestURI = tostring(RawEventData.RequestURI)
| extend PodName = tostring(RawEventData.ObjectRef.name)
| extend PodNamespace = tostring(RawEventData.ObjectRef.namespace)
| extend Username = tostring(RawEventData.User.username)
| where PodName startswith “ubuntu”
| extend Commands = extract_all(@”command=([^&]*)”, RequestURI)
| extend ParsedCommand = url_decode(strcat_array(Commands, ” “))
| project Timestamp, AzureResourceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, Username, ParsedCommand
The query above will produce the following, here you can see who executed the command, when was the command executed and many other fields that are helpful for the context
Fig. Query output
Another scenario you would want to look for are activities performed by a managed identity that might have been related to an alert.
CloudAuditEvents
| where RawEventData.principaloid == <managed identity id>
This query will provide you the actions that this managed identity performed. These might be enumerating key vaults, storage accounts, etc. As before by reviewing the output such as timestamp, geo etc. you can make a determination if this action is malicious.
You can then look at the activities that were performed during this timeframe, like so,
CloudAuditEvents
| where AzureResourceId == <AKS Cluster ID>
| where TimeGenerated > datetime(<start time>) and TimeGenerated < datetime(<end time>)
| where OperationName == “create”
| where UserAgent has “kubectl”
If the attacked conducted any “exec” during this time frame you will be able to see it under “ObjectRef” and “RequestURI” will show you the exact command that was executed.
Important to note that Query results are presented in your local time zone as per settings. Kusto filters, however, work in UTC.
Automating the process and Extending the out of the box detections
Now that you have queries defined that provide you additional context behind the drifts, you can convert them into a custom detection rule that you can run at a defined frequency.
For example, say you want to get alerted on privileged pods:
These pods run with an elevated set of privileges required to do their job, but that could conceivably be used as a jumping off point to gain escalated privileges.
Note: You can also use Azure Policy built-in definitions to prevent this:
CloudAuditEvents
| where Timestamp > ago(1d)
| where DataSource == “Azure Kubernetes Service”
| where OperationName == “create”
| where RawEventData.ObjectRef.resource == “pods” and isnull(RawEventData.ObjectRef.subresource)
| where RawEventData.ResponseStatus.code startswith “20”
| extend PodName = RawEventData.RequestObject.metadata.name
| extend PodNamespace = RawEventData.ObjectRef.namespace
| mv-expand Container = RawEventData.RequestObject.spec.containers
| extend ContainerName = Container.name
| where Container.securityContext.privileged == “true”
| extend Username = RawEventData.User.username
| extend DeviceId = RawEventData.AzureResourceId
| project Timestamp, ReportId, DeviceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, ContainerName, Username
Note that we will be using “Timestamp, ReportId, DeviceId” to turn the query into a custom detection rule. This again can be a starter pattern for your rules.
To create a custom detection/rule you will need to include the columns mentioned under https://learn.microsoft.com/en-us/defender-xdr/custom-detection-rules#required-columns-in-the-query-results
You can now easily convert this query into a custom detection rule
Fig. Creating custom detection rule
Complete the fields that will show up in Alerts
Fig. Privileged pod alert details
Select the impacted object in our case the DeviceId (aka AzureResourceId)
Fig. Identity impacted by the alert
Choose the action
Fig. Action to be taken upon alert
Lastly, create your alert
Fig. Alert created
Once created the rule will appear in your custom detection rules in XDR portal
Fig. Custom detection rules
Call to action
As you saw in this two-part series that unlike many third party solutions, you can maximize your existing investment in Microsoft’s security ecosystem to conduct deeper hunts using the tools that you already know.
Our suggestion is to:
Engage your Platform Engineering team to identify most risky Kubernetes actions that are relevant to your environment. For example, if you are using distroless images exec won’t make much sense
Provide your SOC and Red Team the XDR Portal and Defender for Cloud integration documentation so they can start leveraging the starter queries and customize to your environment
Review your third-party Kubernetes security solutions to see if the value you are getting for features that are not available in Native is worth the investment. In our discussions with dozens of customers, Native proves to be most direct, cost efficient, easier to use, and manage in the longer term
Moreover, if you have not thought about going Native-First in your security strategy reevaluate the choice and talk to your Security Seller about the latest roadmap of Native Security Services
Remember, a lot of on-premises security challenges were because of reliance on “best of breed”. That strategy does not translate well in Cloud. Share the following Native First security resources with your decision makers
https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/native-first-cloud-security-approach/ba-p/4102367
https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/unleashing-the-power-of-microsoft-defender-for-cloud-unique/ba-p/4102392
Microsoft Tech Community – Latest Blogs –Read More
