Checking the Effectiveness of a Transport Rule to Block Spammy Email

No Point in Having Transport Rules if They’re Not Doing the Job

A few weeks ago, I wrote about using the Microsoft Graph to extract a list of spammy domains from messages in the Junk Email folder in user mailboxes. The list is then used as input to a transport rule so that any email arriving from spammy domains end up in the quarantine instead of user mailboxes. Apart from bothering users with system messages notifying them about quarantined email (Figure 1), the technique worked nicely. A scheduled Azure Automation runbook picks up new spammy domains as they appear and add them to the transport rule.

A quarantine message sent to a user after the transport rule intercepted email from a spammy domain. — Figure 1: A quarantine message sent to a user after the transport rule intercepted email from a spammy domain

In fact, the transport rule worked so well that I decided to stop bothering users with quarantine notifications and updated the transport rule to delete messages from the spammy domains without saying anything to the sender. Now the only quarantine notifications that appear in user inboxes are for messages not received from the spammy domains.

Check and Verify

Then I had a nasty feeling that perhaps the transport rule was working too well and that some messages might have been dropped in error. My gut says that everything’s OK, but some verification would improve my confidence. One way of checking is to search user mailboxes to see if any messages from the banned domains have arrived since the rule started to operate, but that seems a little intrusive. Another method is needed.

To seek confirmation, I ran the Get-MailDetailTransportRuleReport cmdlet to fetch data about the outcomes of mail transport rules for the last 10 days (like message trace logs, 10 days is the maximum Exchange Online keeps this data available for immediate interrogation).

The transport rule report can filter on actions taken by rules. The set of supported actions can be found by running the Get-MailFilterListReport cmdlet:

Get-MailFilterListReport -SelectionTarget Actions

SelectionTarget Display                    Value
--------------- -------                    -----
Actions         AddBccRecipient            AddBccRecipient
Actions         AddCcRecipient             AddCcRecipient
Actions         AddManagerAsRecipient      AddManagerAsRecipient
Actions         AddToRecipient             AddToRecipient
Actions         Allow                      Allow
Actions         AllowRedirect              AllowRedirect
Actions         ApplyClassification        ApplyClassification
Actions         ApplyHtmlDisclaimer        ApplyHtmlDisclaimer
Actions         BlockAccess                BlockAccess
etc.

Summarizing the Result of the Transport Rule Block

The transport rule deletes spammy messages quietly, so the right action to use is “DeleteMessage.” Here’s the command I used to find and summarize the events:

[array]$Data = Get-MailDetailTransportRuleReport -Direction inbound -Action DeleteMessage
$Data | Group-Object 'TransportRule' -NoElement | Format-Table Name, Count

Name                                                        Count
----                                                        -----
Block email to administrators from outside the organization    65
Quarantine Traffic from Junk Email Domains                     80

The information contained in a transport rule report event looks like this:

Date              : 02/11/2025 10:42:06
Message ID        : <89a2ec2634a1de9811917cb90a02fe3c@9pz.org>
Message Trace ID  : e07c2b05-ea8a-429d-e933-08de19fc77b1
Domain            : office365itpros.com
Subject           : Re: We make packing bags in China
Message Size      : 31720
Direction         : Inbound
Sender Address    : 1232@9pz.org
Recipient Address : james.a.abrahams@office365itpros.com
Event Type        : TransportRuleActionHits
Action            : DeleteMessage
Transport Rule    : Quarantine Traffic from Junk Email Domains

We can extract and summarize the set of events for the block from spam domain transport rule to discover who would have received the spam had the block not existed:

$TransportRule = 'Quarantine Traffic from Junk Email Domains'
[array]$MData = $Data | Where-Object {$_.'TransportRule' -eq  $TransportRule}

$MData | Group-Object -NoElement RecipientAddress | Sort-Object Count -Descending | Format-Table Name, Count

Name                                        Count
----                                        -----
tony.redmond@office365itpros.com               27
james.a.abrahams@office365itpros.com            7
andy.ruth@office365itpros.com                   5
lotte.vetler@office365itpros.com                5
sean.landy@office365itpros.com                  5
brian.weakliam@office365itpros.com              4
jeff.atkinson@office365itpros.com               4
office365.book.comments@office365itpros.com     4
terry.hegarty@office365itpros.com               4
contact@office365itpros.com                     3
o365itprosrenewals@office365itpros.com          2
marty.king@office365itpros.com                  1
office365book@office365itpros.com               1

Most of these addresses are for dummy accounts or shared mailboxes used for testing. Spammers learn of the addresses by harvesting information from web articles, but they gain nothing by sending messages to the accounts. However, I don’t like cleaning out the spam before taking screenshots for articles, so I’m delighted to see that the transport rule is doing its job.

Finding how many messages came from specific spammy domains takes a little extra effort. Here’s some code to report the number of messages from each domain processed by the transport rule:

[array]$SpammyDomains = (Get-TransportRule -Identity $TransportRule).SenderDomainIs
ForEach ($Domain in $SpammyDomains) {
  $CheckDomain = "*"+$Domain+"*"
  [array]$SpamRecords = $MData | Where-Object {$_.SenderAddress -like $CheckDomain} 
  If ($SpamRecords.Count -gt 0) {
    Write-Host ("Domain {0} sent {1} spammy messages" -f $Domain, $SpamRecords.Count)
  }
}

Domain 9pz.org sent 7 spammy messages
Domain conarh02.com sent 6 spammy messages
Domain conarh03.com.mx sent 25 spammy messages
Domain conarh07.com sent 6 spammy messages
Domain spamareus.com sent 11 spammy messages
Domain fgwexvb.asia sent 4 spammy messages
Domain forkcontact.com sent 1 spammy messages
Domain moonpig.com sent 4 spammy messages
Domain annoyingcompany.co.uk sent 3 spammy messages
Domain tradetpt12.com sent 5 spammy messages
Domain tradetpxs10.com sent 4 spammy messages
Domain getportant.com sent 4 spammy messages

Some Manipulation Required

The lesson here is that a method exists to validate the effectiveness of transport rules. Extracting the desired information from the transport rule report might take some manipulation with PowerShell, but it’s certainly a way to check how well your rules work.

Learn about managing Exchange Online and the rest of the Microsoft 365 ecosystem by subscribing to the Office 365 for IT Pros eBook. Use our experience to understand what’s important and how best to protect your tenant.

Cart

Cart