What’s new in SIEM Migration?

The process of moving from Splunk to Microsoft Sentinel via the SIEM Migration experience has been enhanced with three key additions that help customers get more context aware translations of their detections from Splunk to Sentinel. These features let customers provide more contextual details about their Splunk environment & usage to the Microsoft Sentinel SIEM Migration translation engine so it can account for them when converting the detections from SPL to KQL. These are:

Schema Mapping
Support for Splunk Macros in translation
Support for Splunk Lookups in translation

Let talk about how these can make life easier when migrating to Microsoft Sentinel via the SIEM Migration experience:

Schema Mappings

How does it help?

Most traditional translation tools only factor in Grammar translations when translating from one query language to another. More precisely addressing the “how” in the queries – How are these queries structured? How are operational and computational logics defined? Among other things.

The What is often lost is translation. “What data sources are being queried”? “What do these data sources really map to in the target SIEM”?

The “what” is often environmental customer context that needs to be accounted for in translation to ensure that grammar translations are applied on the right sources.

The Approach

Schema mappings in the SIEM Migration experience allows you to precisely define how Splunk sources (indexes, data models, etc.) map to Microsoft Sentinel tables within the new “Schema mapping” section of the UI Experience. This feature provides the flexibility and customization to ensure that your data is aligned with your migration needs. On uploading the Splunk export, the system extracts all the sources from the SPL queries. Known sources such as Splunk CIM schemas & data models are auto mapped to ASIM Schemas as applicable. The other custom sources queried in the detections are listed without being mapped and these will require manual mapping with existing Microsoft Sentinel/Azure Log Analytics tables. All mappings can then be reviewed, modified or new sources added. Mapping schemas is hierarchical, i.e., the Splunk sources map 1-1 with Sentinel tables in addition to the fields within these sources.

The best part? The manual changes to schema mapping are saved per workspace so that you do not have to repeat it again.

Step-by-Step usage guidance

To leverage Schema Mappings,

Navigate to the SIEM Migration experience from the Microsoft Sentinel Content Hub.
Review prerequisites and click “Next: Upload File”.
Export the inventory of Splunk detections by following the instructions on the screen and once exported, upload to Sentinel. Click “Next: Schema Mapping (Preview)”
Review the Splunk data sources identified from the export process. To review the field mappings within a data source, select the Splunk source which will open a side panel on the right that has the field mappings.

Review, Modify, Add schema mappings

Data Source Mappings: To edit the Sentinel table that the Splunk source is mapped to select the Sentinel Table from the Sentinel Table dropdown.
Field Mappings: To edit field mappings, look for the Splunk field on the left that you wish to change the mapping for and then for this Splunk field, select the corresponding Sentinel field from the dropdown.

Add new Schema Mappings: In a scenario where you do not find the Splunk source identified & listed in the list of data sources, click on “+ Add source”. Now in the right-side panel, continue adding the name the of your Splunk data source and select a Sentinel table from the dropdown menu. Click “+Add mapping” to continue adding field mappings by entering the Splunk field name manually on the left and selecting the corresponding Sentinel field name on the right.

Once the changes have been completed, click on “Save Changes”. Note that the Mapping state now changes to “Manually Mapped”.

Once the Schema Mappings are complete, the changes made are taken into account when the SPL saved searches are translated to KQL queries.

Translation support for Splunk Lookups

Splunk Lookups, like Sentinel Watchlists are lists with field-value combinations that can be queried/correlated against ingested data. The SIEM Migration experience addresses the translation & use of Splunk lookups in SPL queries (in Splunk detections) to Sentinel Watchlists’ use in the KQL queries generated.

Note: Sentinel Watchlists must be created as a pre-requisite to allow mapping these Sentinel Watchlists with Splunk Lookups when you start migrating.

The Approach

Splunk lookups as a complete data set are defined and are available outside the bounds of the SPL query and the SPL query only references the lookups invoking it with the “lookup”, “inputlookup” and/or “outputlookup” keywords. The translation support is only available for the “lookup” & “inputlookup” keywords where lookup data can be queried/correlated against. The “outputlookup” operation – where data is written to a lookup – is not supported in translation but can be achieved by defining an Automation Rule in Microsoft Sentinel.

For translating the invocation of lookups, SIEM Migration’s translation engine uses the “_GetWatchlist()” KQL function to allow mapping to the correct Sentinel watchlist, supplemented in operation by other KQL functions to translate the complete logic.

Step-by-Step usage guidance

To ensure the correct Splunk Lookup à Sentinel Watchlist mapping, its important for the SIEM Migration experience to have this mapping context. The experience now allows for customers to be able to map their Splunk lookups (automatically identified from the Splunk queries uploaded) to Sentinel Watchlists (Created outside the experience as a pre-requisite).

Follow the guidance here to create Sentinel Watchlists.

Once the Watchlists are created, follow the guidance below to map these Sentinel Watchlists to Splunk lookups:

Navigate to the SIEM Migration experience from the Microsoft Sentinel Content Hub.
Review prerequisites and click “Next: Upload File”.
Export the inventory of Splunk detections by following the instructions on the screen and once exported, upload to Sentinel. Click “Next: Schema Mapping (Preview)”.
Click on the “Lookups” tab and start reviewing/mapping the lookups.
To add field mappings, click on the Splunk Lookup that needs to be mapped and on the right-side panel that opens, select the corresponding Sentinel Watchlist on the right-hand side.

Once the Sentinel Watchlist is selected, the field mappings can be completed by selecting the Watchlist field from the field dropdown corresponding to the Lookup field on the left.

On completing the review, click “Save Changes”. Note that the Mapping state now changes to “Manually Mapped”.

Once all Splunk lookups have been reviewed, click on “Next: Configure Rules” to start translations to KQL.

NOTE: When a Splunk lookup does not have a corresponding Sentinel Watchlist mapped, the translation engine keeps the same name for both the Sentinel Watchlist and its fields as the Splunk lookup and fields.

Translation support for Splunk Macros

How does this help?

A core tenet of developers is automation and functionality reuse. Macros are integral for quick development, but every architect silently curses these “shortcuts” when having to migrate to a different tech stack.
When upgrading the SIEM migration experience the team thought: What if someone told the architect “Hey, we got this covered”. All (SPL) detection queries will seamlessly be expanded by making inline replacements of the macro references by the respective macro definitions and passed on to the translation engine to ensure the core detection logic stays retained when the language translations happens.

The Approach

To enable this Macro expansion the experience needs more context and data. Be to context for the data field mapping or the Splunk code associated with the macros. This enrichment is done via the initial file query and uploader which now has a richer query to pull the necessary information – The metadata of the detections and in addition, all macro definitions. This extra information helps identify and ensure all pieces of the puzzle are in the right place before translation.

Step-By-Step Usage Guidance  

Do not worry, there are no extra steps here. 🙂

The experience: “Copy the query & run it on Splunk” to obtain the import file necessary for migration remains the same. As mentioned earlier the query has been enhanced to get a broader context with an updated format.

There are no extra touchpoints. The migration experience will take care of the rest and show you the expanded source query with macros references replaced in-line with the respective definitions in the “Configure Rules” tab.

​Microsoft Tech Community – Latest Blogs –Read More