Tag Archives: microsoft
Migrating SLURM Job Accounting from Azure Database for MariaDB to MySQL Flexible Server
Overview
Azure CycleCloud (CC) is an enterprise-friendly tool for orchestrating and managing High-Performance Computing (HPC) environments on Azure. With CycleCloud, users can provision infrastructure for HPC systems, deploy familiar HPC schedulers, deploy/mount filesystems and automatically scale the infrastructure to run jobs efficiently at any scale. Azure CycleCloud is targeted at HPC administrators and users who want to deploy an HPC environment with a specific scheduler in mind.
One of the supported schedulers is SLURM, which stands for Simple Linux Utility for Resource Management. SLURM is an open source, scalable and fault-tolerant cluster management and job scheduling system for Linux clusters of any size. SLURM can allocate resources to users for a duration of time, manage workloads, provide accounting and monitoring, and support parallel and distributed computing applications.
SLURM job accounting is a feature that allows users to collect and report various metrics about the jobs that run on the cluster. SLURM job accounting can help users optimize their resource utilization, monitor their quotas, and track their costs. To enable job accounting, you need to configure and run a SLURM database daemon (slurmdbd) that can store the job information in a database.
One of the common choices for the database backend for SLURM job accounting is Azure Database for MariaDB. However, Azure Database for MariaDB will be retired on September 19, 2025, and users are encouraged to migrate their data and applications to Azure Database for MySQL Flexible Server.
In this blog, we will walk you through the steps to migrate your SLURM job accounting data from Azure Database for MariaDB to Azure Database for MySQL Flexible Server in Azure CycleCloud.
Requirements/Versions
CycleCloud Server (CC version used is 8.6.2)
Slurm Cluster
CycleCloud project used is 3.0.5
Slurm version used is 23.02.6-1
A source instance of Azure Database for MariaDB
A target instance of Azure Database for MySQL Flexible Server
A Linux VM with access to both the MariaDB and MySQL instances
We used the CycleCloud VM
Running on Alma Linux 8.7
Migration Procedure
Install required packages
Back up your SLURM Accounting DB in Azure Database for MariaDB
Restore the backup to your Azure Database for MySQL Flexible Server
Create SLURM user in Azure Database for MySQL and grant privileges
Update your CycleCloud cluster configuration for SLURM Job Accounting
To perform the migration, you will need a VM that can connect to both the source and target databases. In this blog post, we will use the CycleCloud VM as an example, but you can use any VM that meets this requirement.
Step 1: Install required packages
MySQL Shell (mysqlsh) is an advanced command-line client for MySQL that supports various modes of operation, such as SQL, JavaScript, and Python, and enables interactive and batch execution of queries and scripts. This utility will be used to perform the transfer of the SLURM job accounting DB from MariaDB to MySQL, and requires the installation of the following additional packages:
mysql
mysql-shell
To install the required packages, you will need to run the following commands on the migration VM:
sudo yum install -y msql
wget https://dev.mysql.com/get/mysql84-community-release-el8-1.noarch.rpm /tmp
sudo yum localinstall /tmp/mysql84-community-release-el8-1.noarch.rpm
sudo yum install -y mysql-shell
Step 2: Back up your SLURM Accounting DB in Azure Database for MariaDB
At this point, it is recommended for your SLURM cluster to be terminated to ensure that the accounting DB is no longer being updated.
Connect to MariaDB and check the size of the SLURM accounting DB:
mariadbname=jmslurmmariadbeus #update with your specific db name
mariadbusername=themorey #update with your specific db user name
mysqlsh –uri ${mariadbusername}%40${mariadbname}@${mariadbname}.mariadb.database.azure.com:3306
OPTIONAL – to check the size of the DB:
SELECT table_schema AS “Database”,
ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS “Size (MB)”
FROM information_schema.TABLES
GROUP BY table_schema;
Sample Output:
Check free space on VM to determine where backup can be stored:
df -h
The output of the df -h command shows that we have enough space for us to dump our backup on our local disk:
Create a directory for your MariaDB backup. Note, the target directory for the backup must be empty:
sudo mkdir -p /backup/mysql/mariadb_backup/
sudo chown -R $(whoami) /backup
Log back in to MariaDB using mysqlsh:
mariadbname=jmslurmmariadbeus #update with your specific db name
mariadbusername=themorey #update with your specific db user name
mysqlsh –uri ${mariadbusername}%40${mariadbname}@${mariadbname}.mariadb.database.azure.com:3306
Switch to Javascript mode on the mysql shell:
js
Run the dumpUtil to take a full backup:
util.dumpInstance(“/backup/mysqlsh/mariadb_backup”,{threads: 16, showProgress: true, users: false})
Sample Output:
After the dumpUtil command is complete, you can exit out of the MySQL shell ( q; ) and view the backup on your VM by listing the contents of the backup directory. The backup consists of a series of JSON files that contain the schema and data of the instance.
Step 3: Restore the backup to your Azure Database for MySQL Flexible Server
Now that you have a backup of your accounting data from MariaDB, you can restore it to your Azure Database for MySQL Flexible Server.
To do this, first login to your Azure Database for MySQL Flexible Server using MySQL shell. Note, the syntax to establish the connection is slightly different than the one used to connect to MariaDB:
mysqldbname=jmmysqlfrommariadb #update with your specific db name
mysqldbusername=themorey #update with your specific db user name
mysqlsh –uri ${mysqldbusername}@${mariadbname}.mysqldb.database.azure.com:3306
List the databases in your MySQL server to confirm that the slurm_acct_db does not exist:
show databases;
Sample Output:
Switch to javascript mode and run the loadDump utility to import the MariaDB dump files:
js
util.loadDump(“/backup/mysqlsh/mariadb_backup”, {threads: 16, showProgress: true, ignoreVersion: true})
Sample Output:
Step 4: Create SLURM user in Azure Database for MySQL and grant privileges
When we used the util.loadDump() function to restore the data from the MariaDB backup, we only restored the SLURM accounting database and not the user accounts. This means that the SLURM user account that was used to access the database in MariaDB does not exist in the Azure Database for MySQL instance.
To fix this, we need to switch back to SQL mode, create a new user account with the same name and password as the SLURM user in MariaDB, and grant privileges:
sql
create user slurm@’%’;
ALTER USER slurm IDENTIFIED BY ‘P@ssw0rd!@#’;
grant usage on *.* to slurm@’%’;
grant all privileges on slurm_acct_db.* to slurm@’%’;
flush privileges;
SHOW GRANTS FOR ‘slurm’@’%’;
Sample Output:
Step 5: Update your CycleCloud cluster configuration for Slurm Job Accounting
After creating the user and granting the privileges, we are now ready to connect our cluster to the new MySQL server instance. To do this, we need to navigate to the advanced settings of the CycleCloud SLURM cluster and update following details:
Slurm DBD URL = URL for MySQL Server instance
Slurm DBD User = SLURM accounting DB user
Slurm DBD Password = SLURM accounting DB password
SSL Cert URL = https://dl.cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem
Sample Output:
After updating the required details for your SLURM Cluster, save the configuration and start the cluster.
Once the cluster is operational, run “sacct” to verify that you can view the historical information about the jobs that have been ran in the cluster before the migration took place:
NOTE: Slurm accounting command (sacct) defaults to the same day. You may need to expand the search criteria to see jobs older than current day. For example, “sacct -S 060124“ will show jobs starting from 6/1/2024 until the current day.
SUMMARY
The impending retirement date for MariaDB does not require you to abandon your Slurm accounting history. An Azure MySQL Flexible Server can be the solution moving forward while also loading the historical data from MariaDB.
REFERENCES
Migrating from Azure Database for MariaDB to Azure Database for MySQL – Microsoft Community Hub
Microsoft Tech Community – Latest Blogs –Read More
From Paper to Pixels: Azure AI in Historical Document Digitization and Translation
v:* {behavior:url(#default#VML);}
o:* {behavior:url(#default#VML);}
w:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
Heather MacKinnon-Miller
Normal
Heather MacKinnon-Miller
2
5923
2024-07-16T17:12:00Z
2024-07-16T17:12:00Z
5
1127
6427
53
15
7539
16.00
Clean
Clean
false
false
false
false
EN-US
X-NONE
<w:LidThemeComplexscript>X-NONE</w:LidThemeComplexscript>
For 64 years, a stack of letters lay unread in my grandfather’s trunk. These letters, written by relatives of my great-grandfather who immigrated from Poland in 1906, represent the last remnants of Polish language in my family. Over a decade ago, I promised my mother I would have them translated, but life got in the way. Recently, inspired by the success I’ve seen using OCR, digitization, and translation with various customer documents, I decided to tackle this personal project myself using the Microsoft AI services I am familiar with.
Iterating through Process and Technology
I gathered the letters, took images of each page with my phone, and uploaded them to Azure Storage. Sending the image files directly to Azure OpenAI’s GPT-4o model resulted in a confusing mix of English and Polish, so I converted them to .pdf files and took advantage of Azure’s Document Intelligence Service, specifically the Read model, to identify the language and extract the text. The Document Analysis feature can recognize different styles, including handwritten text, and identify the language, reassuring me that the text would be extracted accurately. I used the Document Intelligence code samples repo to get started, and verified that the text was handwritten and correctly identified as Polish (P1). The best workflow involved pre-processing the documents with Azure Document Intelligence’s OCR capabilities, then passing the extracted Polish text to the GPT-4 model for translation into English. I used this git repo as a quick start, changed the model endpoint, and processed the letters in chunks of two pages per call to the model. This combination provided a reliable method to digitize and translate the documents effectively. Success! I had a general idea of the contents of the letters, so once my initial attempt proved successful, I felt motivated to continue. Within about an hour, I had processed all the letters and was ready to validate the results.
The English translation of the above letter excerpt:
“Dear Joseph, I ask you to please ask your brothers Kazimierz and Jan to write to me. I deeply regret the loss of Piotr who died on the front in France. Tell me how your father is doing and if he is healthy. My dearest nephews, please come to Poland to visit us and improve our lives. If you cannot come, please send my family a parcel of clothes. We need both winter and summer clothing, as well as footwear—shoes and clogs. From your abundance, please collect some money and send it to us, poor farmers. I wish you happiness and success in the future. I send my warmest greetings to Joseph, Kazimierz, Jan, your father, and all our relatives.”
Although the translations seemed accurate, I needed confirmation from a native speaker before sharing with family. Luckily, a colleague who speaks Polish fluently offered to review a sample page. He confirmed that, while there were some nuances and minor errors in the translation, the overall message and sentiment were accurate. For example, “cannot replace me, an old man, in hard work” was interpreted by GPT-4o as “too young to replace me in heavy labor.” While this analysis method was sufficient for my small project, more sophisticated metrics would be best used at scale. While this small personal project used only a subset of Azure services and was not intended for public use, this pattern could easily be integrated into a search service, such as Azure AI Search for discovery, or broadened to a “Chat with Your Data” pattern, for scalability.
Extending to cursive handwriting
After completing this project, I became curious about applying the same process to more complex documents, such as those written in cursive. Since cursive has not been taught as part of the common core at most U.S. schools since 2010, there is a real risk that future generations may lose the ability to read and preserve these historical documents. Institutions like the Smithsonian and the National Archives rely on volunteers to transcribe and preserve these records, but the pace is insufficient to capture written history before these skills disappear. The time has never been better to introduce AI for preservation of handwritten history.
To test this idea, I found a few family recipes written in cursive. This time, I uploaded image files of the recipes directly to GPT-4o. The results were impressive and did not require OCR as an initial step. Here’s a sample recipe from my grandmother:
And here are the results from GPT-4o:
Sandies
Ingredients:
1 cup butter
1/3 cup granulated sugar
2 tablespoons water
2 teaspoons vanilla
2 cups flour
1 cup chopped nuts
Instructions:
Cream butter and sugar.
Add water and vanilla.
Then add flour and nuts.
Chill 4 hours.
Roll in cookie sheet, shape into fingers.
Bake at 325°F for 20 minutes.
Cool and roll in powdered sugar.
Amazingly, the model was even able to identify “cookie sheet,” which was written in very small superscript text.
Prompt nuances matter
While I didn’t have to do much on the prompt engineering side, I realized how small changes in prompts can affect the quality of the result. In the case of the Polish letters, I initially prompted the GPT model to “translate this text from Polish to English”. I iterated a bit and found “translate these family letters from Polish to English” made the result a bit more readable and self-corrected on some misspellings. In the case of the recipes, I specifically prompted the GPT model to “read this family recipe” rather than just read what was in the image. This resulted in not only a very accurate result, but the model output separated the ingredients and the instructions without being explicitly written in the original recipe.
The journey from paper to pixels has never been more accessible or efficient, thanks to Azure Document Intelligence and Azure OpenAI. These powerful tools have proven their capability in digitizing and translating handwritten historical documents, preserving invaluable cultural and personal histories. My experience with translating my great-grandfather’s letters and digitizing family recipes demonstrates the transformative potential of these technologies. By leveraging Azure’s AI tools, we can ensure that the stories and knowledge contained in historical documents are not lost to time but are instead accessible to future generations.
Microsoft Tech Community – Latest Blogs –Read More
ACTION REQUIRED: Users of HDC API may need to update code for deprecated API by August 28th
On August 28th, 2024, Hardware Dev Center APIs will change the type of blob urls that are returned in various API calls from service sas urls to user delegated sas urls. Partners using HDC Hardware dashboard API to automate driver submission process may be affected. The specific area affected is the upload and downloads through the download links in the submission response. All other aspects of the submission process that can be accessed through the API are not affected. Partners are recommended to update to latest version of the libraries they use to upload and download from Azure Blobs during the driver submission process.
Partners using C# are encouraged to move to Azure.Storage.Blobs library to ensure that driver submissions are not interrupted. Any partner already managing blob interactions through Azure.Storage.Blobs library are not affected by this change. Please reference this documentation that covers how to upload to a blob with .NET: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-upload.
If you experience any issues, please open a support ticket with HDC support. For details about how to contact the support team, visit https://aka.ms/GetHardwareSupport .
Microsoft Tech Community – Latest Blogs –Read More
Microsoft Power BI and Microsoft Defender for Cloud
Introduction
As cloud environments grow more complex and threats increase, organizations need robust tools to monitor, analyze, and respond to security issues effectively. Microsoft Defender for Cloud (MDC) offers robust security management, but to unlock its full potential, organizations need powerful visualization and analysis tools.
While Azure Workbooks provide valuable visualizations for MDC data, integrating Microsoft Power BI offers an enhanced approach to data analysis and visualization. Power BI’s advanced features, such as customizable dashboards, interactive elements, and seamless integration with various data sources, make it ideal for enhancing the value derived from MDC data.
This article is the first in a series of correlated blogs that will explore scenarios and applicability in depth. As an introduction to the series, this article provides the foundation on how to start leveraging Power BI to report and dashboard MDC insights.
Benefits of Using Power BI with Microsoft Defender for Cloud
Advanced Data Visualization: Power BI provides a wide array of visualization options, allowing security teams to create highly customized and visually rich dashboards that effectively communicate insights to different stakeholders.
Enhanced Data Analysis: Power BI’s robust analytical tools, including DAX (Data Analysis Expressions) and built-in AI capabilities, enable security teams to perform complex data analysis and uncover deeper insights.
Seamless Integration: Power BI integrates with various data sources, including Azure Resource Graph, allowing you to consolidate data from multiple platforms into a single, unified view.
Collaborative Features: Power BI facilitates collaboration by enabling teams to share dashboards and reports easily, with role-based access controls ensuring data security.
Ease of Use: Power BI’s intuitive drag-and-drop functionality makes it simple for users to create and customize visualizations without extensive technical knowledge, making it accessible to users of all skill levels.
Step-by-Step Guide to Integrating MDC Data into Power BI
To integrate MDC data into Power BI, follow these steps:
Step 1: Set Up Power BI and Azure Resource Graph
Install Power BI Desktop: Download Power BI Desktop.
Enable Azure Resource Graph: Ensure that you have the necessary permissions to access Azure Resource Graph.
Step 2: Connect Power BI to Azure Resource Graph
Open Power BI Desktop: Launch Power BI Desktop on your computer.
Get Data: Click on Get Data on the Home tab.
Select Azure Resource Graph: In the Get Data window, search for Azure Resource Graph and select it.
Connect: Click Connect and sign in with your Azure credentials.
Step 3: Load MDC Data into Power BI
Once you’ve connected Power BI to Azure Resource Graph, you can begin loading MDC data.
Here, we’ll provide a few example queries to retrieve data for recommendations, attack paths, secure scores, and governance. Note that these are just a few examples; you can retrieve any data available in Azure Resource Graph (ARG) according to your needs.
Enter ARG Queries: Write or paste the ARG KQL query and click OK
Load Data: After entering the queries, click Load to import the data into Power BI. The imported data will appear in the Fields pane, ready for you to create visualizations and reports.
Use the following ARG queries to pull the main MDC data points:
Recommendations (by risk):
This query retrieves security recommendations by risk from MDC, allowing you to analyze assessments and identify areas that need attention.
securityresources
| where type =~ “microsoft.security/assessments”
| extend assessmentType = iff(type == “microsoft.security/assessments”, tostring(properties.metadata.assessmentType), dynamic(null))
| where (type == “microsoft.security/assessments” and (assessmentType in~ (“BuiltIn”, “CustomerManaged”)))
| extend assessmentTypeSkimmed = iff(type == “microsoft.security/assessments”, case(
tostring(properties.metadata.assessmentType) == “BuiltIn”, “BuiltIn”,
tostring(properties.metadata.assessmentType) == “BuiltInPolicy”, “BuiltIn”,
tostring(properties.metadata.assessmentType) == “CustomPolicy”, “Custom”,
tostring(properties.metadata.assessmentType) == “CustomerManaged”, “Custom”,
tostring(properties.metadata.assessmentType) == “ManualCustomPolicy”, “Custom”,
tostring(properties.metadata.assessmentType) == “ManualBuiltInPolicy”, “BuiltIn”,
dynamic(null)
), dynamic(null))
| extend assessmentId = tolower(id)
| extend assessmentKey = iff(type == “microsoft.security/assessments”, name, dynamic(null))
| extend source = iff(type == “microsoft.security/assessments”, trim(‘ ‘, tolower(tostring(properties.resourceDetails.Source))), dynamic(null))
| extend statusCode = iff(type == “microsoft.security/assessments”, tostring(properties.status.code), dynamic(null))
| extend resourceId = iff(type == “microsoft.security/assessments”, trim(” “, tolower(tostring(case(source =~ “azure”, properties.resourceDetails.Id,
(type == “microsoft.security/assessments” and (source =~ “aws” and isnotempty(tostring(properties.resourceDetails.ConnectorId)))), properties.resourceDetails.Id,
(type == “microsoft.security/assessments” and (source =~ “gcp” and isnotempty(tostring(properties.resourceDetails.ConnectorId)))), properties.resourceDetails.Id,
source =~ “aws”, properties.resourceDetails.AzureResourceId,
source =~ “gcp”, properties.resourceDetails.AzureResourceId,
extract(“^(?i)(.+)/providers/Microsoft.Security/assessments/.+$”,1,id)
)))), dynamic(null))
| extend resourceName = iff(type == “microsoft.security/assessments”, tostring(coalesce(properties.resourceDetails.ResourceName, properties.additionalData.CloudNativeResourceName, properties.additionalData.ResourceName, properties.additionalData.resourceName, split(resourceId, ‘/’)[-1], extract(@”(.+)/(.+)”, 2, resourceId))), dynamic(null))
| extend resourceType = iff(type == “microsoft.security/assessments”, tolower(properties.resourceDetails.ResourceType), dynamic(null))
| extend riskLevelText = iff(type == “microsoft.security/assessments”, tostring(properties.risk.level), dynamic(null))
| extend riskLevel = iff(type == “microsoft.security/assessments”, case(riskLevelText =~ “Critical”, 4,
riskLevelText =~ “High”, 3,
riskLevelText =~ “Medium”, 2,
riskLevelText =~ “Low”, 1,
0), dynamic(null))
| extend riskFactors = iff(type == “microsoft.security/assessments”, iff(isnull(properties.risk.riskFactors), dynamic([]), properties.risk.riskFactors), dynamic(null))
| extend attackPaths = array_length(iff(type == “microsoft.security/assessments”, iff(isnull(properties.risk.attackPathsReferences), dynamic([]), properties.risk.attackPathsReferences), dynamic(null)))
| extend displayName = iff(type == “microsoft.security/assessments”, tostring(properties.displayName), dynamic(null))
| extend statusCause = iff(type == “microsoft.security/assessments”, tostring(properties.status.cause), dynamic(null))
| extend isExempt = iff(type == “microsoft.security/assessments”, iff(statusCause == “Exempt”, tobool(1), tobool(0)), dynamic(null))
| extend statusChangeDate = tostring(iff(type == “microsoft.security/assessments”, todatetime(properties.status.statusChangeDate), dynamic(null)))
| project assessmentId,
statusChangeDate,
isExempt,
riskLevel,
riskFactors,
attackPaths,
statusCode,
displayName,
resourceId,
assessmentKey,
resourceType,
resourceName,
assessmentTypeSkimmed
| join kind=leftouter (
securityresources
| where type == ‘microsoft.security/assessments/governanceassignments’
| extend assignedResourceId = tolower(iff(type == “microsoft.security/assessments/governanceassignments”, tostring(properties.assignedResourceId), dynamic(null)))
| extend dueDate = iff(type == “microsoft.security/assessments/governanceassignments”, todatetime(properties.remediationDueDate), dynamic(null))
| extend owner = iff(type == “microsoft.security/assessments/governanceassignments”, iff(isempty(tostring(properties.owner)), “unspecified”, tostring(properties.owner)), dynamic(null))
| extend governanceStatus = iff(type == “microsoft.security/assessments/governanceassignments”, case(
isnull(todatetime(properties.remediationDueDate)), “NoDueDate”,
todatetime(properties.remediationDueDate) >= bin(now(), 1d), “OnTime”,
“Overdue”
), dynamic(null))
| project assignedResourceId, dueDate, owner, governanceStatus
) on $left.assessmentId == $right.assignedResourceId
| extend completionStatusNumber = case(governanceStatus == “Overdue”, 5,
governanceStatus == “OnTime”, 4,
statusCode == “Unhealthy”, 3,
isExempt, 7,
1)
| extend completionStatus = case(completionStatusNumber == 5, “Overdue”,
completionStatusNumber == 4, “OnTime”,
completionStatusNumber == 3, “Unassigned”,
completionStatusNumber == 7, “Exempted”,
“Completed”)
| where completionStatus in~ (“OnTime”,”Overdue”,”Unassigned”)
| project-away assignedResourceId, governanceStatus, isExempt
| order by riskLevel desc, attackPaths desc, displayName
Attack Paths:
Use this query to fetch attack path data, providing insights into potential attack vectors within your cloud environment.
securityresources
| where type == “microsoft.security/attackpaths”
| extend riskCategories = tostring(properties.riskCategories)
| extend riskCategories = tostring(split(riskCategories, “[“)[1])
| extend riskCategories = tostring(split(riskCategories, “]”)[0])
| extend riskCategory = iff(‘{riskCategories}’ == “All”, riskCategories, ‘{riskCategories}’)
| where riskCategories has(riskCategory)
| project apId = name, apTemplate = tostring(properties.displayName), riskCategories
| summarize Path_Count = count() by Attack_Path = apTemplate, riskCategories
| project Attack_Path, Path_Count, riskCategories
Secure Score:
This query retrieves secure score data, helping you understand your overall security posture and prioritize remediation efforts.
securityresources
| where type == “microsoft.security/securescores”
| where name == “ascScore”
| extend environment = tostring(properties.environment)
| extend scopeMaxScore = toint(properties.score.max)
| extend scopeWeight = toint(properties.weight)
| extend scopeScorePerc = round(todouble(properties.score.percentage), 0)
Governance:
Use this query to get data on governance rules, enabling you to manage compliance and governance policies effectively.
securityresources
| where type == “microsoft.security/assessments”
| where isnull(properties.resourceDetails.AwsResourceId) and isnull(properties.resourceDetails.GcpResourceId)
| extend DisplayName = tostring(properties.displayName)
| where isempty(DisplayName) == false
| join kind=leftouter (securityresources
| where type == “microsoft.security/assessments/governanceassignments”
| extend assignedResourceId = tostring(todynamic(properties).assignedResourceId)
| extend remediationDueDate = todatetime(properties.remediationDueDate)
| project id = assignedResourceId, governanceassignmentsProperties = todynamic(properties), remediationDueDate) on id
| extend hasAssignment = isempty( governanceassignmentsProperties) == false and isnull( governanceassignmentsProperties) == false
| extend assignmentStatus = iif(tostring(properties.status.code) == “Unhealthy”,iif(hasAssignment == true, iif(bin(remediationDueDate, 1d) < bin(now(), 1d), “Overdue”, “Ontime”), “Unassigned”) , “Completed”)
| summarize count() by assignmentStatus
Compliance:
This query retrieves compliance data from MDC, which is essential for maintaining and demonstrating adherence to various regulatory requirements.
securityresources
| where type == “microsoft.security/regulatorycompliancestandards/regulatorycompliancecontrols/regulatorycomplianceassessments” | extend scope = properties.scope
| where isempty(scope) or scope in~(“Subscription”, “MultiCloudAggregation”)
| parse id with * “regulatoryComplianceStandards/” complianceStandardId “/regulatoryComplianceControls/” complianceControlId “/regulatoryComplianceAssessments” *
| extend complianceStandardId = replace( “-“, ” “, complianceStandardId)
| extend Status = properties.state
Remember, the queries provided above are just examples. ARG allows you to query a wide range of data, so feel free to customize and create queries that suit your specific requirements. With ARG, you have the flexibility to retrieve and analyze any data available within your MDC environment, ensuring comprehensive and tailored insights.
Step 4: Create Visualizations in Power BI
Select Visualization Type: Choose from various visualizations such as charts, graphs, and maps to represent your data.
Customize Visualizations: Use the drag-and-drop functionality to customize your visualizations.
Create Dashboards: Arrange your visualizations into dashboards to provide a comprehensive view of your security data.
Perhaps you can build a report similar to the one shown in the picture below.
If you prefer, you can also use a predefined sample report available for download from the Defender for Cloud GitHub.
This sample report provides a great starting point and can be customized further to meet your specific needs, ensuring you get the most out of your MDC data.
Step 5: Share and Collaborate
Publish Reports: Publish your reports to the Power BI service to share with your team.
Set Permissions: Use role-based access controls to manage who can view or edit the reports.
Conclusion
By leveraging Power BI’s advanced features alongside Azure Workbooks, organizations can unlock deeper insights, create more customized and interactive reports, and improve collaboration across teams. This approach provides a more comprehensive and flexible solution for visualizing and analyzing MDC data, enhancing security posture management and decision-making.
Microsoft Defender for Cloud Additional Resources
Watch a demonstration on how to use Governance Rule in this episode of Defender for Coud in the Field
Download the new Microsoft CNAPP eBook at aka.ms/MSCNAPP
Become a Defender for Cloud Ninja by taking the assessment at aka.ms/MDCNinja
Reviewers
Yuri Diogenes, Principal PM Manager, CxE Defender for Cloud
Tal Rosler, Senior PM lead, Microsoft Defender for Cloud
Microsoft Tech Community – Latest Blogs –Read More
Block the third-party Antivirus installation on endpoint
Is there is any option available to block the third party Antivirus installation on windows endpoints. This will help us to prevent EDR running in passive mode.
Is there is any option available to block the third party Antivirus installation on windows endpoints. This will help us to prevent EDR running in passive mode. Read More
Get $25 USD for reviewing a Microsoft Security product on Gartner Peer Insights
We love hearing more about our customers’ experience with our products!
We’re currently working on growing our product reviews of Microsoft Security products on Gartner Peer Insights. We would love for you to participate and share your thoughts, feedback, and experiences using Microsoft Security products to help others in their buying process.
To provide feedback on the capabilities of the Microsoft Security products, please click on the link below. You will need to first log in to your Gartner Peer Insights account or take 30 seconds to create a free account.
Once you have completed your review, GPI will prompt you to choose a gift card option. Gift cards are valued at $25 USD, and they are available in multiple currencies worldwide. As soon as your review is approved, the card will be made available to you digitally.
Microsoft Defender for Cloud Apps
Each person is limited to one review per product on the above-mentioned site.
Only Microsoft customers are eligible to participate. Microsoft partners and MVPs are not eligible.
The offer is good only for those who submit a product review on Gartner Peer Insights as linked on this page.
Any gift returned as non-deliverable will not be re-sent. Microsoft reserves the right to cancel, change, or suspend this offer at any time without notice.
The offer is non-transferable and cannot be combined with any other offer.
This offer runs through June 30, 2025, or while supplies last, and is not redeemable for cash. Taxes, if there are any, are the sole responsibility of the recipient.
This offer does not apply to customers in Cuba, Iran, North Korea, Sudan, Syria, Region of Crimea, Russia, and China.
Please see the below for more information
Microsoft Privacy Statement
Gartner’s Community Guidelines & Gartner Peer Insights Review Guide
We love hearing more about our customers’ experience with our products!
We’re currently working on growing our product reviews of Microsoft Security products on Gartner Peer Insights. We would love for you to participate and share your thoughts, feedback, and experiences using Microsoft Security products to help others in their buying process.
To provide feedback on the capabilities of the Microsoft Security products, please click on the link below. You will need to first log in to your Gartner Peer Insights account or take 30 seconds to create a free account.
Once you have completed your review, GPI will prompt you to choose a gift card option. Gift cards are valued at $25 USD, and they are available in multiple currencies worldwide. As soon as your review is approved, the card will be made available to you digitally.
Microsoft Defender for Cloud Apps
Microsoft Sentinel
Microsoft Purview eDiscovery
Each person is limited to one review per product on the above-mentioned site.
Only Microsoft customers are eligible to participate. Microsoft partners and MVPs are not eligible.
The offer is good only for those who submit a product review on Gartner Peer Insights as linked on this page.
Any gift returned as non-deliverable will not be re-sent. Microsoft reserves the right to cancel, change, or suspend this offer at any time without notice.
The offer is non-transferable and cannot be combined with any other offer.
This offer runs through June 30, 2025, or while supplies last, and is not redeemable for cash. Taxes, if there are any, are the sole responsibility of the recipient.
This offer does not apply to customers in Cuba, Iran, North Korea, Sudan, Syria, Region of Crimea, Russia, and China.
Please see the below for more information
Microsoft Privacy Statement
Gartner’s Community Guidelines & Gartner Peer Insights Review Guide Read More
Enter a date in a cell after I have scanned a bar code
Hello, I am trying to add a date in column K after I have scanned the bar code label containing the TCN that is in column B. Below is the VBA code that I am using to find the TCN and then highlights the row and adds that TCN to column N. I like to add the date to column K for each day that I scan. Thank you.
Private Sub Worksheet_Change(ByVal target As Range)
If Not Intersect(target, Columns(“M”)) Is Nothing Then
Z = Intersect(target, Columns(“M”)).Value
If IsNumeric(Z) Then
x = Application.Evaluate(“MATCH(” & Z & “,B:B,0)”)
Else
x = Application.Evaluate(“MATCH(” & Chr(34) & Z & Chr(34) & “,B:B,0)”)
End If
If Not IsError(x) Then
Application.Goto Cells(x, 15)
End If
Hello, I am trying to add a date in column K after I have scanned the bar code label containing the TCN that is in column B. Below is the VBA code that I am using to find the TCN and then highlights the row and adds that TCN to column N. I like to add the date to column K for each day that I scan. Thank you. Private Sub Worksheet_Change(ByVal target As Range)If Not Intersect(target, Columns(“M”)) Is Nothing ThenZ = Intersect(target, Columns(“M”)).ValueIf IsNumeric(Z) Thenx = Application.Evaluate(“MATCH(” & Z & “,B:B,0)”)Elsex = Application.Evaluate(“MATCH(” & Chr(34) & Z & Chr(34) & “,B:B,0)”)End IfIf Not IsError(x) ThenApplication.Goto Cells(x, 15)End If Read More
FILTER or any other solution function
Dear Experts,
I have a Data like below (attached file) :-
From the Sheet1, I need to generate a report as in the Desired Output sheet, first 2 rows populated for the reference,
Thanks in Advance,
Br,
Anupam
Dear Experts, I have a Data like below (attached file) :-From the Sheet1, I need to generate a report as in the Desired Output sheet, first 2 rows populated for the reference,Thanks in Advance,Br,Anupam Read More
Send data directly from COM3 to Excel
I need to send data from a data collection device to Excel via USB
I have tried everything available but nothing works.
Do you know how to do that?
I need to send data from a data collection device to Excel via USB I have tried everything available but nothing works. Do you know how to do that? Read More
Office documents opened from onedrive often are write-protected
Hello
Please i need your help on this issue.
WE are having an issue with Office 365 documents randomly opening as write-protected from OneDrive.
We are trying to open the document from the OneDrive desktop application and the same via latest opened files popup list showing when first opening excel without any document active,
The same file can shift between opening fine and the next time being write protected 5 seconds later.
The document are on my personal OneDrive.
The main menu bar icon in MAC OS says files are synced when clicking on it.
There is no issues with these files when I open them from OneDrive in a web browser or when using OneDrive desktop application on windows 11, only problem when using MAC and I have tried 2 different MAC, same issue. I have reinstalled the whole office suite without any solution.
The document is not stored in a shared folder?
For example. The same file, opened from last opened documents list inside Excel is = write protected, and the auto-save is greyed out on one file. Closing Excel and opening the same file again by clicking on it via OneDrive files folder in MAC OS GUI file system makes the file open up fine.
Hello Please i need your help on this issue. WE are having an issue with Office 365 documents randomly opening as write-protected from OneDrive.We are trying to open the document from the OneDrive desktop application and the same via latest opened files popup list showing when first opening excel without any document active, The same file can shift between opening fine and the next time being write protected 5 seconds later.The document are on my personal OneDrive.The main menu bar icon in MAC OS says files are synced when clicking on it.There is no issues with these files when I open them from OneDrive in a web browser or when using OneDrive desktop application on windows 11, only problem when using MAC and I have tried 2 different MAC, same issue. I have reinstalled the whole office suite without any solution. The document is not stored in a shared folder? For example. The same file, opened from last opened documents list inside Excel is = write protected, and the auto-save is greyed out on one file. Closing Excel and opening the same file again by clicking on it via OneDrive files folder in MAC OS GUI file system makes the file open up fine. Read More
Tech Talks Presents: Power Pages Search with Gen AI & File attachment upgrade I July 18th
Join us on Thursday, July 18 at 8am PT as Saumitra Nanda, Principal Program Manager, Nagesh Bhat, Sr. Product Manager, and Ankita Vishwakarma, Sr. Product Manager present Power Pages Search with Gen AI & File attachment upgrade.
Call to Action:
Click on the link to save the calendar invite: https://aka.ms/TechTalksInvite
View past recordings (sign in required): https://aka.ms/TechTalksRecording
Join us on Thursday, July 18 at 8am PT as Saumitra Nanda, Principal Program Manager, Nagesh Bhat, Sr. Product Manager, and Ankita Vishwakarma, Sr. Product Manager present Power Pages Search with Gen AI & File attachment upgrade.
Call to Action:
Click on the link to save the calendar invite: https://aka.ms/TechTalksInvite
View past recordings (sign in required): https://aka.ms/TechTalksRecording
Converting Azure Virtual Machines running Windows from SCSI to NVMe
This is the Windows version of the blog article on converting Azure virtual machines from SCSI to NVMe.
The Linux version can be found here.
Introduction
In the ever-evolving world of cloud computing, maximizing performance and efficiency is crucial for businesses leveraging virtual machines (VMs) on platforms like Microsoft Azure, especially for high I/O workloads like SAP on Azure or database applications. One significant upgrade that can yield substantial performance improvements is converting your Azure VM from a SCSI (Small Computer System Interface) disk setup to NVMe (Non-Volatile Memory Express) using Azure Boost. This blog post will guide you through the process of making this conversion and explore the numerous advantages of NVMe over SCSI.
Advantages of Azure Boost
Azure Boost is a powerful enhancement tool for Azure VMs, offering the following advantages:
Accelerated Disk Performance: Azure Boost optimizes disk I/O operations, significantly increasing the speed and efficiency of your VM’s storage.
Seamless Integration: Easily integrates with existing Azure infrastructure, allowing for a smooth transition and immediate performance benefits.
Cost-Effective Optimization: By enhancing the performance of existing VMs, Azure Boost helps reduce the need for more expensive hardware upgrades or additional resources.
To learn more about Azure Boost visit our documentation or the announcement blog.
What is changing for your VM?
Changing the host interface from SCSI to NVMe will not change the remote storage (OS disk or data disks), but change the way the operating systems sees the disks. Windows will present the OS disk and the remote storage as “Virtual_Disk NVMe Premium” devices.
Migrate your virtual machine (VM) from SCSI to NVMe
To migrate from SCSI to NVMe and benefit from higher performance some steps need to be followed:
Check if your virtual machine series supports NVMe
Check your operating system for NVMe readiness
Convert your virtual machine to NVMe
Check your operating system
1. Check if your virtual machine series supports NVMe
The supported virtual machine SKUs to support NVMe attached disks is available in our documentation and in the table below.
If your VM type is not listed below change the VM type.
Size Series
Series Type
Deployment Status
Dalsv6
General Purpose
Preview
Easv6
Memory Optimized
Preview
DCesv5
General Purpose
Preview
ECesv5
Memory Optimized
Preview
Mv3 Medium Memory
High Memory to CPU Optimized
Production
Falsv6/Famsv6
Compute Optimized
Preview
Dlsv5
General Purpose
Production
Dsv5
General Purpose
Production
Esv5
Memory Optimized
Production
Ebsv5
Managed disks optimized
Production
Lsv3
Local storage optimized
Production
Dplsv5
General Purpose
Production
Dpsv5
General Purpose
Production
Epsv5
Memory Optimized
Production
Nvadsv5
GPU/AI workload optimized
Production
HBv4
High Performance Compute (HPC)
Production
HX
High Performance Compute (HPC)
Production
As the list of supported VM families may change over time, please check the up-to-date documentation.
2. Check your operating system for NVMe readiness
The operating system needs to support NVMe devices, Microsoft supports running Windows Server 2019 and newer with NVMe devices on Azure. Please make sure to have all updates installed before converting the VM. Older releases than Windows Server 2019 are NOT supporting NVMe devices. DO NOT APPLY THE PROCEDURES DESCRIBED IN THE ARTICLE TO WINDOWS SERVER 2016 OR OLDER WINDOWS SERVER RELEASES.
2.1 Check Controller Type of VM
2.1.1 Check Controller Type using PowerShell
PS C:Usersuser1> $vm = Get-AzVM -name nvme-win2022
PS C:Usersuser1> $vm.StorageProfile.DiskControllerType
SCSI
PS C:Usersuser1>
2.1.2 Check Controller Type using Azure CLI
$ az vm show –name nvme-win2022 –resource-group nvme-win2022
{
“additionalCapabilities”: {
…
“storageProfile”: {
…
“diskControllerType”: “SCSI”,
…
2.1.3 Check Controller Type using Azure Portal
2.2 Prepare Windows
To make Windows Server 2019 and newer ready for the conversion you need to delete a registry key. This is required as Windows Setup, when initially deploying the OS, marks the required drivers for the OS Disk. This means that only the storport driver for SCSI is loaded early during boot. While the NVMe driver is installed in all operating systems, it is not loaded early enough for the OS to start.
To make NVMe driver be part of the early start you need to run this command or delete the registry key path in regedit.
2.2.1 Delete the registry path using reg command
PS C:Usersazureuser> reg delete HKLMSYSTEMCurrentControlSetServicesstornvmeStartOverride /f
The operation completed successfully.
PS C:Usersazureuser>
2.2.2. Manually delete the registry path
2.3. Shutdown Windows
Next step is to shutdown windows and convert the Virtual Machine.
3. VM SCSI to NVMe conversion
To convert the operating system multiple steps are required.
Change the metadata of the OS disk to include NVMe capabilities
Change the SCSI controller to NVMe
3.1 Download the PowerShell script
To download the PowerShell script from the GitHub repo use the following command:
Invoke-WebRequest -Uri “https://raw.githubusercontent.com/Azure/SAP-on-Azure-Scripts-and-Utilities/main/NVMe-Preflight-Check/azure-nvme-VM-update.ps1” -OutFile “.azure-nvme-VM-update.ps1”
3.2. Convert the Virtual Machine
To convert run the script, detailed documentation is also available on the GitHub repository.
You can decide if e.g. the VM should automatically be started after the reconfiguration.
PS D:TEMP> .azure-nvme-VM-update.ps1 -subscription_id 232XXXXX-XXXX-XXXX-88c0-75747223XXXX -resource_group_name NVMe-win2022 -vm_name NVMe-win2022 -disk_controller_change_to NVMe -start_vm_after_update $true -vm_size_change_to Standard_E4bds_v5
INFO – OS Disk found
INFO – Access token generated
INFO – Getting VM info
INFO – Getting all VM SKUs available in Region swedencentral
INFO – This will take about a minute …
INFO – Checking for TrustedLaunch
INFO – Checking if VM is stopped and deallocated
INFO – Stopping VM
Tenant: 72f988bf-86f1-41af-91ab-2d7cd011db47
SubscriptionName SubscriptionId Account Environment
—————- ————– ——- ———–
XX-XX-XX-XXXXXXX 232bXXXX-XXXX-XXXX-XXXX-75747223XXXX xyz@microsoft.com AzureCloud
OperationId : 60bffc73-54a9-4d10-8246-881c506f23ee
Status : Succeeded
StartTime : 15.07.2024 17:23:47
EndTime : 15.07.2024 17:23:59
Error :
Name :
INFO – Setting OS Disk to SCSI/NVMe
INFO – Getting VM config to prepare new config
INFO – Setting new VM size
INFO – Setting disk controller for VM
INFO – Updating the VM configuration
RequestId :
IsSuccessStatusCode : True
StatusCode : OK
ReasonPhrase :
INFO – Waiting for 1 min before starting up
INFO – Starting VM
OperationId : aaedaa1d-968a-4e85-a795-979acddb7f83
Status : Succeeded
StartTime : 15.07.2024 17:25:35
EndTime : 15.07.2024 17:25:47
Error :
Name :
PS D:TEMP>
3.3 Check the result
3.3.1 Check result in Azure Portal
3.3.2 Check result in PowerShell
PS C:Users> $vm = Get-AzVM -name nvme-win2022
PS C:Users> $vm.StorageProfile.DiskControllerType
NVMe
PS C:Users>
4. Check your operating system
As a last step check your operating system and all the connected drives. Any existing file system will be available after the migration.
In Windows Device Manager you will see the new device specification. If your Azure Virtual Machine has a temporary disk/resource disk assigned, you will see one “Microsoft Virtual Disk” as those are still presented through the SCSI protocol. This is by design of Azure and can’t be changed.
Microsoft Tech Community – Latest Blogs –Read More
Introducing Coauthoring for SharePoint Pages and News
We’re excited to announce that we’ve started rolling out the ability for multiple authors to collaborate on SharePoint pages. Multiple authors will be able to edit the same page at the same time, without having to take turns. Authors can see real-time changes made by others as they happen in pages, sections, and web parts.
After an author starts editing a page, they can see if other authors are also editing by viewing their avatar in the command bar. Changes made by authors are saved automatically every few seconds. If authors want to leave edit mode, they can select Save and close.
Version history has also been improved to enhance support for collaborative authoring. To undo more changes or changes made by other users, authors can restore a prior version from Version History (under Page details).
Watch a demo:
See this experience in action with the live demo on our Community Learning YouTube channel.
Frequently asked questions
When will this happen?
We have begun rolling out to targeted release tenants and expect to complete by late July 2024. We expect general availability rollout to start in August.
Track the feature status with Microsoft 365 Roadmap ID 124853.
How will this affect your organization?
Before this rollout, only one author can edit a page. While an author is editing the page no other user can edit until the author publishes or saves their draft.
After this rollout, authors will have the ability to edit a page or news post while others are editing as well.
What do you need to prepare?
You do not need to do anything to prepare for this update, but you may want to let your authors know about this new capability.
Microsoft Tech Community – Latest Blogs –Read More
Azure Virtual WAN configuration best practices
Azure Virtual WAN is a networking service that combines networking, security, and routing features in one managed service. It is a hub-and-spoke architecture managed by Microsoft that integrates with other Azure services, such as VPN gateways and Azure Firewall, and partner solutions. It aims to simplify network management and configuration, and enhance performance and reliability, using Microsoft’s global network.
To learn more about Virtual WAN’s features, see Azure Virtual WAN Overview | Microsoft Learn. For the complete list of supported partner solutions in Virtual WAN, see About NVAs in a Virtual WAN hub.
This article details Virtual WAN configuration best practices to help you make the most of the benefits Virtual WAN provides. These best practices are aligned to the five pillars of the Azure Well-Architected Framework:
Reliability
Security
Cost optimization
Operational excellence
Performance efficiency
Reliability
Design checklist
Leverage Availability Zones resiliency.
Adopt active-active configuration in Virtual WAN Site-to-Site VPN deployments.
Use global VPN profile for more reliable point-to-site connections to Virtual WAN.
Allocate a P2S VPN client address pool with enough IP addresses as two times the number of users connecting at the same time.
Choose Network Virtual Appliance (NVA) or Software-as-a-service (SaaS) solutions that integrate natively into the virtual hub.
Review the list of Virtual WAN Known Issues and feature limitations before implementation.
The following table details all the recommendations, and their benefits, mentioned above to optimize your Azure Virtual WAN configuration for reliability.
Recommendation
Benefit
When planning your Virtual WAN deployment, choose an Azure region(s) to create your hub(s) that supports Availability Zones, for a higher service-level agreement (SLA). For more information, see Availability Zone service and region support.
Deploy your hub’s Azure Firewall(s) across Availability Zones too, for higher SLA. To do so, use Azure Firewall Manager Portal, PowerShell, or Azure CLI.
Except for Azure Firewall, all services deployed in a Virtual WAN hub (VPN, ExpressRoute, etc.) will be automatically deployed across Availability Zones, if the deployment region supports this feature.
Deploying the hub’s services across Availability Zones increases Virtual WAN’s service-level agreement (SLA). For more information, see SLA for Azure Virtual WAN. For information about all Azure SLAs, see SLA summary for Azure services.
Leverage the built-in resiliency of hub VPN gateways by fully adopting an active-active configuration in your Site-to-Site VPN deployments.
When creating a VPN connection to an on-premises site, make sure to establish a tunnel between the on-premises device(s) and each VPN gateway instance.
It is highly recommended to become familiar with the concepts of VPN connection, link, and tunnel in Virtual WAN. For more information, see Azure Virtual WAN FAQ | Microsoft Learn.
All gateways provisioned in a Virtual WAN hub are in active-active mode, but to take advantage of this built-in resiliency, you must establish a separate tunnel between your on-premises device(s) and each gateway instance.
Doing so will ensure your connections to Virtual WAN are resilient and reliable. To learn more about different high availability designs for Site-to-Site VPN in Virtual WAN see: Disaster recovery design for Azure Virtual WAN | Microsoft Learn.
Virtual WAN provides two types of connection profiles for User VPN clients – hub profile and global profile.
It is recommended to use a global profile when having multi-hub Virtual WAN deployments, unless there is a specific requirement to restrict access to a certain hub only.
For more information on User VPN Profiles in Virtual WAN, see P2S global and hub profiles.
When using a global profile, VPN clients connect to the closest available virtual hub that offers the best network performance, thanks to a built-in traffic manager.
This configuration also increases resiliency, as the global profile is capable of redirecting users to a back-up Virtual WAN hub.
To learn more about remote user connectivity resiliency in Virtual WAN, see Disaster Recovery design.
To ensure all users can connect, even if one P2S VPN gateway instance is down, allocate a client address pool with a number of IP addresses twice the amount of users connecting at the same time.
To learn more about client address pools for Virtual WAN P2S configurations, see About client address pools for P2S User VPN – Azure Virtual WAN | Microsoft Learn.
When creating a P2S VPN gateway, you must configure a client address pool from which IP addresses will be automatically assigned to VPN clients.
Assigned address pools are split into half and allocated to each gateway instance. These halves are statically assigned to instances and cannot migrate during maintenance or downtime events.
Having a pool of IPs that is twice the number of users ensures all clients are still able to connect in case a gateway instance is down.
Whenever possible, choose to deploy a supported NVA or SaaS solution in the virtual hub over running such services in a spoke.
For the list of supported partners, see NVA in Virtual WAN hub and Software-as-a-Service in Virtual WAN.
Supported solutions in the hub have been tested and validated by Microsoft and the partner.
Natively integrated solutions leverage on the built-in availability and resiliency of Virtual WAN and integrate more seamlessly with other Virtual WAN features, such as Routing Intent, among other benefits.
Review the list of Virtual WAN known issues and feature limitations (Routing Intent limitations, for example) before implementation.
For all the information on recent releases, known issues, and feature limitations, see What’s new in Azure Virtual WAN? | Microsoft Learn.
Because Virtual WAN deployments often involve the creation of different network services, reviewing this information prior to implementation helps plan your deployment better and avoid future issues.
Security
Design checklist
Leverage secured virtual hub(s). Use Routing Intent to secure private and internet traffic.
Follow Azure Firewall or third-party security provider configuration best practices.
Leverage Private Link to connect to Azure PaaS services from Virtual WAN.
Use Network Security Groups (NSGs) in spoke VNets to control intra-VNet traffic.
Use site-to-site/user VPN or ExpressRoute to access Virtual WAN connected networks securely and privately.
Use Azure Firewall DNAT rules, or a similar feature if using a supported NVA, to securely expose non-http(s) applications on the internet. Use Azure Application Gateway to securely expose http(s) applications on the internet.
Protect public IPs in spoke virtual networks against DDoS attacks using Azure DDoS Network or IP Protection.
Apply Zero Trust Principles when configuring Virtual WAN.
Recommendations
The following table details all the recommendations, and their benefits, mentioned above to optimize your Azure Virtual WAN configuration for security.
Recommendation
Benefit
Create secured virtual hub(s) by deploying Azure Firewall or a supported partner solution, (NVA or SaaS), in the hub.
In a secured virtual hub, you can enforce a routing policy using Routing Intent to inspect private and internet traffic using he security solution deployed in the hub. This increases the overall security of your Virtual WAN deployment.
Follow Azure Firewall or third-party security provider configuration best practices.
Following your firewall provider’s configuration guidance ensures your Virtual WAN deployment remains secure and reliable.
To secure access to PaaS services from Azure and non-Azure clients, create Private Endpoints to those services in a spoke virtual network connected to any virtual hub.
To learn more about how to use Private Link in Virtual WAN, see Share a private link service across Virtual WAN – Azure Virtual WAN | Microsoft Learn.
Azure Private Link allows you to access PaaS services without having a public endpoint on those services. You can continue to leverage Private Link in Virtual WAN, and even secure traffic to private endpoints using Azure Firewall in the hub.
To learn more about this scenario, see Secure traffic destined to private endpoints in Azure Virtual WAN | Microsoft Learn.
Use Network Security Groups (NSGs) in spoke virtual networks to control intra-VNet traffic.
If there is a requirement to inspect traffic between subnets of the same VNet, add a subnet level UDR for each subnet whose traffic you want to force through the firewall. For example, if you want to inspect traffic between subnet 10.3.0.0/26 and subnet 10.3.1.0/24, add a route table to subnet 10.3.0.0/26 containing a 10.3.1.0/24 UDR with next hop Azure Firewall or NVA Private IP and vice-versa.
See Azure virtual network traffic routing | Microsoft Learn to learn more.
Virtual WAN hub can’t attract traffic between two subnets in the same virtual network.
For this reason, it is recommended to apply NSGs at subnet level to control traffic between subnets. For more routing considerations, see About virtual hub routing – Azure Virtual WAN | Microsoft Learn.
Even though using NSGs to control intra-VNet traffic is less error prone, it is still possible to inspect traffic between subnets of the same VNet using subnet level UDRs.
Whenever possible, use site-to-site/user VPN and/or ExpressRoute to access workloads in spoke virtual networks connected to the virtual hub, including RDP/SSH access.
For sites where the above connectivity options are not feasible, consider deploying Azure Bastion in a connected spoke virtual network to access virtual machines. To learn more about how Azure Bastion integrates with Virtual WAN, see Azure Bastion FAQ | Microsoft Learn.
Leveraging site-to-site VPN or User VPN ensures you can securely access your Virtual WAN connected networks over the public internet. Azure ExpressRoute, on the other hand, offers a highly reliable and secure connection that does not traverse the public internet.
Azure Bastion lets you securely RDP/SSH to Azure virtual machines, and even on-premises machines, using IP-based connections, without exposing a public IP on target machines.
For publicly facing, non-http(s), workloads running in spoke virtual networks, it is recommended to securely expose them on the internet through a DNAT rule in Azure Firewall (or running in the hub.
Deploy Azure Application Gateway in a spoke virtual network to securely expose publicly facing, regional, http(s) applications, also running in spoke virtual networks.
You can also leverage Application Gateway’s features to access privately facing http(s) applications. To learn more about Application Gateway’s features, see What is Azure Application Gateway | Microsoft Learn.
Leveraging Azure Firewall and/or Application Gateway to expose your applications ensures client traffic is always inspected before being sent to the application servers, which can be kept private.
Azure Firewall offers advanced threat protection features, such as Threat intelligence-based filtering or IDPS, whereas Application Gateway can protect applications against L7 DDoS attacks using WAF, as well as L3 and L4 attacks when combined with Azure DDoS protection.
Azure Firewall and Application Gateway can also be combined in the same design to benefit from the features of both services. To learn more about possible designs, see Firewall, App Gateway for virtual networks – Azure Example Scenarios | Microsoft Learn.
Enable Azure DDoS Network Protection in spoke virtual networks containing services with public IPs. Alternatively, enable protection on specific public IPs using DDoS IP Protection.
It is not possible to enable DDoS protection on services deployed in the Virtual WAN hub at this time.
Azure DDoS Protection provides always-on traffic monitoring, adaptive real time tuning, metrics and alerts for protected virtual networks and public IPs, to ensure services with public endpoints remain available.
To learn more, see Azure DDoS Protection Overview | Microsoft Learn.
In addition to the best practices described in this article, apply Zero Trust principles to your Azure Virtual WAN deployments by following the configuration guidance described here: Apply Zero Trust principles to Azure Virtual WAN | Microsoft Learn.
Increase security even more in your Virtual WAN deployment by applying Zero Trust principles in your configuration – verify explicitly, use least privileged access, and assume breach.
Cost optimization
Design checklist
Understand Virtual WAN pricing components and data transfer costs.
Estimate throughput requirements in advance to achieve cost-effectiveness when selecting gateway scale units and virtual hub capacity.
Monitor and optimize the utilization of hub services to maintain cost-effectiveness.
Keep in mind security and throughput requirements when selecting an Azure Firewall SKU.
Optimize VWAN routes to minimize costs. Consider the cost implications of transferring data between different Virtual WAN components.
Recommendations
The following table details all the recommendations, and their benefits, mentioned above to optimize your Azure Virtual WAN configuration for cost optimization.
Recommendation
Benefit
Virtual WAN deployments often involve the creation of different networking services, such as gateways or firewalls. It’s important to be aware of the costs associated with the use of these services, as well as data processing charges.
A detailed breakdown of Virtual WAN costs can be found here: About Virtual WAN pricing – Azure Virtual WAN | Microsoft Learn.
By understanding Virtual WAN pricing beforehand, you’re able to plan your deployment better and make an informed decision on what services should be included/excluded from the design, for example, therefore avoiding unexpected costs in the long run.
Estimate throughput requirements in advance to achieve cost-effectiveness when selecting gateway scale units and virtual hub capacity.
While the virtual hub router is capable of scaling out, it is important to secure enough minimum capacity when creating the virtual hub.
To learn more about virtual hub capacity, see About virtual hub settings – Azure Virtual WAN | Microsoft Learn.
To learn more about gateway scale units, see About gateway settings for Virtual WAN – Azure Virtual WAN | Microsoft Learn.
This will allow you to select the appropriate number of scale units for your hub gateways and number of routing infrastructure units for your virtual hub, which can be adjusted if needed and allow you to avoid overspending.
Leverage Virtual WAN metrics to monitor the utilization of hub services to maintain cost-effectiveness.
To learn more about supported Virtual WAN metrics and recommended alerts, see Monitoring Virtual WAN – Best practices – Azure Virtual WAN | Microsoft Learn.
By continuously monitoring the utilization of hub services, in particular hub gateways, you’re able to quickly detect if these services are underutilized, and if so, adjust the number of scale units accordingly.
Azure Firewall comes in three SKUs, with different features and pricing associated. Choose the SKU that fulfills your security requirements, as well as throughput needs.
For a feature comparison across the three Azure Firewall SKUs, see Choose the right Azure Firewall version to meet your needs | Microsoft Learn.
Selecting the appropriate Firewall SKU ensures you don’t incur unnecessary costs in your Virtual WAN deployment.
Optimize your Virtual WAN environment to minimize costs. Consider the cost implications of transferring data between different Virtual WAN components.
For example, clients should access spoke virtual networks in a region primarily through the hub where those spokes are connected in the same region. From a cost perspective (and performance), this is a better approach when compared to accessing a spoke in region B through a hub in region A first, which requires traversing the hubs in both regions. This latter approach implies inter-hub and inter-region processing charges, whereas the former doesn’t.
To learn more about Virtual WAN pricing, see About Virtual WAN pricing – Azure Virtual WAN | Microsoft Learn.
Optimizing your Virtual WAN environment ensures your deployment remains cost-effective. By accessing latency-sensitive workloads directly via the hub connected to these spoke virtual networks, you will also experience better traffic performance.
Operational excellence
Design checklist
Use Infrastructure-as-Code (IaC) technologies to provision and maintain your Virtual WAN deployment.
Leverage Azure Monitor Insights to keep track of Virtual WAN topology, services deployed, and dependencies.
Configure Azure alerts to quickly detect and act on connectivity and performance issues.
Leverage customer-controlled gateway maintenance.
Assign a /23 subnet when creating virtual hubs.
Recommendations
The following table details all the recommendations, and their benefits, mentioned above to optimize your Azure Virtual WAN configuration for operational excellence.
Recommendation
Benefit
Use Infrastructure-as-Code (IaC) technologies, such as Azure Resource Manager (ARM) templates or Bicep, to provision and maintain your Virtual WAN deployment.
Provides consistency and acts as a safeguard in case there’s a need to redeploy your Virtual WAN.
Leverage Azure Monitor Insights to monitor your Virtual WAN deployment.
Azure Monitor Insights for Virtual WAN provides a centralized view of your Virtual WAN topology, services deployed and dependencies, as well as metrics at hub, gateway, and connection level.
Closely monitor health and utilization metrics for hub services like VPN (point-to-site or site-to-site), ExpressRoute, or Azure Firewall. You can also configure alerts for these metrics.
Configure Azure alerts to quickly detect and act on connectivity and performance issues. For a list of recommended alerts, see Monitoring Virtual WAN – Best practices – Azure Virtual WAN | Microsoft Learn.
For the complete list of supported Virtual WAN metrics and logs, see Monitoring Azure Virtual WAN – Data reference | Microsoft Learn.
Proactively configuring alerts for key Virtual WAN metrics and logs minimizes the chances of downtime, which is crucial in a production environment.
Configure a maintenance window1 for site-to-site VPN and ExpressRoute gateways.
To learn more about this feature, see Azure Virtual WAN FAQ | Microsoft Learn.
Gives customers more control over periodic maintenance updates. Guest OS and Service maintenance events will happen during the window specified by the user.
A Virtual WAN hub requires a /24 minimum subnet size, however, the recommended subnet size at creation is /23, to accommodate the creation of multiple hub services such as gateways, Azure Firewall, the virtual hub router, or NVAs.
The virtual hub’s address space cannot be changed after creation. Thus, to avoid having to redeploy your virtual hub and experiencing downtime, make sure you create your hub with enough address space in advance to enable the creation of planned hub services, as well as to accommodate potential changes to the design in the long run.
To learn more about subnet size requirements in Virtual WAN, see Azure Virtual WAN FAQ | Microsoft Learn.
1Customer-controlled gateway maintenance is currently in public preview, and is therefore not recommended for production environments. To learn more about this feature, see Configure customer-controlled maintenance for your Virtual WAN gateways.
Performance efficiency
Design checklist
Consider the per-hub limits when choosing how many virtual hubs to create in each region.
Review the routing implications of redundant connectivity when planning for high availability and disaster recovery in Virtual WAN.
Choose the hub routing preference option that works best for your scenario.
Prioritize hub-to-hub path over ExpressRoute for VNet-to-VNet connectivity.
Estimate the need per VPN tunnel when planning for a VPN gateway.
Use the GCMAES256 algorithm for both IPSec Encryption and Integrity for optimal performance when configuring VPN site-to-site connections.
Regularly monitor the utilization of virtual hub gateways (VPN or ExpressRoute) and resize when needed.
Monitor virtual hub capacity.
Recommendations
The following table details all the recommendations, and their benefits, mentioned above to optimize your Azure Virtual WAN configuration for performance efficiency.
Recommendation
Benefit
Evaluate if your regional presence exceeds or is near to reaching the capacity of a single virtual hub. Additional hubs in the same region can be added if there is a requirement to scale beyond the limits of a single hub, or if there’s a requirement for a different hub configuration.
To learn more about Virtual WAN limits, see Azure subscription limits and quotas – Azure Resource Manager | Microsoft Learn.
Ensures your Virtual WAN maintains optimal performance.
Review the routing implications of redundant connectivity for VPN (Site-to-Site, Point-to-Site) and ExpressRoute when planning for high availability and disaster recovery in Virtual WAN.
Having more than one path to the same network can cause asymmetric routing or lead to suboptimal performance when not properly architected. This is why it is important to review supported designs and their routing implications, detailed in this article: Disaster recovery design for Azure Virtual WAN | Microsoft Learn.
Moreover, it is important to test your high availability and disaster recovery mechanisms regularly to ensure they are working as intended.
Ensure you adopt a design that meets your business continuity and disaster recovery (BCDR) requirements, while maintaining optimal performance during steady state.
Review the supported hub routing preference options in Virtual WAN and choose the one that works best for your scenario.
To learn more about virtual hub routing preference, see Virtual WAN virtual hub routing preference – Azure Virtual WAN | Microsoft Learn.
The default hub routing preference in Virtual WAN is ‘ExpressRoute’, however, ‘AS Path’ may be the best option to fulfill your specific routing requirements, for example.
To make an informed decision on which hub routing preference option fulfills your requirements the best, see Azure Virtual WAN FAQ | Microsoft Learn.
By default, VNet-to-VNet and VNet to Virtual WAN connectivity is disabled through an ExpressRoute circuit.
However, when two hubs are connected and there is a single ExpressRoute connected as a bow-tie to both hubs, the ExpressRoute circuit will be the preferred path over hub-to-hub for a VNet connected to the first hub to reach a VNet connected to the second hub.
To make sure hub-to-hub is the preferred path, it is recommended to configure AS-Path as the hub routing preference. Alternatively, configure multiple ExpressRoute circuits to connect to each hub.
To learn more about this scenario, see Azure Virtual WAN FAQ | Microsoft Learn.
The ExpressRoute path is not ideal for VNet-to-VNet traffic. ExpressRoute gateways have resource limitations (bandwidth, for example) and can therefore become bottlenecks. In addition, ExpressRoute doesn’t offer optimal performance when compared to hub-to-hub. With ExpressRoute there is an extra hop because traffic must pass through the MSEE devices in the peering location.
Preferring the hub-to-hub path ensures optimal performance for VNet-to-VNet connectivity.
To learn more, see Connectivity between virtual networks over ExpressRoute | Microsoft Learn.
The throughput of a VPN gateway instance is available across all tunnels connecting to that instance. Thus, it is important to select the adequate number of gateway scale units to avoid performance issues down the line.
Estimate the need per VPN tunnel when planning for a VPN gateway to ensure your gateway has enough aggregate throughput.
For more information on supported scale units in Virtual WAN for VPN gateway, see Azure Virtual WAN FAQ | Microsoft Learn.
Avoid having the gateway become a performance bottleneck in the long run.
Virtual WAN VPN supports many algorithm combinations. The full list of supported parameters can be found here: Virtual WAN site-to-site IPsec policies – Azure Virtual WAN | Microsoft Learn.
For optimal performance, the recommended algorithm for both IPSEC Encryption and Integrity is GCMAES256.
Optimal performance in your VPN site-to-site connections in Virtual WAN.
Regularly monitor your virtual hub gateways (VPN or ExpressRoute) to make sure they’re not overutilized and adjust the number of scale units as needed.
To do so, leverage on Virtual WAN metrics and logs. Consider configuring alerts for recommended metrics and logs. For more information, see Monitoring Virtual WAN – Best practices – Azure Virtual WAN | Microsoft Learn.
By regularly monitoring hub gateways you’re able detect potential performance bottlenecks early and act on them.
Estimating throughput requirements in advance before configuring your virtual hub capacity is important not only from a cost perspective, but also from a performance standpoint.
Moreover, while the virtual hub can automatically scale out, it is still important to regularly monitor virtual hub metrics such as ‘Virtual Hub Data Processed’ or ‘Spoke VM Utilization’.
For more information on virtual hub metrics and virtual hub metrics, see Monitoring Azure Virtual WAN – Data reference | Microsoft Learn.
These metrics can help prevent situations, such as nearing the limits of a single hub, and acting on it early by deploying an additional hub, for example.
For more information on virtual hub capacity, see About virtual hub settings – Azure Virtual WAN | Microsoft Learn.
Next steps
Now that we’ve gone through the list of configuration best practices, here’s some additional useful resources:
Virtual WAN routing deep dive – Azure Virtual WAN | Microsoft Learn
How to configure Virtual WAN Hub routing policies – Azure Virtual WAN | Microsoft Learn
Microsoft Tech Community – Latest Blogs –Read More
Partner Case Study Series | Exasol With Azure and Power BI
Exasol, a Microsoft partner enabling clients to quickly turn their data into value
Exasol, an analytics database company with offices in the United Kingdom, the United States, and Germany, is redefining what it means to work with data. Its high-performance in-memory analytics database transforms how organizations work with data on-premises, in the cloud, or both, quickly turning it into value. Exasol’s core verticals include retail and e-commerce, banking and fintech, and healthcare and life sciences. Clients have worked with Exasol when pursuing digital modernization, cloud migration, performance enhancement strategies, and legal hardware replacement. Blue Yonder, a software and consultancy company based in Scottsdale, Arizona, has used Exasol for years.
Continue reading here
**Explore all case studies or submit your own**
Microsoft Tech Community – Latest Blogs –Read More
Windows Security keeps deleting cache files from Discord app
Don’t know why but Windows Security keeps giving threat report on my Discord app, which should be quite safe to use.
Don’t know why but Windows Security keeps giving threat report on my Discord app, which should be quite safe to use. Read More
Deep Fake – what do u think about it ?
Hi, what do u think about deepfake technology ? I found this article Before you believe – how to recognize a deepfake and is it inherently evil? – Marek Jeleśniański (jelesnianski.com) Do you think that AI is more of a threat or an opportunity for development?
Hi, what do u think about deepfake technology ? I found this article Before you believe – how to recognize a deepfake and is it inherently evil? – Marek Jeleśniański (jelesnianski.com) Do you think that AI is more of a threat or an opportunity for development? Read More
Blank Report Notes View
Hello,
How can I view the entirety of a note on the blank report?
The far right column is only showing first line of any notes against tasks.
Thank you
Hello,How can I view the entirety of a note on the blank report?The far right column is only showing first line of any notes against tasks.Thank you Read More
What is the Quick___Books Connection Diagnostic Tool?
The Quick____Books Connection Diagnostic Tool is a utility provided by Intuit that helps diagnose and resolve network connectivity issues, database errors, and multi-user mode problems in Quick____Books Desktop. This tool is designed to identify and fix issues that might occur when trying to connect to the Quick___Books company file. More information you can Contact our Quick__Books ProAdvisor
The Quick____Books Connection Diagnostic Tool is a utility provided by Intuit that helps diagnose and resolve network connectivity issues, database errors, and multi-user mode problems in Quick____Books Desktop. This tool is designed to identify and fix issues that might occur when trying to connect to the Quick___Books company file. More information you can Contact our Quick__Books ProAdvisor Read More
Group expiration policy – what criteria does the policy use to determine when a group expires?
We are considering enabling group expiration on 80k+ groups. Before we do so we would like to understand which groups will be affected. There isn’t any documentation that I could find on which criteria the group expiration policy uses, and when we opened a case with Microsoft they weren’t able to tell us much other than ‘check audit logs’, which doesn’t help since that only keeps data for 90 days.
Does anyone have any insight on this one? Maybe a way to run a ‘what-if’ scenario before we kick off the policy? Thanks!
We are considering enabling group expiration on 80k+ groups. Before we do so we would like to understand which groups will be affected. There isn’t any documentation that I could find on which criteria the group expiration policy uses, and when we opened a case with Microsoft they weren’t able to tell us much other than ‘check audit logs’, which doesn’t help since that only keeps data for 90 days. Does anyone have any insight on this one? Maybe a way to run a ‘what-if’ scenario before we kick off the policy? Thanks! Read More