Introduction to AI Automation with Azure OpenAI Assistants

Intro

Welcome to the future of automation! In the world of Azure, AI assistants are becoming your trusty sidekicks, ready to tackle the repetitive tasks that once consumed your valuable time. But what if we could make these assistants even smarter? In this post, we’ll dive into the exciting realm of integrating Azure AI assistants with Logic Apps – Microsoft’s powerful workflow automation tool. Get ready to discover how this dynamic duo can transform your workflows, freeing you up to focus on the big picture and truly innovative work.

Azure OpenAI Assistants (preview)

Azure OpenAI Assistants (Preview) allows you to create AI assistants tailored to your needs through custom instructions and augmented by advanced tools like code interpreter, and custom functions. To accelerate and simplify the creation of intelligent applications, we can now enable the ability to call Logic Apps workflows through function calling in Azure OpenAI Assistants. The Assistants playground enumerates and lists all the workflows in your subscription that are eligible for function calling. Here are the requirements for these workflows:

Schema: The workflows you want to use for function calling should have a JSON schema describing the inputs and expected outputs. Using Logic Apps you can streamline and provide schema in the trigger, which would be automatically imported as a function definition.

Consumption Logic Apps: Currently supported consumption workflows.

Request trigger: Function calling requires a REST-based API. Logic Apps with a request trigger provides a REST endpoint. Therefore only workflows with a request trigger are supported for function calling.

AI Automation

So apart from the Assistants API, which we will explore in another post, we know that we can Integrate Azure Logic Apps workflows! Isn’t that amazing ? The road now is open for AI Automation and we are on the genesis of it, so let’s explore it. We need an Azure Subscription and:

Azure OpenAI in the supported regions. This demo is on Sweden Central.Logic Apps consumption Plan.

We will work in Azure OpenAI Studio and utilize the Playground. Our model deployment is GPT-4o.

The Assistants Playground offers the ability to create and save our Assistants, so we can start working and return later, open the Assistant and continue. We can find the System Message option and the three tools that enhance the Assistants with Code Interpreter, Function Calling ( Including Logic Apps) and Files upload. The following table describes the configuration elements of our Assistants:

Name Description

Assistant nameYour deployment name that is associated with a specific model.InstructionsInstructions are similar to system messages this is where you give the model guidance about how it should behave and any context it should reference when generating a response. You can describe the assistant’s personality, tell it what it should and shouldn’t answer, and tell it how to format responses. You can also provide examples of the steps it should take when answering responses.DeploymentThis is where you set which model deployment to use with your assistant.FunctionsCreate custom function definitions for the models to formulate API calls and structure data outputs based on your specificationsCode interpreterCode interpreter provides access to a sandboxed Python environment that can be used to allow the model to test and execute code.FilesYou can upload up to 20 files, with a max file size of 512 MB to use with tools. You can upload up to 10,000 files using AI Studio.

The Studio provides 2 sample Functions (Get Weather and Get Stock Price) to get an idea of the schema requirement in JSON for Function Calling. It is important to provide a clear message that makes the Assistant efficient and productive, with careful consideration since the longer the message the more Tokens are consumed.

Challenge #1 – Summarize WordPress Blog Posts

How about providing a prompt to the Assistant with a URL instructing it to summarize a WordPress blog post? It is WordPress cause we have a unified API and we only need to change the URL. We can be more strict and narrow down the scope to a specific URL but let’s see the flexibility of Logic Apps in a workflow.

We should start with the Logic App. We will generate the JSON schema directly from the Trigger which must be an HTTP request.

{
“name”: “__ALA__lgkapp002”, // Remove this for the Logic App Trigger
“description”: “Fetch the latest post from a WordPress website,summarize it, and return the summary.”,
“parameters”: {
“type”: “object”,
“properties”: {
“url”: {
“type”: “string”,
“description”: “The base URL of the WordPress site”
},
“post”: {
“type”: “string”,
“description”: “The page number”
}
},
“required”: [
“url”,
“post”
]
}
}

In the Designer this looks like this :

As you can see the Schema is the same, excluding the name which is need only in the OpenAI Assistants. We will see this detail later on. Let’s continue with the call to WordPress. An HTTP Rest API call:

And finally mandatory as it is, a Response action where we tell the Assistant that the Call was completed and bring some payload, in our case the body of the previous step:

Now it is time to open our Azure OpenAI Studio and create a new Assistant. Remember the prerequisites we discussed earlier!

From the Assistants menu create a [+New] Assistant, give it a meaningful name, select the deployment and add a System Message . For our case it could be something like : ” You are a helpful Assistant that summarizes the WordPress Blog Posts the users request, using Functions. You can utilize code interpreter in a sandbox Environment for advanced analysis and tasks if needed “. The Code interpreter here could be an overkill but we mention it to see the use of it ! Remember to save the Assistant. Now, in the Functions, do not select Logic Apps, rather stay on the custom box and add the code we presented earlier. The Assistant will understand that the Logic App named xxxx must be called, aka [“name”: “__ALA__lgkapp002“,] in the schema! In fact the Logic App is declared by 2 underscores as prefix and 2 underscores as suffix, with ALA inside and the name of the Logic App.

Let’s give our Assistant a Prompt and see what happens:

The Assistant responded pretty solidly with a meaningful summary of the post we asked for! Not bad at all for a Preview service.

Challenge #2 – Create Azure Virtual Machine based on preferences

For the purpose of this task we have activated System Assigned managed identity to the Logic App we use, and a pre-provisioned Virtual Network with a subnet as well. The Logic App must reside in the same subscription as our Azure OpenAI resource.

This is a more advanced request, but after all it translates to Logic Apps capabilities. Can we do it fast enough so the Assistant won’t time out? Yes we do, by using the Azure Resource Manager latest API which indeed is lightning fast! The process must follow the same pattern, Request – Actions – Response. The request in our case must include such input so the Logic App can carry out the tasks. The Schema should include a “name” input which tells the Assistant which Logic App to look up:

{
“name”: “__ALA__assistkp02” //remove this for the Logic App Trigger
“description”: “Create an Azure VM based on the user input”,
“parameters”: {
“type”: “object”,
“properties”: {
“name”: {
“type”: “string”,
“description”: “The name of the VM”
},
“location”: {
“type”: “string”,
“description”: “The region of the VM”
},
“size”: {
“type”: “string”,
“description”: “The size of the VM”
},
“os”: {
“type”: “string”,
“description”: “The OS of the VM”
}
},
“required”: [
“name”,
“location”,
“size”,
“os”
]
}
}

And the actual screenshot from the Trigger, observe the absence of the “name” here:

Now as we have number of options, this method allows us to keep track of everything including the user’s inputs like VM Name , VM Size, VM OS etc.. Of Course someone can expand this, since we use a default resource group and a default VNET and Subnet, but it’s also configurable! So let’s store the input into variables, we Initialize 5 variables. The name, the size, the location (which is preset for reduced complexity since we don’t create a new VNET), and we break down the OS. Let’s say the user selects Windows 10. The API expects an offer and a sku. So we take Windows 10 and create an offer variable, the same with OS we create an OS variable which is the expected sku:

if(equals(triggerBody()?[‘os’], ‘Windows 10’), ‘Windows-10’, if(equals(triggerBody()?[‘os’], ‘Windows 11’), ‘Windows-11’, ‘default-offer’))

if(equals(triggerBody()?[‘os’], ‘Windows 10’), ‘win10-22h2-pro-g2’, if(equals(triggerBody()?[‘os’], ‘Windows 11’), ‘win11-22h2-pro’, ‘default-sku’))

As you understand this is narrowed to Windows Desktop only available choices, but we can expand the Logic App to catch most well know Operating Systems.

After the Variables all we have to do is create a Public IP (optional) , a Network Interface, and finally the VM. This is the most efficient way i could make, so we won’t get complains from the API and it will complete it very fast ! Like 3 seconds fast ! The API calls are quite straightforward and everything is available in Microsoft Documentation. Let’s see an example for the Public IP:

And the Create VM action with highlight to the storage profile – OS Image setup:

Finally we need the response which can be as we like it to be. I am facilitating the Assistant’s response with an additional Action “Get Virtual Machine” that allows us to include the properties which we add in the response body:

Let’s make our request now, through the Assistants playground in Azure OpenAI Studio. Our prompt is quite clear: “Create a new VM with size=Standard_D4s_v3, location=swedencentral, os=Windows 11, name=mynewvm02”. Even if we don’t add the parameters the Assistant will ask for them as we have set in the System Message.

Pay attention to the limitation also . When we ask about the Public IP, the Assistant does not know it. Yet it informs us with a specific message, that makes sense and it is relevant to the whole operation. If we want to have a look of the time it took we will be amazed :

The sum of the time starting from the user request till the response from the Assistant is around 10 seconds. We have a limit of 10 minutes for Function Calling execution so we can built a whole Infrastructure using just our prompts.

Conclusion

In conclusion, this experiment highlights the powerful synergy between Azure AI Assistant’s Function Calling capability and the automation potential of Logic Apps. By successfully tackling two distinct challenges, we’ve demonstrated how this combination can streamline workflows, boost efficiency, and unlock new possibilities for integrating intelligent decision-making into your business processes. Whether you’re automating customer support interactions, managing data pipelines, or optimizing resource allocation, the integration of AI assistants and Logic Apps opens doors to a more intelligent and responsive future. We encourage you to explore these tools further and discover how they can revolutionize your own automation journey.

References:

Getting started with Azure OpenAI Assistants (Preview)Call Azure Logic apps as functions using Azure OpenAI AssistantsAzure OpenAI Assistants function callingAzure OpenAI Service modelsWhat is Azure Logic Apps?Azure Resource Manager – Rest Operations

​Introduction to AI Automation with Azure OpenAI Assistants IntroWelcome to the future of automation! In the world of Azure, AI assistants are becoming your trusty sidekicks, ready to tackle the repetitive tasks that once consumed your valuable time. But what if we could make these assistants even smarter? In this post, we’ll dive into the exciting realm of integrating Azure AI assistants with Logic Apps – Microsoft’s powerful workflow automation tool. Get ready to discover how this dynamic duo can transform your workflows, freeing you up to focus on the big picture and truly innovative work.Azure OpenAI Assistants (preview)Azure OpenAI Assistants (Preview) allows you to create AI assistants tailored to your needs through custom instructions and augmented by advanced tools like code interpreter, and custom functions. To accelerate and simplify the creation of intelligent applications, we can now enable the ability to call Logic Apps workflows through function calling in Azure OpenAI Assistants. The Assistants playground enumerates and lists all the workflows in your subscription that are eligible for function calling. Here are the requirements for these workflows:Schema: The workflows you want to use for function calling should have a JSON schema describing the inputs and expected outputs. Using Logic Apps you can streamline and provide schema in the trigger, which would be automatically imported as a function definition.Consumption Logic Apps: Currently supported consumption workflows.Request trigger: Function calling requires a REST-based API. Logic Apps with a request trigger provides a REST endpoint. Therefore only workflows with a request trigger are supported for function calling.AI AutomationSo apart from the Assistants API, which we will explore in another post, we know that we can Integrate Azure Logic Apps workflows! Isn’t that amazing ? The road now is open for AI Automation and we are on the genesis of it, so let’s explore it. We need an Azure Subscription and:Azure OpenAI in the supported regions. This demo is on Sweden Central.Logic Apps consumption Plan.We will work in Azure OpenAI Studio and utilize the Playground. Our model deployment is GPT-4o.The Assistants Playground offers the ability to create and save our Assistants, so we can start working and return later, open the Assistant and continue. We can find the System Message option and the three tools that enhance the Assistants with Code Interpreter, Function Calling ( Including Logic Apps) and Files upload. The following table describes the configuration elements of our Assistants:Name DescriptionAssistant nameYour deployment name that is associated with a specific model.InstructionsInstructions are similar to system messages this is where you give the model guidance about how it should behave and any context it should reference when generating a response. You can describe the assistant’s personality, tell it what it should and shouldn’t answer, and tell it how to format responses. You can also provide examples of the steps it should take when answering responses.DeploymentThis is where you set which model deployment to use with your assistant.FunctionsCreate custom function definitions for the models to formulate API calls and structure data outputs based on your specificationsCode interpreterCode interpreter provides access to a sandboxed Python environment that can be used to allow the model to test and execute code.FilesYou can upload up to 20 files, with a max file size of 512 MB to use with tools. You can upload up to 10,000 files using AI Studio.The Studio provides 2 sample Functions (Get Weather and Get Stock Price) to get an idea of the schema requirement in JSON for Function Calling. It is important to provide a clear message that makes the Assistant efficient and productive, with careful consideration since the longer the message the more Tokens are consumed.Challenge #1 – Summarize WordPress Blog PostsHow about providing a prompt to the Assistant with a URL instructing it to summarize a WordPress blog post? It is WordPress cause we have a unified API and we only need to change the URL. We can be more strict and narrow down the scope to a specific URL but let’s see the flexibility of Logic Apps in a workflow.We should start with the Logic App. We will generate the JSON schema directly from the Trigger which must be an HTTP request.   {
“name”: “__ALA__lgkapp002”, // Remove this for the Logic App Trigger
“description”: “Fetch the latest post from a WordPress website,summarize it, and return the summary.”,
“parameters”: {
“type”: “object”,
“properties”: {
“url”: {
“type”: “string”,
“description”: “The base URL of the WordPress site”
},
“post”: {
“type”: “string”,
“description”: “The page number”
}
},
“required”: [
“url”,
“post”
]
}
}   In the Designer this looks like this : As you can see the Schema is the same, excluding the name which is need only in the OpenAI Assistants. We will see this detail later on. Let’s continue with the call to WordPress. An HTTP Rest API call: And finally mandatory as it is, a Response action where we tell the Assistant that the Call was completed and bring some payload, in our case the body of the previous step: Now it is time to open our Azure OpenAI Studio and create a new Assistant. Remember the prerequisites we discussed earlier!From the Assistants menu create a [+New] Assistant, give it a meaningful name, select the deployment and add a System Message . For our case it could be something like : ” You are a helpful Assistant that summarizes the WordPress Blog Posts the users request, using Functions. You can utilize code interpreter in a sandbox Environment for advanced analysis and tasks if needed “. The Code interpreter here could be an overkill but we mention it to see the use of it ! Remember to save the Assistant. Now, in the Functions, do not select Logic Apps, rather stay on the custom box and add the code we presented earlier. The Assistant will understand that the Logic App named xxxx must be called, aka [“name”: “__ALA__lgkapp002“,] in the schema! In fact the Logic App is declared by 2 underscores as prefix and 2 underscores as suffix, with ALA inside and the name of the Logic App.Let’s give our Assistant a Prompt and see what happens:The Assistant responded pretty solidly with a meaningful summary of the post we asked for! Not bad at all for a Preview service.Challenge #2 – Create Azure Virtual Machine based on preferencesFor the purpose of this task we have activated System Assigned managed identity to the Logic App we use, and a pre-provisioned Virtual Network with a subnet as well. The Logic App must reside in the same subscription as our Azure OpenAI resource.This is a more advanced request, but after all it translates to Logic Apps capabilities. Can we do it fast enough so the Assistant won’t time out? Yes we do, by using the Azure Resource Manager latest API which indeed is lightning fast! The process must follow the same pattern, Request – Actions – Response. The request in our case must include such input so the Logic App can carry out the tasks. The Schema should include a “name” input which tells the Assistant which Logic App to look up:  {
“name”: “__ALA__assistkp02” //remove this for the Logic App Trigger
“description”: “Create an Azure VM based on the user input”,
“parameters”: {
“type”: “object”,
“properties”: {
“name”: {
“type”: “string”,
“description”: “The name of the VM”
},
“location”: {
“type”: “string”,
“description”: “The region of the VM”
},
“size”: {
“type”: “string”,
“description”: “The size of the VM”
},
“os”: {
“type”: “string”,
“description”: “The OS of the VM”
}
},
“required”: [
“name”,
“location”,
“size”,
“os”
]
}
}  And the actual screenshot from the Trigger, observe the absence of the “name” here:   Now as we have number of options, this method allows us to keep track of everything including the user’s inputs like VM Name , VM Size, VM OS etc.. Of Course someone can expand this, since we use a default resource group and a default VNET and Subnet, but it’s also configurable! So let’s store the input into variables, we Initialize 5 variables. The name, the size, the location (which is preset for reduced complexity since we don’t create a new VNET), and we break down the OS. Let’s say the user selects Windows 10. The API expects an offer and a sku. So we take Windows 10 and create an offer variable, the same with OS we create an OS variable which is the expected sku:  if(equals(triggerBody()?[‘os’], ‘Windows 10’), ‘Windows-10’, if(equals(triggerBody()?[‘os’], ‘Windows 11’), ‘Windows-11’, ‘default-offer’))     if(equals(triggerBody()?[‘os’], ‘Windows 10’), ‘win10-22h2-pro-g2’, if(equals(triggerBody()?[‘os’], ‘Windows 11’), ‘win11-22h2-pro’, ‘default-sku’))  As you understand this is narrowed to Windows Desktop only available choices, but we can expand the Logic App to catch most well know Operating Systems.After the Variables all we have to do is create a Public IP (optional) , a Network Interface, and finally the VM. This is the most efficient way i could make, so we won’t get complains from the API and it will complete it very fast ! Like 3 seconds fast ! The API calls are quite straightforward and everything is available in Microsoft Documentation. Let’s see an example for the Public IP: And the Create VM action with highlight to the storage profile – OS Image setup: Finally we need the response which can be as we like it to be. I am facilitating the Assistant’s response with an additional Action “Get Virtual Machine” that allows us to include the properties which we add in the response body: Let’s make our request now, through the Assistants playground in Azure OpenAI Studio. Our prompt is quite clear: “Create a new VM with size=Standard_D4s_v3, location=swedencentral, os=Windows 11, name=mynewvm02”. Even if we don’t add the parameters the Assistant will ask for them as we have set in the System Message. Pay attention to the limitation also . When we ask about the Public IP, the Assistant does not know it. Yet it informs us with a specific message, that makes sense and it is relevant to the whole operation. If we want to have a look of the time it took we will be amazed :The sum of the time starting from the user request till the response from the Assistant is around 10 seconds. We have a limit of 10 minutes for Function Calling execution so we can built a whole Infrastructure using just our prompts.ConclusionIn conclusion, this experiment highlights the powerful synergy between Azure AI Assistant’s Function Calling capability and the automation potential of Logic Apps. By successfully tackling two distinct challenges, we’ve demonstrated how this combination can streamline workflows, boost efficiency, and unlock new possibilities for integrating intelligent decision-making into your business processes. Whether you’re automating customer support interactions, managing data pipelines, or optimizing resource allocation, the integration of AI assistants and Logic Apps opens doors to a more intelligent and responsive future. We encourage you to explore these tools further and discover how they can revolutionize your own automation journey.References:Getting started with Azure OpenAI Assistants (Preview)Call Azure Logic apps as functions using Azure OpenAI AssistantsAzure OpenAI Assistants function callingAzure OpenAI Service modelsWhat is Azure Logic Apps?Azure Resource Manager – Rest Operations  Read More

​

Windows updates are essential for keeping your devices secure and up to date with the latest security, performance, and reliability improvements. One of the top customer requests we receive is to enable Windows updates during provisioning in the out-of-box experience (OOBE), so that devices are fully patched and ready to use as soon as they are enrolled with mobile device management (MDM).

In the coming weeks, the Windows MDM enrollment experience will be updated to automatically enable quality updates during OOBE. Quality updates are monthly updates that provide security and reliability fixes, as well as enhancements to existing features. These updates are critical for the performance and security of your devices, and we want to make sure they’re delivered as soon as possible. Please note that not every monthly quality update will be made available through the OOBE. Microsoft will determine the availability of these updates based on the value of the update and how it relates to a device setup situation.

What’s changing

With the upcoming October Windows update, all Windows 11, version 22H2 and higher, devices that are enrolled with an MDM, e.g. Microsoft Intune, will automatically download and install quality updates during OOBE. This will apply to all MDM-enrolled devices, regardless of whether they’re pre-registered with Windows Autopilot or not. The updates will be applied before the user reaches the desktop, ensuring that the device is fully patched before logging in.

The new experience will look like this:

After the device connects to the internet and checks for updates, if there are available quality updates found, the device displays a message on the updates page stating that updates are available and being installed.
The device then downloads and installs the quality updates in the background, while showing installation progress.
Once the updates are installed, the device restarts and continues to the desktop. The user then signs in to the device and the device completes enrollment.

Please note that this change only applies to quality updates. Feature updates, which are major updates that introduce new functionality, and driver updates, which provide hardware-specific fixes or enhancements will not be applied during OOBE but will be managed by your MDM according to your policies.

Impacts and what this means for you

While we believe that this change will improve the Windows enrollment experience and provide more security and reliability for your devices, we also want to make you aware of some potential impacts and what you need to do to prepare.

Additional time in OOBE

Quality update installation during OOBE adds some additional time to the device setup process, depending on when the device was most recently updated, internet speed, and device performance. We recommend notifying your vendors and customers of this additional time, and plan accordingly for your device deployment scenarios.

Organizations using temporary passwords

With the additional time for setup, if using Temporary Access Pass (TAP), the passcode may expire before the user signs onto the desktop. To avoid this, we recommend that you extend the validity period of the temporary passwords during enrollment.

Summary

There may be instances where the update is not initiated if the Windows Update for Business (WUfB) policies that block or delay updates are applied to the device before reaching the New Device Update Page (NDUP). This is particularly possible if app installations significantly delay the Enrollment Status Page (ESP).

At this time, there’s no option to control or disable quality updates during OOBE. As mentioned earlier in this blog, we’re exploring when all monthly quality updates can be available and manageable during OOBE to provide the best overall experience.

We hope that this change will improve your Windows Autopilot experience and provide more security and reliability for your devices. If you have any feedback or questions, please let us know in the comments or reach out on X @IntuneSuppTeam.

​Microsoft Tech Community – Latest Blogs –Read More 

Happy Monday and what a way to kick off the week! Just started back this morning after a 10-week Sabbatical (THANK YOU MICROFT!) and was greeted with the kickoff of the Microsoft 365 Copilot Wave 2 presented by Microsoft’s Satya Nadella and Jared Spataro. The streaming Wave 2 kickoff centered on 3 main areas of focus: 1- Copilot Pages, 2- Copilot in Microsoft 365 Apps, and 3- Copilot Agents. If you missed the session no worries. I have grabbed a link to the recording, the follow-up Copilot blog post, some individual deep dive videos as well as some additional content to help you on your Copilot Wave 2 journey.

Watch the recording of today’s announcements on LinkedIn by clicking here.
Check out the blog post “Microsoft 365 Copilot Wave 2: Pages, Python in Excel, and agents”
Watch videos on:

Copilot Agents in Sharepoint

Copilot Studio Agent Builder
Copilot Pages
Prioritize my Inbox in Outlook
Python in Excel
Narrative Builder in PowerPoint

Get powerful Microsoft 365 Copilot adoption resources available to help your organization on its Copilot journey
Microsoft 365 Copilot home page 

Thanks for visiting – Michael Gannotti   LinkedIn

​Microsoft Tech Community – Latest Blogs –Read More 

With channel meetings, all members get a meeting invite regardless of whether the organizer invites them or not.

Why even offer the ability to invite individuals in the first place?

It’s been so long, I’m starting to think this was by design and MS has no intention of fixing it.

​With channel meetings, all members get a meeting invite regardless of whether the organizer invites them or not. Why even offer the ability to invite individuals in the first place? It’s been so long, I’m starting to think this was by design and MS has no intention of fixing it.  Read More

​

For context we recieve a monthly Excel report every month that is automatically upload to our Sharepoint.

At the moment we have someone manually copy and paste the content from these newly uploaded files into a “master worksheet” that contains all the reports data in a single file. I want to know if there is a way that we that we can automate the process of updating this excl file?

The tabs and columns on all the Excel’s are exactly the same.

​For context we recieve a monthly Excel report every month that is automatically upload to our Sharepoint. At the moment we have someone manually copy and paste the content from these newly uploaded files into a “master worksheet” that contains all the reports data in a single file. I want to know if there is a way that we that we can automate the process of updating this excl file? The tabs and columns on all the Excel’s are exactly the same.  Read More

​

This Blog was Co-Authored by Don Freed – Sr. Bioinformatics Scientist, Brendan Gallagher – Head of Business Development at Sentieon, Inc. 

In our previous blog, we discussed benchmarking  the performance of Sentieon’s, DNAseq and DNAscope pipelines using Azure instances using v202112.05 of the software. Since the publication of those results, there have been significant updates to the Sentieon software. As a result, we have updated the benchmarking to use Sentieon version 202308.01. We break down the runtime and cost of the pipelines on a wide range of currently available instances. These benchmarks use publicly available datasets, and the pipeline is available on Github.

Additionally, we have worked with Sentieon to develop a Terraform template for deploym

ent of the license server.

Running Sentieon on Azure

The pipelines and scripts needed for setup used in this benchmarking are provided on GitHub.

Instance Setup

The script at misc/instance_setup.sh performs initial setup of the instance and download/installation of software packages used in the benchmark.

Input datasets

In these benchmarks, as we stated before, we use the GIAB HG002 sample sequenced on multiple sequencing platforms. Input datasets for the benchmark are recorded in the config/config.yaml. With the exception of the Element dataset, that you will have to download on your own.

We recommend downloading all the files and placing them in an azure blob storage. You can use AzCopy to transfer the required files to your own Storage account using a shared access signature with “Write” access. Then we recommend updating the configs to use a shared access signature to each file. The pipeline will automatically download input files.

Input FASTQ were obtained as previously outlined, we have added the new ONT dataset below:

ONT HPRC

https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_1_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_2_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_3_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_4_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_5_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_6_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_7_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_8_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_9_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_10_Dordo_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz
https://human-pangenomics.s3.amazonaws.com/submissions/0CB931D5-AE0C-4187-8BD8-B3A9C9BFDADE–UCSC_HG002_R1041_Duplex_Dorado/Dorado_v0.1.1/stereo_duplex/11_15_22_R1041_Duplex_HG002_11_Dorado_v0.1.1_400bps_sup_stereo_duplex_pass.fastq.gz

The input files vary in their coverage, so the datasets with FASTQ input were down-sampled to approximately 93 billion bases (~30x coverage) prior to processing with the Sentieon secondary analysis pipelines. The Ultima CRAM file was not down-sampled and is at 40x coverage as recommended by Ultima Genomics. The ONT duplex sample was not down-sampled and is at approximately 30x coverage.

The data were processed using the hg38 reference genome. The reference genome at https://giab.s3.amazonaws.com/release/references/GRCh38/GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta.gz was used for files with input in the FASTQ format. The reference genome at https://broad-references.s3.amazonaws.com/hg38/v0/Homo_sapiens_assembly38.fasta was used with the Ultima data in CRAM format, as this dataset was already aligned to this reference genome.

Running benchmarks on Azure

The script at misc/run_benchmarks.sh was used to run the benchmarks. This orchestrates the localization of the input datasets, references, model files and execution of Snakemake workflows on the machine. The workflow will down-sample the input data to be consistent to run on the Sentieon analysis workflows and will calculate variant calling accuracy against the Genome in a Bottle (GIAB) v 4.2.1 truth set. For the ARM benchmarking we didn’t run ONT and Pacbio data as minimap2 is not support by Sentieon on that architecture in version 202308.01. Support for minimap2 on ARM was added in version 202308.03 of the Sentieon software.

Improved Benchmarking with HBv3

To test the improvement of the software we wanted to retest on the HBv3 series of machines, that we previously recommended. These machines are optimized for applications that are driven by memory bandwidth, such as fluid dynamics, finite element analysis, and reservoir simulation and would be a good fit for Sentieon’s analysis pipelines. Figure 1 presents the runtime and Spot compute cost of running Sentieon’s analysis pipelines for germline variant calling across multiple sequencing technologies on Standard_HB120rs_v3 instance in US East at the time of publication.

Figure 1: Runtime and Spot compute cost of Sentieon DNAseq and DNAscope pipelines on Standard_HB120rs_v3.

Using the Standard_HB120rs_v3, we analyzed 30x Illumina NovaSeq and HiSeqX samples from FASTQ to VCF using the DNAseq and DNAscope pipelines. The DNAseq pipeline took around 28 minutes with a cost of $0.17. Sentieon’s DNAscope pipeline has been speed up and takes only 10 minutes shorter– around 18 minutes with a cost of $0.11, about 6 cents less, see Table 1

The Ultima UG100 dataset is already aligned to the reference genome and pipeline performed variant calling without alignment. The DNAscope pipeline finished in 18 minutes for Spot cost of $0.10.

Sentieon’s DNAscope LongRead pipeline for PacBio HiFi data is more computationally intensive as it includes multiple passes of variant calling along with a read-backed phasing. The DNAscope LongRead pipeline finished in 41 minutes with a Spot cost of $0.25. We add in ONT data in this round of tests, similar to the PacBio data, the ONT pipeline is more computationally involved. The DNAscope LongRead pipeline finished in 88 minutes with a Spot cost of $0.53 with the ONT long reads.

The Element Biosciences AVITI system is supported by a customized Sentieon DNAscope pipeline. Sentieon’s DNAscope pipeline for Element Biosciences finished in 21 minutes with a Spot cost of $0.13.

All run times and costs can be found in Table 1.

Sample
Pipeline
Alignment (min)
Preprocessing (min)
Variant Calling (min)
Total Runtime (min)
On Demand($)
Spot ($)

Element Aviti
DNAscope
11.05
2.30
7.39
20.74
1.241
0.121

Illumina HiSeq X
DNAseq
21.09
2.97
4.11
28.18
1.691
0.171

Illumina HiSeq X
DNAscope
9.47
1.40
7.71
18.57
1.111
0.111

Illumina NovaSeq
DNAseq
21.53
2.63
4.43
28.59
1.721
0.171

Illumina NovaSeq
DNAscope
9.74
1.39
7.78
18.92
1.141
0.111

ONT Duplex
DNAscope
32.91
N/A
55.37
88.28
5.301
0.531

PacBio HiFi
DNAscope
11.49
N/A
29.75
41.24
2.471
0.251

Ultima UG100
DNAscope
N/A
N/A
17.87
17.87
1.071
0.111

Table 1: Runtime and On Demand and Spot compute cost of Sentieon DNAseq and DNAscope pipelines on Standard_HB120rs_v3. Alignment includes alignment with Sentieon BWA-MEM for short-read data and alignment with Sentieon minimap2 for PacBio HiFi and ONT Duplex data. Preprocessing includes duplicate marking, base-quality score recalibration, and merging of multiple aligned files into a single file. Variant calling includes variant calling or variant candidate identification along with variant genotyping and filtering. Variant calling for PacBio HiFi data is implemented as a multi-stage pipeline.   All runs were in the eastus region1 Pricing is accurate at the time of publication.

Let’s compare the improvements between v202112.05 and v202308.01 of the software results based on the provided information:

1. DNAseq Pipeline Performance:

   – v202112.05: Took around 30 minutes with a cost Spot of $0.18.

   – v202308.01: Took around 28 minutes with a cost Spot of $0.17.

   – Improvement: In v202308.01, the runtime decreased by 2 minutes; and the cost decreased by $0.01.

2. DNAscope Pipeline Performance:

   – v202112.05: Took around 32 minutes with a cost of $0.19.

   – v202308.01: Improved to 19 minutes with a cost of $0.11.

   – Improvement: In v202308.01, the runtime decreased significantly to 19 minutes, and the cost decreased by $0.07.

3. DNAscope LongRead Pipeline Performance (PacBio HiFi Data):

   – v202112.05: Finished in 72 minutes with a Spot cost of $0.42.

   – v202308.01: Improved to 41 minutes with a Spot cost of $0.25.

   – Improvement: In v202308.01, decreased significantly to 41minutes, and the cost decreased by $0.17.

4. Element Biosciences AVITI System Performance:

   – v202112.05: Finished in 31 minutes with a Spot cost of $0.18.

   – v202308.01: Improved to 20 minutes with a Spot cost of $0.12.

   – Improvement: In v202308.01, the runtime decreased slightly to 20 minutes, and the cost decreased by $0.06.

Overall, in v202308.01, significant improvements were observed in the runtime and cost efficiency of the DNAscope pipeline, whereas minor fluctuations were noted in other pipeline performances. It’s also important to note that v202308.01 introduced support for ONT data in the DNAscope LongRead pipeline.

Sentieon benchmark across multiple instance families and architectures

The Sentieon pipelines and software can scale to smaller or larger instances depending on data as well as instance availability. To provide an accurate representation of performance across various architectures, we again benchmarked the Sentieon DNASeq and DNAscope pipeline with Illumina NovaSeq dataset on ARM and x86 architecture. The runtime, On Demand and Spot compute cost is shown in Figures 2 and 3 respectively. On Demand VMs are pay for compute capacity by the second, with no commitments or upfront payments. While Spot VMs are pay for unused compute capacity at a discount.

Figure 2: Runtime and Dedicated and Spot compute cost of Sentieon DNAseq pipeline across various Azure machine types using Illumina NovaSeq dataset sorted by overall runtime. Larger instances provide lower runtime, while cost is generally consistent within a family but does differ between architectures.

Figure 3: Runtime and Dedicated and Spot compute cost of Sentieon DNAscope pipeline across various Azure machine types using Illumina NovaSeq dataset sorted by overall runtime. Larger instances provide lower runtime, while cost is generally consistent within a family but does differ between architectures.

For the fastest turnaround, the Sentieon DNAseq pipeline can process the Illumina 30x NovaSeq dataset in 28 minutes on a Standard_HB120rs_v3, with a Dedicated cost of $1.72 or a Spot cost of $0.11, see Figure 2. As another cost-effective option, DNAseq can be used on the Standard_D96ads_v5 instance with an On-Demand cost of $3.38, a spot cost of $0.34 and a turnaround time of under 40 minutes, see Figure 2. The DNAscope pipeline for Standard_D96ads_v5 instance with an On-Demand cost of $2.55, a spot cost of $0.26 and a turnaround time of 31 minutes, see Figure 3. Note, for the Standard_F48s_v2, an additional external disk was used to accommodate all the test data for the analysis but wasn’t included in the overall cost.

Let’s compare the performance and cost efficiency between version v202308.01 and v202112.05:

1. DNAseq Pipeline Performance:

   – v202112.05: Processed Illumina 30x NovaSeq dataset in 30 minutes on a Standard_HB120rs_v3 with a Spot cost of $0.18.

   – v202308.01: Processes the dataset in 28 minutes on a Standard_HB120rs_v3 with a Spot cost of $0.11. Alternatively, it can be processed on a Standard_D96ads_v5 instance in under 40 minutes with a Spot cost of $0.34.

   – Improvement: The turnaround time for the Standard_HB120rs_v3 decrased slightly to 28 minutes, with a decrease in Spot cost by $0.07. Additionally, a new option is available on the Standard_D96ads_v5 instance with a slightly longer turnaround time of under 40 minutes but at a higher Spot cost of $0.34 compared to $0.11.

2. DNAscope Pipeline Performance:

   – v202112.05: Turnaround time of under 50 minutes with a Spot cost of $0.39.

   – v202308.01: Turnaround time of 31 minutes on a Standard_D96ads_v5 instance with an On-Demand cost of $2.55 and a Spot cost of $0.26.

   – Improvement: In v202308.01, the turnaround time decreased to 31 minutes, with a Spot cost of $0.26, offering improved performance and cost efficiency compared to the previous version.

3. Comparison Against ARM CPUs:

   – v202112.05: ARM runtime was within 10-20 minutes of X86 equivalent for Intel and AMD. Spot price of $0.33 for DNAscope and $0.30 for DNAseq pipeline.

   – v202308.01: ARM runtime was within 10-20 minutes of X86 equivalent for Intel and AMD. No significant difference in cost between architectures.

   – Improvement: No significant difference in cost between the architectures is noted in v202308.01, whereas in v202112.05, there was a significant difference in cost for AMD architecture compared to Intel.

Overall, in v202308.01, while the DNAseq pipeline on the Standard_HB120rs_v3 shows a slight increase in turnaround time and cost, the DNAscope pipeline on the Standard_D96ads_v5 instance demonstrates improved performance and cost efficiency compared to the previous

version. Additionally, there is no significant difference in cost between ARM and X86 architectures in v202308.01, unlike in v202112.05. We would also like to note that the order of the machine types is slightly different but not with significant changes.

We were able to also run comparison against ARM CPUs. For direct comparison we were able to use the equivalent 32 vCPU machines, but the highest available is 64 vCPU when compared to 96 vCPU in X86 (Figure 2 and 3).  In Table 2, we can see that ARM runtime was within 10-20 minutes of X86 equivalent for Intel and AMD. Additionally, Dedicated cost was comparable for DNAscope and DNAseq pipeline comparable across the board. However, this time there was not significant difference in cost between the architectures.

VM Size

Architecture

Pipeline

Total Runtime (min)

On Demand ($)

Spot ($)

D32ds_v5

x86 (Intel)

DNAscope

64.51

1.941

0.191

D32ads_v5

x86 (AMD)

DNAscope

76.95

2.111

0.211

D32pds_v5

ARM

DNAscope

82.00

1.981

0.201

D32ds_v5

x86 (Intel)

DNAseq

121.51

3.661

0.371

D32ads_v5

x86 (AMD)

DNAseq

115.12

3.161

0.321

D32pds_v5

ARM

DNAseq

123.72

2.981

0.301

Table 2: Runtime, Dedicated and Spot compute cost of Sentieon DNAseq and DNAscope pipelines on across 32cpu architectures.  All runs were in the eastus region.
 1 Pricing is accurate at the time of publication.

These results highlight the ability of the Sentieon software to scale up large instances for faster turnaround and down to smaller instances as needed. We only included a subset of potential compute, based on optimized compute-to-price ratios. However, the Sentieon tools can also be used with other machine families, based on availability in a given region.

Conclusion

Sentieon’s updated DNAseq and DNAscope pipelines are highly scalable and can be used on a variety of machine types. The software can scale up to the 120 vCPU Standard_HB120rs_v3, instances for turnaround times of 28 minutes or down to Standard_D32pds_v5 instances for better pricing on Spot instance of $0.30

If you can get Standard_HB120rs_v3 in your preferred region, it is the cheapest per run. However, if not available, all other Spot pricing options are great with the following two being your best cost advantage, Standard_D32ds_v5 and Standard_D96ds_v5. If you are looking for turnaround time, we recommend any of the 96vCPU options. Sentieon’s FASTQ to VCF pipelines can process Illumina 30x whole genomes for less than $3.60 on On Demand machines or $0.33 on Spot machines and in under 120 minutes.  Standard_D32ds_v5 process the DNAseq pipeline in for $3.66 on On Demand machines or $0.37 on Spot machines and in about 121 minutes. While on Spot machines Sentieon DNAseq is capable of processing 30x genomes from FASTQ to VCF with a Spot machine cost of less than $1.50 on a variety of machine types that we tested.

Overall, the new version of the software has decreased cost and, in some cases, decreased turnaround time, with increased performance and range of datasets it can analyze.

Readers should note that all costs represent hardware costs and don’t represent software licensing costs.

To get started with the Sentieon software on Azure, please reach out to info@sentieon.com or visit the Sentieon website at www.sentieon.com

​Microsoft Tech Community – Latest Blogs –Read More