When I execute the PowerShell command, I get a sign-in error. I have tested this in a couple of SharePoint tenants, but I’m unable to run the script. Also, previously, these scripts were working fine. 

Unable to execute the “Connect-PnPOnline” command. 

Please someone help with this.

​When I execute the PowerShell command, I get a sign-in error. I have tested this in a couple of SharePoint tenants, but I’m unable to run the script. Also, previously, these scripts were working fine.  Unable to execute the “Connect-PnPOnline” command.  Please someone help with this.     Read More

​

Introduction

If you are an enterprise customer who uses Azure for your compute needs, you might be looking for ways to optimize your spending and reduce your costs. One of the features that can help you achieve this goal is Savings Plans for Compute Insights, a flexible and easy way to save up to 65% on your compute costs by committing to a consistent amount of usage for a one-year or three-year term[1].

In this article, we will explain everything you need to know about savings plans for Compute Insights, including how they work, how they compare to reserved instances, how to use them at different levels, how to distribute the costs, and how to leverage Azure Advisor recommendations. By reading this article, you will be able to make informed decisions about whether savings plans are right for you and how to use them effectively.

What are savings plans and how do they work?

Savings Plans are a pricing model that allows you to save money on your compute costs by committing to a certain amount of usage per hour for a one-year or three-year term[2]. You can choose to pay for your savings plans monthly or upfront, and you can apply them to any eligible compute resource in Azure, such as virtual machines, Azure Kubernetes Service, Azure App Service, Azure Functions, and more.

The amount of usage you commit to is called the hourly commitment, and it is measured in dollars per hour[3]. For example, if you commit to $5 per hour for a one-year term, you will pay $5 per hour for any eligible compute usage in Azure, regardless of the region, size, or operating system of the resource. If your usage exceeds your commitment, you will pay the regular pay-as-you-go rates for the excess usage. If your usage is lower than your commitment, you will still pay the committed amount, but you can use the unused portion for future usage within the term.

Figure 1 Hourly spend distribution of compute savings plan usage

Savings plans are different from reserved instances, which are another way to save money on Azure compute by pre-purchasing a specific resource type for a one-year or three-year term. Reservations offer a higher discount rate than Savings Plans, but they are less flexible and more complex to manage[4]. Reservations are tied to a specific region, size, and operating system, and they require you to exchange or cancel them if you want to change your resource configuration[5]. Savings plans, on the other hand, are not tied to any specific resource type, and they automatically apply to any eligible compute usage in Azure, giving you more flexibility and simplicity.

One of the factors that you need to consider before purchasing a savings plan is your existing Microsoft Azure Consumption Commitment (MACC) discount, if you have one[6]. MACC is a discount program that offers you a lower pay-as-you-go rate for your Azure usage, based on your annual commitment and upfront payment. MACC discounts do not stack with savings plan discounts, so you need to compare the two options and choose the one that gives you the best savings. For example, if you have a MACC discount of 5% for your Azure usage, and you purchase a savings plan that offers you a 30% discount for your eligible compute usage, you will only get the 30% discount for the compute usage, and the 5% discount for the rest of the usage. You will not get a 35% discount for the compute usage. This example is for illustration purpose only, and the actual MACC discounts may vary depending on your individual negotiation and commitment.

Another factor that you need to consider before purchasing a savings plan is that it is a 24/7 model, which means that you will pay for your savings plan regardless of whether your compute resources are running or not. This is similar to reserved instances, which also charge you for the entire term, even if you shut down your resources. However, the difference is that savings plans are more flexible and can apply to different resources, whereas reserved instances are fixed and can only apply to the specific resource type that you purchased. Therefore, you need to estimate your usage and commitment carefully, and make sure that you can utilize your savings plan as much as possible. If you have automated shutdown patterns for your resources, you need to be aware that you cannot pick individual hours to use your savings plan, and that you will still pay for the hours that your resources are not running[7].

How to use savings plans at different levels?

Savings plans can be used at different levels in Azure, depending on your organizational structure and needs. You can purchase and manage savings plans at the subscription level, the billing account level, the management group level, or the resource group level[8].

If you purchase savings plans at the subscription level, they will only apply to the eligible compute usage within that subscription. This is the simplest and most granular way to use savings plans, as you can easily track and control your usage and costs. However, this option might not be suitable for customers who have multiple subscriptions or who want to share savings plans across different teams or departments.

If you purchase savings plans at the billing account level, they will apply to the eligible compute usage across all the subscriptions within that billing account. This is a more flexible and efficient way to use savings plans, as you can optimize your spending and utilization across multiple subscriptions. However, this option might not be suitable for customers who have multiple billing accounts or who want to have more visibility and accountability for their usage and costs.

If you purchase savings plans at the management group level, they will apply to the eligible compute usage across all the subscriptions within that management group. This is the most scalable and comprehensive way to use savings plans, as you can leverage the hierarchical structure of management groups to organize and govern your savings plans across your entire organization. However, this option might not be suitable for customers who have a flat or simple organizational structure or who want to have more granularity and flexibility for their usage and costs.

If you purchase savings plans at the resource group level, they will only apply to the eligible compute usage within that resource group. This is a more targeted and specific way to use savings plans, as you can align your Savings Plans with your resource groups, which are logical containers for your resources. However, this option might not be suitable for customers who have multiple resource groups or who want to have more scope and diversity for their usage and costs.

Microsoft recommends that you use savings plans at the level that best suits your organizational needs and goals. You can also use a combination of different levels to achieve the optimal balance of flexibility and efficiency. For example, you can use savings plans at the management group level for your baseline usage, and use savings plans at the subscription level or the resource group level for your variable or project-based usage.

How to distribute the costs for savings plans?

One of the challenges of using savings plans at the billing account level or the management group level is how to distribute the costs for savings plans among the different subscriptions or teams that benefit from them. This is especially important if you want to implement a showback or chargeback model for your internal accounting or budgeting purposes.

When you purchase a savings plan, you cannot choose which resources or subscriptions will use the savings plan. Azure automatically applies the savings plan to the resources or subscriptions that have the highest eligible compute usage and the highest pay-as-you-go rates, so that you can maximize your savings[9]. For example, if you have a savings plan of $5 per hour at the billing account level, and you have three resources with eligible compute usage of 4, 3, and 2 CPU hours per hour respectively, and pay-as-you-go rates of $1.5, $1.2, and $1 per CPU hour respectively, then Azure will apply the savings plan to the first two resources, and you will pay $2 per hour for the third resource at the pay-as-you-go rate.

If you want to distribute the costs for savings plans among the different subscriptions or teams that use them, you have two main options: showback or chargeback[10]. Showback is a method of reporting the costs for savings plans to the different subscriptions or teams, without actually billing them. This is useful for increasing the visibility and awareness of the costs and savings for savings plans, and for encouraging the optimal usage and allocation of resources. Chargeback is a method of billing the costs for savings plans to the different subscriptions or teams, based on their actual usage or a predefined allocation[11]. This is useful for implementing a fair and accurate cost recovery model, and for incentivizing the efficient and responsible use of resources.

To implement either showback or chargeback, you can use tags to identify and group the resources or subscriptions that use savings plans. Tags are key-value pairs that you can assign to any resource or subscription in Azure, and that you can use to filter and analyze your costs and savings for savings plans[12]. For example, you can tag your resources or subscriptions with the name of the team, project, or department that they belong to, and then use Cost analysis in Microsoft Cost Management to view and report the costs and savings for savings plans by tag. However, you cannot tag a Savings Plan itself, as it is not a resource, but a pricing model.

You can also use the Azure portal or the API to see which resources or subscriptions are using a savings plan, and how much they are saving[13]. In the Azure portal, you can go to the savings plans blade, and select the savings plan that you want to view[14]. You will see the details of the savings plan, such as the term, the commitment, the payment option, and the expiration date. You will also see the utilization and savings of the savings plan, and the breakdown by resource type, region, and subscription. You can also export the data to a CSV file for further analysis. Alternatively, you can use the API to query the savings plan usage and savings data programmatically[15].

How to use Azure Advisor recommendations for savings plans?

Azure Advisor is a free service that provides personalized and actionable recommendations to help you optimize your Azure resources and services[16]. One of the recommendations that Azure Advisor provides is Savings Plans for Compute Insights, which helps you identify the optimal amount of savings plans to purchase based on your historical usage and projected savings[17].

To use Azure Advisor recommendations for savings plans, you need to follow these steps:

Go to the Azure Advisor dashboard and select the Cost tab.
Review the Savings Plans for Compute Insights recommendation, which shows you the potential savings, the hourly commitment, the term, and the payment option for the recommended savings plan.
Adjust the lookback period, which is the time range that Azure Advisor uses to analyze your usage and calculate your recommendation. You can choose from 7, 30, 60, or 90 days.
Click on Purchase savings plan to go to the Azure portal and complete the purchase process.

Factors to consider before purchasing a savings plan

Before you purchase a savings plan based on the Azure Advisor recommendation or your own estimation, you should also consider some factors that might affect your decision, such as:

You cannot cancel your savings plans once you purchase them, so you should be confident about your usage and commitment before you buy them[18].
You need to calculate your break-even point, which is the minimum amount of usage that you need to achieve to make your savings plans worthwhile. You can use the Azure Pricing Calculator to estimate your break-even point[19].
You need to monitor and manage your savings plans utilization and performance, and adjust your usage or purchase additional savings plans as needed. You can use the Microsoft Cost Management dashboard and reports to track and optimize your savings plans. You can also use Azure Monitor to monitor the CPU metrics and usage of your compute resources, and to set up alerts and notifications for any anomalies or issues[20].

As an example, if you purchase a savings plan of $5 per hour for a one-year term, and you pay monthly, you will pay $3,600 per month for your savings plan, regardless of your usage. This is because you will pay $5 per hour for 24 hours a day for 30 days a month. If you pay upfront, you will pay $43,200 for the entire term, which is the same as paying $3,600 per month for 12 months. This is the total cost of your savings plan, and it does not include the cost of any other Azure services or resources that you use. You can compare this cost to the cost of your eligible compute usage at the pay-as-you-go rates, and see how much you are saving with your savings plan.

Summary and feedback

In this article, we have covered the main aspects of Savings Plans for Compute Insights, a feature that can help you save money on your Azure compute costs by committing to a consistent amount of usage for a one-year or three-year term. We have explained how Savings Plans work, how they compare to reserved instances, how to use them at different levels, how to distribute the costs, and how to leverage Azure Advisor recommendations. We have also discussed some factors that you need to consider before purchasing a savings plan, such as your existing MACC discount, your usage patterns, your break-even point, and your utilization and performance monitoring.

We hope that this article has helped you understand and use Savings Plans for Compute Insights effectively, and that you can benefit from the flexibility and simplicity that they offer. We would love to hear your feedback on your experience with savings plans, and any suggestions for improvement or enhancement. You can contact us through the Azure feedback portal or leave a comment below. Thank you for reading, and happy saving!

References

[1] What is Azure savings plans for compute? – Microsoft Cost Management | Microsoft Learn

[2] Azure Savings Plan for Compute | Microsoft Azure

[3] Choose an Azure saving plan commitment amount – Microsoft Cost Management | Microsoft Learn

[4] Decide between a savings plan and a reservation – Microsoft Cost Management | Microsoft Learn

[5] How an Azure reservation discount is applied – Microsoft Cost Management | Microsoft Learn

[6] Track your Microsoft Azure Consumption Commitment (MACC) – Microsoft Cost Management | Microsoft Learn

[7] Start/Stop VMs v2 overview | Microsoft Learn

[8] Savings plan scopes – Microsoft Cost Management | Microsoft Learn

[9] How an Azure saving plan discount is applied – Microsoft Cost Management | Microsoft Learn

[10] IT chargeback and showback – Wikipedia

[11] Charge back Azure saving plan costs – Microsoft Cost Management | Microsoft Learn

[12] Group and allocate costs using tag inheritance – Microsoft Cost Management | Microsoft Learn

[13] View Azure savings plan utilization – Microsoft Cost Management | Microsoft Learn

[14] View Azure savings plan cost and usage – Microsoft Cost Management | Microsoft Learn

[15] Benefit Utilization Summaries – List By Billing Account Id – REST API (Azure Cost Management) | Microsoft Learn

[16] Introduction to Azure Advisor – Azure Advisor | Microsoft Learn

[17] Azure savings plan recommendations – Microsoft Cost Management | Microsoft Learn

[18] Azure saving plan cancellation policies – Microsoft Cost Management | Microsoft Learn

[19] Pricing Calculator | Microsoft Azure

[20] Metrics in Azure Monitor – Azure Monitor | Microsoft Learn

​Microsoft Tech Community – Latest Blogs –Read More 

Hello, 

I have two required fields on my bookings page. 

If those are empty and I click on “book meeting”, there is no error message. 

The bookings page is just reloading itself and a potential customer doesn’t get informed, that the meeting IS NOT booked.

Is there any chance to have an error message if something isn’t filled out correctly?

​Hello,  I have two required fields on my bookings page. If those are empty and I click on “book meeting”, there is no error message. The bookings page is just reloading itself and a potential customer doesn’t get informed, that the meeting IS NOT booked. Is there any chance to have an error message if something isn’t filled out correctly?  Read More

​

Hey everyone,

I accidentally deleted some important files on my Windows 10 PC and I could’t be able to find them in the Recycle Bin. I’m wondering if there’s any way to recover them. I know Windows 10 doesn’t make this easy after files are permanently deleted, but I’ve read that there are a few methods and tools out there that might help.

Have any of you successfully recovered permanently deleted files in Windows 10 using these methods, or are there other solutions I should try? Let me know if you have any recommendations or tips!

Thanks

​Hey everyone, I accidentally deleted some important files on my Windows 10 PC and I could’t be able to find them in the Recycle Bin. I’m wondering if there’s any way to recover them. I know Windows 10 doesn’t make this easy after files are permanently deleted, but I’ve read that there are a few methods and tools out there that might help. Have any of you successfully recovered permanently deleted files in Windows 10 using these methods, or are there other solutions I should try? Let me know if you have any recommendations or tips! Thanks  Read More

​

I have a test environment witch contains two farms:

2 Sharepoint Server 2019 Enterprise (Custom Role) + 2 SQL Server 2019, Both farms are located in the same domain. The URLs of the sp.domain.local and spdr.domain.local farms correspondingly. Firewall is disabled on all VMs.

I tried to restore SSA from the SP farm to the SPDR farm, using standard Sharepoint backup tools (via the Center Administration ). For some unknown reason, after the recovery process is completed, the SSA stuck in the status “Paused for:Backup/Restore”. At the same time, the backup and recovery processes themselves are completed without errors and warnings. ULS are clean. From PowerShell, the Search Service Application itself is in the ‘Online’ status, all search components are in the ‘Active’ status. At the same time, any actions with the SSA add to its ‘External request’ status and it remains in this form until it is deleted. If you try to make an Index Reset, then the same status is also displayed for a while in the ‘Administrative Status’ field, while the index itself remains in place.

Over the past week, I’ve tried all the instructions I could find on the Internet:

$SSA.ForceResume($SSA.IsPaused())
#—
$SSA.Pause() | $SSA.Resume()
#—
Resume-SPEnterpriseSearchServiceApplication $ssa
#—
Foreach ($cs in ContentSources){$cs.ResumeCrawl()}

I tried Stop the Timer Service and cleaned the cache

According to this article I tried to change the value 130 to 0 in the search database, in the ‘MSSStatusChange’ table. In this case, SSA switched to the ‘Running’ status, but Index Reset and Full Crawl were started, but never completed (I waited almost a day).

When I created a new SSA, it accepted the status of ‘Running’, Search works as expected.
I’m not sure if CU July 2024 is to blame for the problem, but on test farms without this patch, the search service is being restored normally.

Any ideas, how I can restore SSA from backup?

​I have a test environment witch contains two farms:2 Sharepoint Server 2019 Enterprise (Custom Role) + 2 SQL Server 2019, Both farms are located in the same domain. The URLs of the sp.domain.local and spdr.domain.local farms correspondingly. Firewall is disabled on all VMs.I tried to restore SSA from the SP farm to the SPDR farm, using standard Sharepoint backup tools (via the Center Administration ). For some unknown reason, after the recovery process is completed, the SSA stuck in the status “Paused for:Backup/Restore”. At the same time, the backup and recovery processes themselves are completed without errors and warnings. ULS are clean. From PowerShell, the Search Service Application itself is in the ‘Online’ status, all search components are in the ‘Active’ status. At the same time, any actions with the SSA add to its ‘External request’ status and it remains in this form until it is deleted. If you try to make an Index Reset, then the same status is also displayed for a while in the ‘Administrative Status’ field, while the index itself remains in place.Over the past week, I’ve tried all the instructions I could find on the Internet:  $SSA.ForceResume($SSA.IsPaused())
#—
$SSA.Pause() | $SSA.Resume()
#—
Resume-SPEnterpriseSearchServiceApplication $ssa
#—
Foreach ($cs in ContentSources){$cs.ResumeCrawl()}  I tried Stop the Timer Service and cleaned the cacheAccording to this article I tried to change the value 130 to 0 in the search database, in the ‘MSSStatusChange’ table. In this case, SSA switched to the ‘Running’ status, but Index Reset and Full Crawl were started, but never completed (I waited almost a day).When I created a new SSA, it accepted the status of ‘Running’, Search works as expected.I’m not sure if CU July 2024 is to blame for the problem, but on test farms without this patch, the search service is being restored normally.Any ideas, how I can restore SSA from backup?  Read More

​

Hi There, 

I have around 50 M365 customers to manage. Previously, in each tenant I create an unlicensed global admin account ( with .onmicrosoft.com account) which shared by our team. when customer need help, we will access it.  understand that Microsoft will force all global admin to use MFA, but we have a challenge that each of the tenant will be access by multiple support from my team, if use MFA, it will affect our support efficiency. 

2.  yes, we do have a partner portal, we did use this way to access our customer’s tenant, and, currently, Microsoft force to set expiry date for the GDAP, which we require to get consent from our customer to click the link. 

I just wonder how u guys manage or use which approach to support multiple tenants.  Any idea? 

​Hi There,  I have around 50 M365 customers to manage. Previously, in each tenant I create an unlicensed global admin account ( with .onmicrosoft.com account) which shared by our team. when customer need help, we will access it.  understand that Microsoft will force all global admin to use MFA, but we have a challenge that each of the tenant will be access by multiple support from my team, if use MFA, it will affect our support efficiency.  2.  yes, we do have a partner portal, we did use this way to access our customer’s tenant, and, currently, Microsoft force to set expiry date for the GDAP, which we require to get consent from our customer to click the link.  I just wonder how u guys manage or use which approach to support multiple tenants.  Any idea?   Read More

​

Scenario

In this blog post, I will demonstrate how to leverage Azure API Management to enhance the resiliency and capacity of your OpenAI Service.
Azure API Management is a tool that assists in creating, publishing, managing, and securing APIs. It offers features like routing, caching, throttling, authentication, transformation, and more.
By utilizing Azure API Management, you can:

Distribute requests across multiple instances of the Azure OpenAI Service using the priority-based load balancing technique, which includes groups with weight distribution inside the group. This helps spread the load across various resources and regions, thereby enhancing the availability and performance of your service.
Implement the circuit breaker pattern to protect your backend service from being overwhelmed by excessive requests. This helps prevent cascading failures and improves the stability and resiliency of your service. You can configure the circuit breaker property in the backend resource and define rules for tripping the circuit breaker, such as the number or percentage of failure conditions within a specified time frame and a range of status codes indicating failures.

Diagram 1: API Management with circuit breaker implementation.

>Note: Backends in lower priority groups will only be used when all backends in higher priority groups are unavailable because circuit breaker rules are tripped.

Diagram 2: API Management load balancer with circuit breaker in action.

In the following section I will guide you through circuit breaker deployment with API Management and Azure Open AI services. you can use the same solution with native OpenAI service.

The GitHub repository for this article can be found in github.com/eladtpro/api-management-ai-policies

Prerequisites

If you don’t have an Azure subscription, create a free account before you begin.
Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.
If you prefer to run CLI reference commands locally, install the Azure CLI.
If you’re using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.
if you don’t have Azure API Management, create a new instance.
Azure OpenAI Services for the backend pool, each service should have the same model deployed with the same name and version across all the services.

Step I: Provision Azure API Management Backend Pool (bicep)

Bicep CLI

Install or Upgrade Bicep CLI.

# az bicep install
az bicep upgrade

Deploy the Backend Pool using Bicep

Login to Azure.

az login

>Important: Update the names of the backend services in the deploy.bicep file before running the next command.

Create a deployment at resource group from a remote template file, update the parameters in the file.

az deployment group create –resource-group <resource-group-name> –template-file <path-to-your-bicep-file> –name apim-deployment

>Note: You can learn more about the bicep backend resource Microsoft.ApiManagement service/backends. Also about the CircuitBreakerRule Note: The following warning may be displayed when running the above command:

>Note: The following warning may be displayed when running the above command:

/path/to/deploy.bicep(102,3) : Warning BCP035: The specified “object” declaration is missing the following required properties: “protocol”, “url”. If this is an inaccuracy in the documentation, please report it to the Bicep Team. [https://aka.ms/bicep-type-issues]

Output:

{
“id”: “<deployment-id>”,
“location”: null,
“name”: “apim-deployment”,
“properties”: {
“correlationId”: “754b1f5b-323f-4d4d-99e0-7303d8f64695”,
.
.
.
“provisioningState”: “Succeeded”,
“templateHash”: “8062591490292975426”,
“timestamp”: “2024-09-07T06:54:37.490815+00:00”,
},
“resourceGroup”: “azure-apim”,
“type”: “Microsoft.Resources/deployments”
}

>Note: To view failed operations, filter operations with the ‘Failed’ state.

az deployment operation group list –resource-group <resource-group-name> –name apim-deployment –query “[?properties.provisioningState==’Failed’]”

The following is the deploy.bicep backend circuit breaker and load balancer configuration:

resource apiManagementService ‘Microsoft.ApiManagement/service@2023-09-01-preview’ existing = {
name: apimName
}

resource backends ‘Microsoft.ApiManagement/service/backends@2023-09-01-preview’ = [for (name, i) in backendNames: {
name: name
parent: apiManagementService
properties: {
url: ‘https://${name}.openai.azure.com/openai’
protocol: ‘http’
description: ‘Backend for ${name}’
type: ‘Single’
circuitBreaker: {
rules: [
{
acceptRetryAfter: true
failureCondition: {
count: 1
interval: ‘PT10S’
statusCodeRanges: [
{
min: 429
max: 429
}
{
min: 500
max: 503
}
]
}
name: ‘${name}BreakerRule’
tripDuration: ‘PT10S’
}
]
}
}
}]

And the part for the backend pool:

resource aoailbpool ‘Microsoft.ApiManagement/service/backends@2023-09-01-preview’ = {
name: ‘openaiopool’
parent: apiManagementService
properties: {
description: ‘Load balance openai instances’
type: ‘Pool’
pool: {
services: [
{
id: ‘/backends/${backendNames[0]}’
priority: 1
weight: 1
}
{
id: ‘/backends/${backendNames[1]}’
priority: 2
weight: 1
}
{
id: ‘/backends/${backendNames[2]}’
priority: 2
weight: 1
}
]
}
}
}

Step II: Create the API Management API

>Note: The following policy can be used in existing APIs or new APIs. the important part is to set the backend service to the backend pool created in the previous step.

Option I: Add to existing API

All you need to do is to add the following set-backend-service and the retry policies for activating the Load Balancer with Circuit Breaker module:

<set-backend-service id=”lb-backend” backend-id=”openaiopool” /><retry condition=”@(context.Response.StatusCode == 429)” count=”3″ interval=”1″ first-fast-retry=”true”>
<forward-request buffer-request-body=”true” />
</retry>

Option II: Create new API

Add new API

Go to your API Management instance.
Click on APIs.
Click on Add API.
Select ‘HTTP’ API.
Give it a name and set the URL suffix to ‘openai’.

>Note: The URL suffix is the path that will be appended to the API Management URL. For example, if the API Management URL is ‘https://apim-ai-features.azure-api.net‘, the URL suffix is ‘openai’, and the full URL will be ‘https://apim-ai-features.azure-api.net/openai‘.

Add “catch all” operation

Click on the API you just created.
Click on the ‘Design’ tab.
Click on Add operation.
Set the method to ‘POST’.
Set the URL template to ‘/{*path}’.
Set the name.
Click on ‘Save’.

>Note: The ‘catch all’ operation is planned to match all OpenAI requests, we achieve this by setting the URL template to ‘/{*path}’. for example:
Base URL will be: https://my-apim.azure-api.net/openai
Postfix URL will be: /deployments/gpt-4o/chat/completions?api-version=2024-06-01
The full URL will be: https://my-apim.azure-api.net/openai/deployments/gpt-4o/chat/completions?api-version=2024-06-01

Add the Load Balancer Policy

Select the operation you just created.
Click on the ‘Design’ tab.
Click on ‘Inbound processing’ policy button ‘</>’.
Replace the existing policy with this policy (showed below).
Click on ‘Save’.

This policy is set up to distribute requests across the backend pool and retry requests if the backend service is unavailable:

<policies>
<inbound>
<base />
<set-backend-service id=”lb-backend” backend-id=”openaiopool” />
<azure-openai-token-limit tokens-per-minute=”400000″ counter-key=”@(context.Subscription.Id)” estimate-prompt-tokens=”true” tokens-consumed-header-name=”consumed-tokens” remaining-tokens-header-name=”remaining-tokens” />
<authentication-managed-identity resource=”https://cognitiveservices.azure.com/” />
<azure-openai-emit-token-metric namespace=”genaimetrics”>
<dimension name=”Subscription ID” />
<dimension name=”Client IP” value=”@(context.Request.IpAddress)” />
</azure-openai-emit-token-metric>
<set-variable name=”traceId” value=”@(Guid.NewGuid().ToString())” />
<set-variable name=”traceparentHeader” value=”@(“00” + context.Variables[“traceId”] + “-0000000000000000-01″)” />
<set-header name=”traceparent” exists-action=”skip”>
<value>@((string)context.Variables[“traceparentHeader”])</value>
</set-header>
</inbound>
<backend>
<retry condition=”@(context.Response.StatusCode == 429)” count=”3″ interval=”1″ first-fast-retry=”true”>
<forward-request buffer-request-body=”true” />
</retry>
</backend>
<outbound>
<base />
<set-header name=”backend-host” exists-action=”skip”>
<value>@(context.Request.Url.Host)</value>
</set-header>
<set-status code=”@(context.Response.StatusCode)” reason=”@(context.Response.StatusReason)” />
</outbound>
<on-error>
<base />
<set-header name=”backend-host” exists-action=”skip”>
<value>@(context.LastError.Reason)</value>
</set-header>
<set-status code=”@(context.Response.StatusCode)” reason=”@(context.LastError.Message)” />
</on-error>
</policies>

>Important: The main policies taking part of the load balancing that will distribute the requests to the backend pool created in the previous step are the following: set-backend-service: This policy sets the backend service to the backend pool created in the previous step.

<set-backend-service id=”lb-backend” backend-id=”openaiopool” />

retry: This policy retries the request if the backend service is unavailable. in case the circuit breaker gets triggered, the request will be retried immediately to the next available backend service.

>Important: The value of count should be equal to the number of backend services in the backend pool.

<retry condition=”@(context.Response.StatusCode == 429)” count=”3″ interval=”1″ first-fast-retry=”true”>
<forward-request buffer-request-body=”true” />
</retry>

Step III: Configure Monitoring

Go to your API Management instance.
Click on ‘APIs’.
Click on the API you just created.
Click on ‘Settings’.
Scroll down to ‘Diagnostics Logs’.
Check the ‘Override global’ checkbox.
Add the ‘backend-host’ and ‘Retry-After’ headers to log.
Click on ‘Save’.

>Note: The ‘backend-host‘ header is the host of the backend service that the request was actually sent to. The ‘Retry-After‘ header is the time in seconds that the client should wait before retrying the request sent by the Open AI service overriding tripDuration of the backend circuit breaker setting.

>Note: Also you can add the request and response body to the HTTP requests in the ‘Advanced Options’ section.

Step IV: Prepare the OpenAI Service

Deploy the model

>Important: In order to use the load balancer configuration seamlessly, All the OpenAI services should have the same model deployed. The model should be deployed with the same name and version across all the services.

Go to the OpenAI service.
Select the ‘Model deployments’ blade.
Click the ‘Manage Deployments’ button.
Configure the model.
Click on ‘Create’.
Repeat the above steps for all the OpenAI services, making sure that the model is deployed with the same name and version across all the services.

Set the Managed Identity

>Note: The API Management instance should have the System/User ‘Managed Identity’ set to the OpenAI service.

Go to the OpenAI service.
Select the ‘Access control (IAM)’ blade.
Click on ‘Add role assignment’.
Select the role ‘Cognitive Services OpenAI User’.
Select the API Management managed identity.
Click on ‘Review + assign’.
Repeat the above steps for all the OpenAI services.

Step V: Test the Load Balancer

>Note: Calling the API Management API will require the ‘api-key’ header to be set to the subscription key of the API Management instance.

We are going to run the Chat Completion API from the OpenAI service through the API Management API. The API Management API will distribute the requests to the backend pool created in the previous steps.

Run the Python load-test script

Execute the test python script main.py to test the load balancer and circuit breaker configuration.

python main.py –apim-name apim-ai-features –subscription-key APIM_SUBSCRIPTION_KEY –request-max-tokens 200 –workers 5 –total-requests 1000 –request-limit 30

Explanation

python main.py: This runs the main.py script.
–apim-name apim apim-ai-features: The name of the API Management.
–key APIM_SUBSCRIPTION_KEY: This passes the API subscription key.
–request-max-tokens 200: The maximum number of tokens to generate per request in the completion (optional, as it defaults to 200).
–workers 5: The number of parallel requests to send (optional, as it defaults to 20).
–total_requests 1000: This sets the total number of requests to 1000 (optional, as it defaults to 1000).
–request-limit 30: The number of requests to send per second (optional, as it defaults to 20).

>Note: You can adjust the values of –batch_size and –total_requests as needed. If you omit them, the script will use the default values specified in the argparse configuration.

Test Results

ApiManagementGatewayLogs
| where OperationId == “chat-completion”
| summarize CallCount = count() by BackendId, BackendUrl
| project BackendId, BackendUrl, CallCount
| order by CallCount desc
| render barchart

Conclusion

In conclusion, leveraging Azure API Management significantly enhances the resiliency and capacity of Azure OpenAI service by distributing requests across multiple instances and implementing load-balancer with retry/circuit-breaker patterns.
These strategies improve service availability, performance, and stability. To read more look in Backends in API Management.

References

Azure API Management

Azure API Management terminology
API Management policy reference
API Management policy expressions
Backends in API Management
Error handling in API Management policies

Azure Tech Community

AI Hub Gateway Landing Zone accelerator

​Microsoft Tech Community – Latest Blogs –Read More 

Retrieval Augmented Generation (RAG) is a popular technique to get LLMs to provide answers that are grounded in a data source. What do you do when your knowledge base includes images, like graphs or photos? By adding multimodal models into your RAG flow, you can get answers based off image sources, too!

Our most popular RAG solution accelerator, azure-search-openai-demo, now has an optional feature for RAG on image sources. In the example question below, the app answers a question that requires correctly interpreting a bar graph: 

This blog post will walk through the changes we made to enable multimodal RAG, both so that developers using the solution accelerator can understand how it works, and so that developers using other RAG solutions can bring in multimodal support. 

First let’s talk about two essential ingredients: multimodal LLMs and multimodal embedding models. 

Multimodal LLMs 

Azure now offers multiple multimodal LLMs: gpt-4o and gpt-4o-mini, through the Azure OpenAI service, and Phi-3.5-vision-instruct, through the Azure AI Model Catalog. These models allow you to send in both images and text, and return text responses. (In the future, we may have LLMs that take audio input and return non-text inputs!)

For example, an API call to the gpt-4o model can contain a question along with an image URL: 

{
“role”: “user”,
“content”: [
{
“type”: “text”,
“text”: “Whats in this image?”
},
{
“type”: “image_url”,
“image_url”: { “url”: “https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg” }
}
]
}

Those image URLs can be specified as full HTTP URLs, if the image happens to be available on the public web, or they can be specified as base-64 encoded Data URIs, which is particularly helpful for privately stored images.

For more examples working with gpt-4o, check out openai-chat-vision-quickstart, a repo which can deploy a simple Chat+Vision app to Azure, plus includes Jupyter notebooks showcasing scenarios. 

Multimodal embedding models 

Azure also offers a multimodal embedding API, as part of the Azure AI Vision APIs, that can compute embeddings in a multimodal space for both text and images. The API uses the state-of-the-art Florence model from Microsoft Research. 

For example, this API call returns the embedding vector for an image: 

curl.exe -v -X POST “https://<endpoint>/computervision/retrieval:vectorizeImage?api-version=2024-02-01-preview&model-version=2023-04-15”
–data-ascii ” { ‘url’:’https://learn.microsoft.com/azure/ai-services/computer-vision/media/quickstarts/presentation.png’ }”

Once we have the ability to embed both images and text in the same embedding space, we can use vector search to find images that are similar to a user’s query. For an example, check out this notebook that setups a basic multimodal search of images using Azure AI Search. 
 

Multimodal RAG 

With those two multimodal models, we were able to give our RAG solution the ability to include image sources in both the retrieval and answering process.

At a high-level, we made the following changes: 

Search index: We added a new field to the Azure AI Search index to store the embedding returned by the multimodal Azure AI Vision API (while keeping the existing field that stores the OpenAI text embeddings). 
Data ingestion: In addition to our usual PDF ingestion flow, we also convert each PDF document page to an image, store that image with the filename rendered on top, and add the embedding to the index. 
Question answering: We search the index using both the text and multimodal embeddings. We send both the text and the image to gpt-4o, and ask it to answer the question based on both kinds of sources. 
Citations: The frontend displays both image sources and text sources, to help users understand how the answer was generated. 

Let’s dive deeper into each of the changes above. 

Search index 

For our standard RAG on documents approach, we use an Azure AI search index that stores the following fields: 

content: The extracted text content from Azure Document Intelligence, which can process a wide range of files and can even OCR images inside files. 
sourcefile: The filename of the document 
sourcepage: The filename with page number, for more precise citations
embedding: A vector field with 1536 dimensions, to store the embedding of the content field, computed using text-only OpenAI ada-002 model.

For RAG on images, we add an additional field:

imageEmbedding: A vector field with 1024 dimensions, to store the embedding of the image version of the document page, computed using the AI Vision vectorizeImage API endpoint.

Data ingestion 

For our standard RAG approach, data ingestion involves these steps: 

Use Azure Document Intelligence to extract text out of a document 
Use a splitting strategy to chunk the text into sections. This is necessary in order to keep chunk sizes at a reasonable size, as sending too much content to an LLM at once tends to reduce answer quality. 
Upload the original file to Azure Blob storage. 
Compute ada-002 embeddings for the content field. 
Add each chunk to the Azure AI search index. 

For RAG on images,  we add two additional steps before indexing: uploading an image version of each document page to Blob Storage and computing multi-modal embeddings for each image. 

Generating citable images 

The images are not just a direct copy of the document page. Instead, they contain the original document filename written in the top left corner of the image, like so: 

This crucial step will enable the GPT vision model to later provide citations in its answers. From a technical perspective, we achieved this by first using the PyMuPDF Python package to convert documents to images, then using the Pillow Python package to add a top border to the image and write the filename there.

Question answering 

Now that our Blob storage container has citable images and our AI search index has multi-modal embeddings, users can start to ask questions about images. 

Our RAG app has two primary question asking flows, one for “single-turn” questions, and the other for “multi-turn” questions which incorporates as much conversation history that can fit in the context window. To simplify this explanation, we’ll focus on the single-turn flow.  

Our single-turn RAG on documents flow looks like: 

Receive a user question from the frontend. 
Compute an embedding for the user question using the OpenAI ada-002 model. 
Use the user question to fetch matching documents from the Azure AI search index, using a hybrid search that does a keyword search on the text and a vector search on the question embedding. 
Pass the resulting document chunks and the original user question to the gpt-3.5 model, with a system prompt that instructs it to adhere to the sources and provide citations with a certain format. 

Our single-turn RAG on documents-plus-images flows looks like this: 

Receive a user question from the frontend. 
Compute an embedding for the user question using the OpenAI ada-002 model AND an additional embedding using the AI Vision API multimodal model. 
Use the user question to fetch matching documents from the Azure AI search index, using a hybrid multivector search that also searches on the imageEmbedding field using the additional embedding. This way, the underlying vector search algorithm will find results that are both similar semantically to the text of the document but also similar semantically to any images in the document (e.g. “what trends are increasing?” could match a chart with a line going up and to the right). 
For each document chunk returned in the search results, convert the Blob image URL into a base64 data-encoded URI. Pass both the text content and the image URIs to a GPT vision model, with this prompt that describes how to find and format citations: The documents contain text, graphs, tables and images.

Each image source has the file name in the top left corner of the image with coordinates (10,10) pixels and is in the format SourceFileName:<file_name>

Each text source starts in a new line and has the file name followed by colon and the actual information. Always include the source name from the image or text for each fact you use in the response in the format: [filename]

Answer the following question using only the data provided in the sources below.

The text and image source can be the same file name, don’t use the image title when citing the image source, only use the file name as mentioned.  

Now, users can ask questions where the answers are entirely contained in the images and get correct answers! This can be a great fit for diagram-heavy domains, like finance. 

Considerations 

We have seen some really exciting uses of this multimodal RAG approach, but there is much to explore to improve the experience.

More file types: Our repository only implements image generation for PDFs, but developers are now ingesting many more formats, both image files like PNG and JPEG as well as non-image files like HTML, docx, etc. We’d love help from the community in bringing support for multimodal RAG to more file formats. 

More selective embeddings: Our ingestion flow uploads images for *every* PDF page, but many pages may be lacking in visual content, and that can negatively affect vector search results. For example, if your PDF contains completely blank pages, and the index stored the embeddings for those, we have found that vector searches often retrieve those blank pages. Perhaps in the multimodal space, “blankness” is considered similar to everything. We’ve considered approaches like using a vision model in the ingestion phase to decide whether an image is meaningful, or using that model to write a very descriptive caption for images instead of storing the image embeddings themselves.

Image extraction: Another approach would be to extract images from document pages, and store each image separately. That would be helpful for documents where the pages contain multiple distinct images with different purposes, since then the LLM would be able to focus more on only the most relevant image.

We would love your help in experimenting with RAG on images, sharing how it works for your domain, and suggesting what we can improve. Head over to our repo and follow the steps for deploying with the optional GPT vision feature enabled!

​Microsoft Tech Community – Latest Blogs –Read More 

Introduction

This post is part of the SQL Database series that I am compiling. This specific topic assumes that you have already built a .dacpac file via Azure DevOps YAML Pipeline and are ready to now deploy your .dacpac to Azure. Congratulations!  If you’d like to follow along all source code is in my GitHub repository.

PreReqs

To be successful here we’d require some items to be setup:

An Azure SQL Server and Database already deployed in Azure
An Azure DevOps Service Connection that has access to deploy the database (for me I like to have the Service Connection be a member of the Entra SQL Admin group, more on that later)
A .dacpac built by ADO and ready to publish. I used the SDK style project to create mine, you could use other methods as long as you have the .dacpac file.
Network connectivity to the Database. For this specific example we will be using an Azure SQL instance and leverage MS Hosted Azure DevOps Agents. Variations of this process is possible leveraging either a Windows self hosted build agents or the newer Managed DevOps Pools. 

Deploy Steps

When writing one of these I have found it can be helpful to write out the individual steps required for our build. In our case it will consist of:

Download the pipeline artifact
Open up on the Azure SQL Server Firewall to the Agent
Deploy .dacpac
Delete the Azure SQL Server Firewall rule

The good news here is that first, the job will automatically download the pipeline artifact. To reiterate this will download the .dacpac which was built in the previous stage so that subsequent jobs can leverage it for deployments to one or more environments. The second piece of good news is the opening of the Azure SQL Server Firewall rules can be handled by SqlAzureDacpacDeployment@1 task. In addition, there is the option to delete the firewall rule after the task has been completed.

So, this means we effectively just need a single job in our deployment stage!

SqlAzureDacpacDeployment@1

Here is the YAML code for the job to handle the deployment:

jobs:
– deployment: sqlmoveemecicd_app_dev_eus
environment:
name: dev
dependsOn: []
strategy:
runOnce:
deploy:
steps:
– task: SqlAzureDacpacDeployment@1
displayName: Publish sqlmoveme on sql-moveme-dev2-eus.database.windows.net
inputs:
DeploymentAction: Publish
azureSubscription: [Insert Service Connection Name]
AuthenticationType: servicePrincipal
ServerName: [SQL Server Destination]
DatabaseName: [SQL Database Destination]
deployType: DacpacTask
DacpacFile: $(Agent.BuildDirectory)sqlmoveme***.dacpac
AdditionalArguments: ”
DeleteFirewallRule: True

So, an item here to discuss which is a bit of a pre-requisite when discussing Azure DevOps YAML Pipelines is the deployment job concept. I do cover this in a previous post in the YAML Pipeline series on Azure DevOps Pipelines: Tasks, Jobs, Stages. Suffice it to say deployment jobs are special types of jobs in Azure DevOps which are to be leveraged for the actual deployment of artifacts and one of the key capabilities of deployment jobs is the ability to tie them to an environment. Environments can have gates which is a set of criteria that can include manual or automatic checks prior to deployment.

Lets take a step back and talk a little more on some of the requirements. First, we need to establish authentication from Azure DevOps to our Azure SQL Server. The most secure way to do this would be through Entra Authentication. The credentials we will be using would be the service principal associated with an Azure DevOps Service connection. This connection will either have credentials stored as part of the App Registration or leveraged workload identity federation.

Personally, I would recommend using the workload identity federation process as this will eliminate the need for a secret. This Service Connection can be the same one used to deploy other resources in the environment, though I understand and respect the separation of data and management plane activities so a separate one specific to the database is acceptable. If you’d rather not use Entra Auth for authenticating to the database, you can alternatively pass credentials stored in a Variable Group, though usually it’s a good idea not to use passwords when possible.

So now that we know how and what we are going to authenticate with it’s time to go over how the access to the service account would be provisioned. When configuring an Azure SQL Server one can designate an Entra Security group as the Admin.

Below is a screenshot showing that the Microsoft Entra Admin ID has been granted to an Entra Security Group. If this is a new concept please follow up with Using automation to set up the Microsoft Entra admin for SQL Server – SQL Server | Microsoft Learn. Additionally, here is a great walkthrough put together by MVP Stephan van Rooij, Azure SQL and Entra ID authentication, tips from the field.  

The service principle being used for the deployment is in turn added to this group. Thus, our deployment will have full access to deploy the .dacpac to the Azure SQL Server. Additionally, I have Microsoft Entra Authentication only configured; this is considered a best practice as SQL user credentials expose a potential credential liability. If new to this concept feel free to read more on Microsoft Entra-only authentication – Azure SQL Database & Azure SQL Managed Instance & Azure Synapse Analytics | Microsoft Learn

Results

After we add this deployment stage to our previous build stage our results in ADO will look like:

A two stage pipeline where the first stage will generate an artifact of our .dacpac and the second stage which will take the .dacpac produced in the first stage and deploy it. A complete YAML definition of this pipeline can be found on my GitHub repository.

Next Steps

Now that we have covered how to effectively build .sqlproj into a .dacpac anddeploy said .dacpac to Azure our next step will be to deploy tto multiple environments via different configurations! Feel free to subscribe to this series on SQL Databases alternatively if you like my posts feel free to follow me.

​Microsoft Tech Community – Latest Blogs –Read More