Scenario

In this blog post, I will demonstrate how to leverage Azure API Management to enhance the resiliency and capacity of your OpenAI Service.
Azure API Management is a tool that assists in creating, publishing, managing, and securing APIs. It offers features like routing, caching, throttling, authentication, transformation, and more.
By utilizing Azure API Management, you can:

Distribute requests across multiple instances of the Azure OpenAI Service using the priority-based load balancing technique, which includes groups with weight distribution inside the group. This helps spread the load across various resources and regions, thereby enhancing the availability and performance of your service.
Implement the circuit breaker pattern to protect your backend service from being overwhelmed by excessive requests. This helps prevent cascading failures and improves the stability and resiliency of your service. You can configure the circuit breaker property in the backend resource and define rules for tripping the circuit breaker, such as the number or percentage of failure conditions within a specified time frame and a range of status codes indicating failures.

Diagram 1: API Management with circuit breaker implementation.

>Note: Backends in lower priority groups will only be used when all backends in higher priority groups are unavailable because circuit breaker rules are tripped.

Diagram 2: API Management load balancer with circuit breaker in action.

In the following section I will guide you through circuit breaker deployment with API Management and Azure Open AI services. you can use the same solution with native OpenAI service.

The GitHub repository for this article can be found in github.com/eladtpro/api-management-ai-policies

Prerequisites

If you don’t have an Azure subscription, create a free account before you begin.
Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.
If you prefer to run CLI reference commands locally, install the Azure CLI.
If you’re using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.
if you don’t have Azure API Management, create a new instance.
Azure OpenAI Services for the backend pool, each service should have the same model deployed with the same name and version across all the services.

Step I: Provision Azure API Management Backend Pool (bicep)

Bicep CLI

Install or Upgrade Bicep CLI.

# az bicep install
az bicep upgrade

Deploy the Backend Pool using Bicep

Login to Azure.

az login

>Important: Update the names of the backend services in the deploy.bicep file before running the next command.

Create a deployment at resource group from a remote template file, update the parameters in the file.

az deployment group create –resource-group <resource-group-name> –template-file <path-to-your-bicep-file> –name apim-deployment

>Note: You can learn more about the bicep backend resource Microsoft.ApiManagement service/backends. Also about the CircuitBreakerRule Note: The following warning may be displayed when running the above command:

>Note: The following warning may be displayed when running the above command:

/path/to/deploy.bicep(102,3) : Warning BCP035: The specified “object” declaration is missing the following required properties: “protocol”, “url”. If this is an inaccuracy in the documentation, please report it to the Bicep Team. [https://aka.ms/bicep-type-issues]

Output:

{
“id”: “<deployment-id>”,
“location”: null,
“name”: “apim-deployment”,
“properties”: {
“correlationId”: “754b1f5b-323f-4d4d-99e0-7303d8f64695”,
.
.
.
“provisioningState”: “Succeeded”,
“templateHash”: “8062591490292975426”,
“timestamp”: “2024-09-07T06:54:37.490815+00:00”,
},
“resourceGroup”: “azure-apim”,
“type”: “Microsoft.Resources/deployments”
}

>Note: To view failed operations, filter operations with the ‘Failed’ state.

az deployment operation group list –resource-group <resource-group-name> –name apim-deployment –query “[?properties.provisioningState==’Failed’]”

The following is the deploy.bicep backend circuit breaker and load balancer configuration:

resource apiManagementService ‘Microsoft.ApiManagement/service@2023-09-01-preview’ existing = {
name: apimName
}

resource backends ‘Microsoft.ApiManagement/service/backends@2023-09-01-preview’ = [for (name, i) in backendNames: {
name: name
parent: apiManagementService
properties: {
url: ‘https://${name}.openai.azure.com/openai’
protocol: ‘http’
description: ‘Backend for ${name}’
type: ‘Single’
circuitBreaker: {
rules: [
{
acceptRetryAfter: true
failureCondition: {
count: 1
interval: ‘PT10S’
statusCodeRanges: [
{
min: 429
max: 429
}
{
min: 500
max: 503
}
]
}
name: ‘${name}BreakerRule’
tripDuration: ‘PT10S’
}
]
}
}
}]

And the part for the backend pool:

resource aoailbpool ‘Microsoft.ApiManagement/service/backends@2023-09-01-preview’ = {
name: ‘openaiopool’
parent: apiManagementService
properties: {
description: ‘Load balance openai instances’
type: ‘Pool’
pool: {
services: [
{
id: ‘/backends/${backendNames[0]}’
priority: 1
weight: 1
}
{
id: ‘/backends/${backendNames[1]}’
priority: 2
weight: 1
}
{
id: ‘/backends/${backendNames[2]}’
priority: 2
weight: 1
}
]
}
}
}

Step II: Create the API Management API

>Note: The following policy can be used in existing APIs or new APIs. the important part is to set the backend service to the backend pool created in the previous step.

Option I: Add to existing API

All you need to do is to add the following set-backend-service and the retry policies for activating the Load Balancer with Circuit Breaker module:

<set-backend-service id=”lb-backend” backend-id=”openaiopool” /><retry condition=”@(context.Response.StatusCode == 429)” count=”3″ interval=”1″ first-fast-retry=”true”>
<forward-request buffer-request-body=”true” />
</retry>

Option II: Create new API

Add new API

Go to your API Management instance.
Click on APIs.
Click on Add API.
Select ‘HTTP’ API.
Give it a name and set the URL suffix to ‘openai’.

>Note: The URL suffix is the path that will be appended to the API Management URL. For example, if the API Management URL is ‘https://apim-ai-features.azure-api.net‘, the URL suffix is ‘openai’, and the full URL will be ‘https://apim-ai-features.azure-api.net/openai‘.

Add “catch all” operation

Click on the API you just created.
Click on the ‘Design’ tab.
Click on Add operation.
Set the method to ‘POST’.
Set the URL template to ‘/{*path}’.
Set the name.
Click on ‘Save’.

>Note: The ‘catch all’ operation is planned to match all OpenAI requests, we achieve this by setting the URL template to ‘/{*path}’. for example:
Base URL will be: https://my-apim.azure-api.net/openai
Postfix URL will be: /deployments/gpt-4o/chat/completions?api-version=2024-06-01
The full URL will be: https://my-apim.azure-api.net/openai/deployments/gpt-4o/chat/completions?api-version=2024-06-01

Add the Load Balancer Policy

Select the operation you just created.
Click on the ‘Design’ tab.
Click on ‘Inbound processing’ policy button ‘</>’.
Replace the existing policy with this policy (showed below).
Click on ‘Save’.

This policy is set up to distribute requests across the backend pool and retry requests if the backend service is unavailable:

<policies>
<inbound>
<base />
<set-backend-service id=”lb-backend” backend-id=”openaiopool” />
<azure-openai-token-limit tokens-per-minute=”400000″ counter-key=”@(context.Subscription.Id)” estimate-prompt-tokens=”true” tokens-consumed-header-name=”consumed-tokens” remaining-tokens-header-name=”remaining-tokens” />
<authentication-managed-identity resource=”https://cognitiveservices.azure.com/” />
<azure-openai-emit-token-metric namespace=”genaimetrics”>
<dimension name=”Subscription ID” />
<dimension name=”Client IP” value=”@(context.Request.IpAddress)” />
</azure-openai-emit-token-metric>
<set-variable name=”traceId” value=”@(Guid.NewGuid().ToString())” />
<set-variable name=”traceparentHeader” value=”@(“00” + context.Variables[“traceId”] + “-0000000000000000-01″)” />
<set-header name=”traceparent” exists-action=”skip”>
<value>@((string)context.Variables[“traceparentHeader”])</value>
</set-header>
</inbound>
<backend>
<retry condition=”@(context.Response.StatusCode == 429)” count=”3″ interval=”1″ first-fast-retry=”true”>
<forward-request buffer-request-body=”true” />
</retry>
</backend>
<outbound>
<base />
<set-header name=”backend-host” exists-action=”skip”>
<value>@(context.Request.Url.Host)</value>
</set-header>
<set-status code=”@(context.Response.StatusCode)” reason=”@(context.Response.StatusReason)” />
</outbound>
<on-error>
<base />
<set-header name=”backend-host” exists-action=”skip”>
<value>@(context.LastError.Reason)</value>
</set-header>
<set-status code=”@(context.Response.StatusCode)” reason=”@(context.LastError.Message)” />
</on-error>
</policies>

>Important: The main policies taking part of the load balancing that will distribute the requests to the backend pool created in the previous step are the following: set-backend-service: This policy sets the backend service to the backend pool created in the previous step.

<set-backend-service id=”lb-backend” backend-id=”openaiopool” />

retry: This policy retries the request if the backend service is unavailable. in case the circuit breaker gets triggered, the request will be retried immediately to the next available backend service.

>Important: The value of count should be equal to the number of backend services in the backend pool.

<retry condition=”@(context.Response.StatusCode == 429)” count=”3″ interval=”1″ first-fast-retry=”true”>
<forward-request buffer-request-body=”true” />
</retry>

Step III: Configure Monitoring

Go to your API Management instance.
Click on ‘APIs’.
Click on the API you just created.
Click on ‘Settings’.
Scroll down to ‘Diagnostics Logs’.
Check the ‘Override global’ checkbox.
Add the ‘backend-host’ and ‘Retry-After’ headers to log.
Click on ‘Save’.

>Note: The ‘backend-host‘ header is the host of the backend service that the request was actually sent to. The ‘Retry-After‘ header is the time in seconds that the client should wait before retrying the request sent by the Open AI service overriding tripDuration of the backend circuit breaker setting.

>Note: Also you can add the request and response body to the HTTP requests in the ‘Advanced Options’ section.

Step IV: Prepare the OpenAI Service

Deploy the model

>Important: In order to use the load balancer configuration seamlessly, All the OpenAI services should have the same model deployed. The model should be deployed with the same name and version across all the services.

Go to the OpenAI service.
Select the ‘Model deployments’ blade.
Click the ‘Manage Deployments’ button.
Configure the model.
Click on ‘Create’.
Repeat the above steps for all the OpenAI services, making sure that the model is deployed with the same name and version across all the services.

Set the Managed Identity

>Note: The API Management instance should have the System/User ‘Managed Identity’ set to the OpenAI service.

Go to the OpenAI service.
Select the ‘Access control (IAM)’ blade.
Click on ‘Add role assignment’.
Select the role ‘Cognitive Services OpenAI User’.
Select the API Management managed identity.
Click on ‘Review + assign’.
Repeat the above steps for all the OpenAI services.

Step V: Test the Load Balancer

>Note: Calling the API Management API will require the ‘api-key’ header to be set to the subscription key of the API Management instance.

We are going to run the Chat Completion API from the OpenAI service through the API Management API. The API Management API will distribute the requests to the backend pool created in the previous steps.

Run the Python load-test script

Execute the test python script main.py to test the load balancer and circuit breaker configuration.

python main.py –apim-name apim-ai-features –subscription-key APIM_SUBSCRIPTION_KEY –request-max-tokens 200 –workers 5 –total-requests 1000 –request-limit 30

Explanation

python main.py: This runs the main.py script.
–apim-name apim apim-ai-features: The name of the API Management.
–key APIM_SUBSCRIPTION_KEY: This passes the API subscription key.
–request-max-tokens 200: The maximum number of tokens to generate per request in the completion (optional, as it defaults to 200).
–workers 5: The number of parallel requests to send (optional, as it defaults to 20).
–total_requests 1000: This sets the total number of requests to 1000 (optional, as it defaults to 1000).
–request-limit 30: The number of requests to send per second (optional, as it defaults to 20).

>Note: You can adjust the values of –batch_size and –total_requests as needed. If you omit them, the script will use the default values specified in the argparse configuration.

Test Results

ApiManagementGatewayLogs
| where OperationId == “chat-completion”
| summarize CallCount = count() by BackendId, BackendUrl
| project BackendId, BackendUrl, CallCount
| order by CallCount desc
| render barchart

Conclusion

In conclusion, leveraging Azure API Management significantly enhances the resiliency and capacity of Azure OpenAI service by distributing requests across multiple instances and implementing load-balancer with retry/circuit-breaker patterns.
These strategies improve service availability, performance, and stability. To read more look in Backends in API Management.

References

Azure API Management

Azure API Management terminology
API Management policy reference
API Management policy expressions
Backends in API Management
Error handling in API Management policies

Azure Tech Community

AI Hub Gateway Landing Zone accelerator

​Microsoft Tech Community – Latest Blogs –Read More 

Retrieval Augmented Generation (RAG) is a popular technique to get LLMs to provide answers that are grounded in a data source. What do you do when your knowledge base includes images, like graphs or photos? By adding multimodal models into your RAG flow, you can get answers based off image sources, too!

Our most popular RAG solution accelerator, azure-search-openai-demo, now has an optional feature for RAG on image sources. In the example question below, the app answers a question that requires correctly interpreting a bar graph: 

This blog post will walk through the changes we made to enable multimodal RAG, both so that developers using the solution accelerator can understand how it works, and so that developers using other RAG solutions can bring in multimodal support. 

First let’s talk about two essential ingredients: multimodal LLMs and multimodal embedding models. 

Multimodal LLMs 

Azure now offers multiple multimodal LLMs: gpt-4o and gpt-4o-mini, through the Azure OpenAI service, and Phi-3.5-vision-instruct, through the Azure AI Model Catalog. These models allow you to send in both images and text, and return text responses. (In the future, we may have LLMs that take audio input and return non-text inputs!)

For example, an API call to the gpt-4o model can contain a question along with an image URL: 

{
“role”: “user”,
“content”: [
{
“type”: “text”,
“text”: “Whats in this image?”
},
{
“type”: “image_url”,
“image_url”: { “url”: “https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg” }
}
]
}

Those image URLs can be specified as full HTTP URLs, if the image happens to be available on the public web, or they can be specified as base-64 encoded Data URIs, which is particularly helpful for privately stored images.

For more examples working with gpt-4o, check out openai-chat-vision-quickstart, a repo which can deploy a simple Chat+Vision app to Azure, plus includes Jupyter notebooks showcasing scenarios. 

Multimodal embedding models 

Azure also offers a multimodal embedding API, as part of the Azure AI Vision APIs, that can compute embeddings in a multimodal space for both text and images. The API uses the state-of-the-art Florence model from Microsoft Research. 

For example, this API call returns the embedding vector for an image: 

curl.exe -v -X POST “https://<endpoint>/computervision/retrieval:vectorizeImage?api-version=2024-02-01-preview&model-version=2023-04-15”
–data-ascii ” { ‘url’:’https://learn.microsoft.com/azure/ai-services/computer-vision/media/quickstarts/presentation.png’ }”

Once we have the ability to embed both images and text in the same embedding space, we can use vector search to find images that are similar to a user’s query. For an example, check out this notebook that setups a basic multimodal search of images using Azure AI Search. 
 

Multimodal RAG 

With those two multimodal models, we were able to give our RAG solution the ability to include image sources in both the retrieval and answering process.

At a high-level, we made the following changes: 

Search index: We added a new field to the Azure AI Search index to store the embedding returned by the multimodal Azure AI Vision API (while keeping the existing field that stores the OpenAI text embeddings). 
Data ingestion: In addition to our usual PDF ingestion flow, we also convert each PDF document page to an image, store that image with the filename rendered on top, and add the embedding to the index. 
Question answering: We search the index using both the text and multimodal embeddings. We send both the text and the image to gpt-4o, and ask it to answer the question based on both kinds of sources. 
Citations: The frontend displays both image sources and text sources, to help users understand how the answer was generated. 

Let’s dive deeper into each of the changes above. 

Search index 

For our standard RAG on documents approach, we use an Azure AI search index that stores the following fields: 

content: The extracted text content from Azure Document Intelligence, which can process a wide range of files and can even OCR images inside files. 
sourcefile: The filename of the document 
sourcepage: The filename with page number, for more precise citations
embedding: A vector field with 1536 dimensions, to store the embedding of the content field, computed using text-only OpenAI ada-002 model.

For RAG on images, we add an additional field:

imageEmbedding: A vector field with 1024 dimensions, to store the embedding of the image version of the document page, computed using the AI Vision vectorizeImage API endpoint.

Data ingestion 

For our standard RAG approach, data ingestion involves these steps: 

Use Azure Document Intelligence to extract text out of a document 
Use a splitting strategy to chunk the text into sections. This is necessary in order to keep chunk sizes at a reasonable size, as sending too much content to an LLM at once tends to reduce answer quality. 
Upload the original file to Azure Blob storage. 
Compute ada-002 embeddings for the content field. 
Add each chunk to the Azure AI search index. 

For RAG on images,  we add two additional steps before indexing: uploading an image version of each document page to Blob Storage and computing multi-modal embeddings for each image. 

Generating citable images 

The images are not just a direct copy of the document page. Instead, they contain the original document filename written in the top left corner of the image, like so: 

This crucial step will enable the GPT vision model to later provide citations in its answers. From a technical perspective, we achieved this by first using the PyMuPDF Python package to convert documents to images, then using the Pillow Python package to add a top border to the image and write the filename there.

Question answering 

Now that our Blob storage container has citable images and our AI search index has multi-modal embeddings, users can start to ask questions about images. 

Our RAG app has two primary question asking flows, one for “single-turn” questions, and the other for “multi-turn” questions which incorporates as much conversation history that can fit in the context window. To simplify this explanation, we’ll focus on the single-turn flow.  

Our single-turn RAG on documents flow looks like: 

Receive a user question from the frontend. 
Compute an embedding for the user question using the OpenAI ada-002 model. 
Use the user question to fetch matching documents from the Azure AI search index, using a hybrid search that does a keyword search on the text and a vector search on the question embedding. 
Pass the resulting document chunks and the original user question to the gpt-3.5 model, with a system prompt that instructs it to adhere to the sources and provide citations with a certain format. 

Our single-turn RAG on documents-plus-images flows looks like this: 

Receive a user question from the frontend. 
Compute an embedding for the user question using the OpenAI ada-002 model AND an additional embedding using the AI Vision API multimodal model. 
Use the user question to fetch matching documents from the Azure AI search index, using a hybrid multivector search that also searches on the imageEmbedding field using the additional embedding. This way, the underlying vector search algorithm will find results that are both similar semantically to the text of the document but also similar semantically to any images in the document (e.g. “what trends are increasing?” could match a chart with a line going up and to the right). 
For each document chunk returned in the search results, convert the Blob image URL into a base64 data-encoded URI. Pass both the text content and the image URIs to a GPT vision model, with this prompt that describes how to find and format citations: The documents contain text, graphs, tables and images.

Each image source has the file name in the top left corner of the image with coordinates (10,10) pixels and is in the format SourceFileName:<file_name>

Each text source starts in a new line and has the file name followed by colon and the actual information. Always include the source name from the image or text for each fact you use in the response in the format: [filename]

Answer the following question using only the data provided in the sources below.

The text and image source can be the same file name, don’t use the image title when citing the image source, only use the file name as mentioned.  

Now, users can ask questions where the answers are entirely contained in the images and get correct answers! This can be a great fit for diagram-heavy domains, like finance. 

Considerations 

We have seen some really exciting uses of this multimodal RAG approach, but there is much to explore to improve the experience.

More file types: Our repository only implements image generation for PDFs, but developers are now ingesting many more formats, both image files like PNG and JPEG as well as non-image files like HTML, docx, etc. We’d love help from the community in bringing support for multimodal RAG to more file formats. 

More selective embeddings: Our ingestion flow uploads images for *every* PDF page, but many pages may be lacking in visual content, and that can negatively affect vector search results. For example, if your PDF contains completely blank pages, and the index stored the embeddings for those, we have found that vector searches often retrieve those blank pages. Perhaps in the multimodal space, “blankness” is considered similar to everything. We’ve considered approaches like using a vision model in the ingestion phase to decide whether an image is meaningful, or using that model to write a very descriptive caption for images instead of storing the image embeddings themselves.

Image extraction: Another approach would be to extract images from document pages, and store each image separately. That would be helpful for documents where the pages contain multiple distinct images with different purposes, since then the LLM would be able to focus more on only the most relevant image.

We would love your help in experimenting with RAG on images, sharing how it works for your domain, and suggesting what we can improve. Head over to our repo and follow the steps for deploying with the optional GPT vision feature enabled!

​Microsoft Tech Community – Latest Blogs –Read More 

Over the last few weeks we have been working together with Altair engineers to verify and validate their nanoFluidX v2024 product on Azure. This software offers significant advantages for engineers tackling problems where traditional CFD technology requires significant manual time and heavy computational resources. Vehicle wading is an important durability attribute where engineers monitor water reach and accumulation, and assess the potential for damage caused by water impact.

nanoFluidX’s Lagrangian meshless approach was designed from inception for GPU compute using NVIDIA CUDA, making it one of the fastest SPH solvers on the market. Models can be setup incredibly quickly, giving engineers the power to iterate faster.

With this validation, the intention was to look at the GPU compute possibilities in two ways: how will nanoFluidX perform on the Nvidia H100 series GPUs, and how it will work while scaling up to 8-way GPU virtual machines (VMs). Let’s look at the A100 and the H100 first.

The NC_A100_v4 has 3 flavors, with 1, 2 or 4 A100 80GB GPUs. In the basis, these are PCIe based GPUs, but internally they are NVlink connected in pairs. The rest of the system consists of 24 (non-multithreaded) AMD Milan CPU cores, 220GB main memory, and a 960TB NVME local scratch disk per GPU. When selecting a 2 or 4 GPU VMs, these numbers are multiplied up to a total of 880GB main memory.

The NC_H100_v5 has grown together with the GPU capabilities. It is available in a 1 or 2 GPU configuration built around the Nvidia 94GB H100 NVL. While this GPU has a PCIe interface towards the main system, many of the capabilities are in line with the SXM H100 series. The CPU cores are increased to 40 (non-multi-threaded) AMD Genoa CPU cores and 320 GB of main memory together with an upgraded 3.5TB NVME local scratch disk.

The benchmark that was run for this validation is the Altair CX-1 car model. This benchmark represents a production-scale model of a full size vehicle traveling at 10 km/h through a 24 meter wading channel in 15 seconds.

“Collaborating with Microsoft and NVIDIA, we have successfully validated nanoFluidX v2024 on NVIDIA’s A100 and H100 GPUs. The latest release boasts a solver that is 1.5x faster than previously, and offers improved scaling on multiple GPUs. These benchmarks show the use of NVIDIA H100 significantly enhances performance by up to 1.8x, cutting simulation times and accelerating design cycles. These advancements solidify nanoFluidX as one of the fastest Smoothed-particle Hydrodynamic (SPH) GPU code on the market.” – David Curry, Senior Vice President, CFD and EDEM, Altair.

As can been seen in the table below, the H100 delivers higher performance than the A100, which is in line with the published performance increase between the two generations by Nvidia. Therefore, both the software and the Azure VMs allow these GPUs to reach their compute potential.

Since nanoFluidX supports multi-GPU systems, we wanted to validate the scalability and test them on the 8-way GPU ND series. Again, we tested on both the Nvidia A100 systems, the NDads_A100_v4, and its successor based on the Nvidia H100: the NDisr_H100_v5. Both of these systems have all 8 GPUs interconnected through NVlink.

Chart showing performance increase for H100 (NCv5) over A100 (NCv4)

As shown in the table above, nanoFluidX effectively utilizes all GPU power. On the NDisr_H100_v5, it achieved a 1-hour simulation duration, significantly impacting turnaround and design cycle time.

While you can simply go to the portal, request a quota, and spin up these VMs, we often see customers seeking an HPC environment that integrates better into their workflow for production. Altair offers a solution to run projects on Azure through their Altair One platform. Please collaborate with your Altair representative to enable this Azure-based solution for you. Alternatively, you can use the Altair SaaS solution, Altair Unlimited, from the virtual appliance marketplace to deploy and manage your own HPC cluster on Azure. To enable GPU quotas for HPC, please coordinate with your Azure account manager.

#AzureHPCAI

​Microsoft Tech Community – Latest Blogs –Read More 

Introduction:

In this post, I explore a recent experiment aimed at creating a RAG pipeline tailored for the insurance industry, specifically for handling automobile insurance claims, with the goal of potentially reducing processing times.

I also showcase the implementation of Autogen AI Agents to enhance search retrieval through agent interaction and function calls on sample auto insurance claims documents, a Q&A use case, and how this workflow can substantially reduce the time required for claims processing.

RAG workflows in my opinion represent a novel data stack, distinct from traditional ETL processes. Although they encompass data ingestion and processing similar to traditional ETL in data engineering, they introduce additional pipeline stages like chunking, embedding, and the loading of data into vector databases, diverging from the standard Lakehouse or data warehouse pipelines.

Each stage of the RAG application workflow is pivotal to the accuracy and pertinence of the downstream LLM application. One of these stages is the chunking method, and for this proof of concept, I chose to test a page-based chunking technique that leverages the document’s layout without relying on third party packages.

Key Services and Features:

By leveraging enterprise-grade features of Azure AI services, I can securely integrate Azure AI Document Intelligence, Azure AI Search, and Azure OpenAI through private endpoints. This integration ensures that the solution adheres to best practice cybersecurity standards. In addition, it offers secure network isolation and private connectivity to and from virtual networks and associated Azure services.

Some of these services are:

Azure AI Document Intelligence and the prebuilt-layout model.

Azure AI Search Index and Vector database configured with the HNSW search algorithm.

Azure OpenAI GPT-4-o model.

Page-based Chunking technique.

Autogen AI Agents.

Azure Open AI Embedding model: text-ada-003.
Azure Key Vault.
Private Endpoints integration across all services.
Azure Blob Storage.
Azure Function App. (This serverless compute platform can be replaced with Microsoft Fabric or Azure Databricks)

Document Extraction and Chunking:

These templates include forms with data detailing the accident location, description, vehicle information of the involved parties, and any injuries sustained. Thanks to the folks at LlamaIndex for providing the sample claims documents. Below is a sample of the forms template.

claims sample form

The claim documents are PDF files housed in Azure Blob Storage. Data ingestion begins from the container URL of the blob storage using the Azure AI Document Intelligence Python SDK.

This implementation of a page-based chunking method utilizes the markdown output from the Azure AI Document Intelligence SDK. The SDK, setup with the prebuilt-layout extraction model, extracts the content of pages, including forms and text, into markdown formats, preserving the document’s specific structure, such as paragraphs and sections, and its context.

The SDK facilitates the extraction of documents page by page, via the pages collection of the documents, allowing for the sequential organization of markdown output data. Each page is preserved as an element within a list of pages, streamlining the process of efficiently extracting page numbers for each segment. More details about the document intelligence service and layout model can be found at this link.

The snippet below illustrates the process of page-based extraction, preprocessing of page elements, and their assignment to a Python list:

page extraction

Each page content will be used as the value of the content field in the vector database index, alongside other metadata fields in the vector index. Each page content is its own chunk and will be embedded before being loaded into the vector database. The following snippet demonstrates this operation:

Define Autogen AI Agents and Agent Tool/Function:

The concept of an AI Agent is modeled after human reasoning and the question-and-answer process. The agent is driven by a Large Language Model (its brain), which assists in determining whether additional information is required to answer a question or if a tool needs to be executed to complete a task.

In contrast, non-agentic RAG pipelines incorporate meticulously designed prompts that integrate context information (typically through a context variable within the prompt) sourced from the vector store before initiating a request to the LLM for a response. AI agents possess the autonomy to determine the “best” method for accomplishing a task or providing an answer. This experiment presents a straightforward agentic RAG workflow. In upcoming posts, I will delve into more complex, agent-driven RAG solutions. More details about Autogen Agents can be accessed here.

I set up two Autogen agent instances designed to simulate or engage in a question-and-answer chat conversation among themselves to carry out search tasks based on the input messages. To facilitate the agents’ ability to search and fetch query results from the Azure AI Search vector store via function calls, I authored a Python function that will be associated with these agents. The AssistantAgent, which is configured to invoke the function, and the UserProxyAgent, which is tasked with executing the function, are both examples of the Autogen Conversable Agent class.

The user agent begins a dialogue with the assistant agent by asking a question about the search documents. The assistant agent then gathers and synthesizes the response according to the system message prompt instructions and the context data retrieved from the vector store.

The snippets below provide the definition of Autogen agents and a chat conversation between the agents. The complete notebook implementation is available in the linked GitHub repository.

Last Thoughts:

The assistant agent correctly answered all six questions, aligning with my assessment of the documents’ information and ground truth. This proof of concept demonstrates the integration of pertinent services into a RAG workflow to develop an LLM application, which aims to substantially decrease the time frame for processing claims in the auto insurance industry scenario.

As previously stated, each phase of the RAG workflow is crucial to the response quality. The system message prompt for the Assistant agent needs precise crafting, as it can alter the response outcomes based on the set instructions. Similarly, the custom retrieval function’s logic plays a significant role in the agent’s ability to locate and synthesize responses to the messages.

The accuracy of the responses has been assessed manually. Ideally, this process should be automated.

In an upcoming post, I intend to explore the automated evaluation of the RAG workflow. Which methods can be utilized to accurately assess and subsequently refine the RAG pipeline?

Both the retrieval and generative stages of the RAG process require thorough evaluation.

What tools can we use to accurately evaluate the end-to-end phases of a RAG workflow, including extraction, processing, and chunking strategies? How can we compare various chunking methods, such as the page-based chunking described in this article versus the recursive character text split chunking option?

How do we compare the retrieval results of an HNSW vector search algorithm against the KNN exhaustive algorithm?

What kind of evaluation tools are available and what metrics can be captured for agent-based systems?

Is a one-size-fits-all tool available to manage these?  We will find answers to these questions.

Moreover, I would also like to examine and assess how this and other RAG and generative ai workflows are reviewed to ensure alignment with the standards of fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability as defined in the Responsible AI Ethics framework for building and developing these systems.

​Microsoft Tech Community – Latest Blogs –Read More 

Join us for the DEVintersection Conference from September 10 to 12, 2024, in Las Vegas, Nevada. This event gathers technology enthusiasts from across the globe. Whether you are a developer, IT professional, or business leader, this conference provides an exceptional chance to explore the latest in cloud technology.

Our experts from Azure Communication Services will be there at the event and will be hosting the following sessions.

Take Your Apps to the Next Level: Azure OpenAI, Communication, and Organizational Data Features

Sept 10, 14:00 – 15:00 | Lab | Grand Ballroom 118 | Dan Wahlin

Many of us are building Line of Business (LOB) apps that can integrate data from custom APIs and 3rd party data sources. That’s great, but when was the last time you sat down to think through how you can leverage the latest technologies to take the user experience to the next level?

In this session, Dan Wahlin introduces different ways to enhance customer experience by adding calling and SMS. You can integrate organizational data to minimize user context shifts, leverage the power of AI to enhance customer productivity, and take your LOB apps to the next level.

Register and add this session.

Bridging the Gap: Integrating Custom Applications with Microsoft Using Azure

Sept 11, 15:30 – 16:30 | Session | Grand Ballroom 122 | Milan Kaur

Discover how Azure can facilitate seamless integration between non-Teams users and Microsoft Teams. If you’re invested in Teams and seeking to develop audio-video solutions to connect your custom third-party applications with Teams, this session is for you. Join us to explore the possibilities and streamline collaboration beyond internal Teams.

Register and add this session.

Beyond chatbots: add multi-channel communication to your AI apps

Sept 12, 08:30 – 09:30 | Session | Grand Ballroom 122 | Milan Kaur

Unlock the potential of conversational AI with Azure!
In this session, discover how to extend your bot’s functionality beyond standard chat interactions. We’ll learn together how to add voice and other messaging channels such as WhatsApp to build pro-code AI bots grounded in custom data.

Register and add this session.

About our speakers

Dan Wahlin
Principal Cloud Developer Advocate

Milan Kaur

Senior Product Manager

Dan Wahlin is a Principal Cloud Developer Advocate at Microsoft focusing on Microsoft 365 and Azure integration scenarios. In addition to his work at Microsoft, Dan creates training courses for Pluralsight, speaks at conferences and meetups around the world, and offers webinars on a variety of technical topics. 

Twitter: @DanWahlin

Milan is a seasoned software engineer turned product manager passionate about building innovative communication tools. With over a decade of experience in the industry, she has a deep understanding of the challenges and opportunities in the field of cloud communications. 
LinkedIn: @milankaurintech 

​Microsoft Tech Community – Latest Blogs –Read More 

Microsoft and Oracle are excited to announce that we are expanding the general availability of Oracle Database@Azure for the Azure Australia East region.  

Customer demand for Oracle Database@Azure continues to grow – that’s why we’re announcing plans to expand regional availability to a total of 21 regions around the world. Oracle Database@Azure is now available in six Azure regions – Australia East, Canada Central, East US, France Central, Germany West Central, and UK South. To meet growing global demand, the service will soon be available in more regions, including Brazil South, Central India, Central US, East US 2, Italy North, Japan East, North Europe, South Central US, Southeast Asia, Spain Central, Sweden Central, United Arab Emirates North, West Europe, West US 2, and West US 3. In addition to the 21 primary regions, we will also add support for disaster recovery in a number of other Azure regions including Brazil Southeast, Canada East, France South, Germany North, Japan West, North Central US, South India, Sweden South, UAE Central, UK West, and West US.

As part of the continued expansion of Oracle services on Azure, we have new integrations with Microsoft Fabric and Microsoft Sentinel and support for Oracle Autonomous Recovery Service. Visit our sessions at Oracle CloudWorld and read our blog to learn more. 

Learn more: https://aka.ms/oracle   

Technical documentation: Overview – Oracle Database@Azure | Microsoft Learn  

Get skilled: https://aka.ms/ODAA_Learn   

​Microsoft Tech Community – Latest Blogs –Read More 

Introduction
In the ever-evolving landscape of artificial intelligence, the ability to efficiently load test large language models (LLMs) is crucial for ensuring optimal performance and scalability. llm-load-test-azure is a powerful tool designed to facilitate load testing of LLMs running in various Azure deployment settings.

 Why Use llm-load-test-azure? 

The ability to load test LLMs is essential for ensuring that they can handle real-world usage scenarios. By using llm-load-test-azure, developers can identify potential bottlenecks, optimize performance, and ensure that their models are ready for deployment. The tool’s flexibility, comprehensive feature set, and support for various Azure AI models make it an invaluable resource for anyone working with LLMs on Azure.

Some scenarios where this tool is helpful:

You set up an endpoint and need to determine the number of tokens it can process per minute and the latency expectations.
You implemented a Large Language Model (LLM) on your own infrastructure and aim to benchmark various compute types for your application.
You intend to test the real token throughput and conduct a stress test on your premium PTUs. 

Key Features

llm-load-test-azure is packed with features that make it an indispensable tool for anyone working with LLMs on Azure. Here are some of the highlights:

Customizable Testing Dataset: Generate a custom testing dataset tailored to settings similar to your use case. This flexibility ensures that the load tests are as relevant and accurate as possible.
Load Testing Options: The tool supports customizable concurrency, duration, and warmup options, allowing users to simulate various load scenarios and measure the performance of their models under different conditions.
Support for Multiple Azure AI Models: Whether you’re using Azure OpenAI, Azure OpenAI Embedding, Azure Model Catalog serverless (Maas), or managed-compute (MaaP), llm-load-test-azure has you covered. The tool’s modular design enables developers to integrate new endpoints with minimal effort.
Detailed Results: Obtain comprehensive statistics like throughput, time-to-first-token, time-between-tokens and end2end latency in JSON format, providing valuable insights into the performance of your models.

Getting Started 

Using llm-load-test-azure is straightforward. Here’s a quick guide to get you started:

Generate Dataset (Optional): Create a custom dataset using the generate_dataset.py script. Specify the input and output lengths, the number of samples, and the output file name.

[ python datasets/generate_dataset.py –tok_input_length 250 –tok_output_length 50 –N 100 –output_file datasets/random_text_dataset.jsonl ]

–tok_input_length: The length of the input. minimum 25.

–tok_output_length: The length of the output.

–N: The number of samples to generate.

–output_file: The name of the output file (default is random_text_dataset.jsonl).

Run the Tool: Execute the load_test.py script with the desired configuration options. Customize the tool’s behavior using a YAML configuration file, specifying parameters such as output format, storage type, and warmup options.

load_test.py [-h] [-c CONFIG] [-log {warn,warning,info,debug}]

optional arguments:

-h, –help show this help message and exit

-c CONFIG, –config CONFIG

config YAML file name

-log {warn,warning,info,debug}, –log_level {warn,warning,info,debug}

Provide logging level. Example –log_level debug, default=warning

Results

The tool will produce comprehensive statistics like throughput, time-to-first-token, time-between-tokens and end2end latency in JSON format, providing valuable insights into the performance of your LLM Azure endpoint.

Example of the json output:

“results”: [ # stats on a request level
…
],
“config”: { # the run settings
…
“load_options”: {
“type”: “constant”,
“concurrency”: 8,
“duration”: 20
…
},
“summary”: { # overall stats
“output_tokens_throughput”: 159.25729928295627,
“input_tokens_throughput”: 1592.5729928295625,
“full_duration”: 20.093270540237427,
“total_requests”: 16,
“complete_request_per_sec”: 0.79, # number of competed requests / full_duration
“total_failures”: 0,
“failure_rate”: 0.0

#time per ouput_token
“tpot”: {
“min”: 0.010512285232543946,
“max”: 0.018693844079971312,
“median”: 0.01216195583343506,
“mean”: 0.012808671338217597,
“percentile_80”: 0.012455177783966065,
“percentile_90”: 0.01592913103103638,
“percentile_95”: 0.017840550780296324,
“percentile_99”: 0.018523185420036312
},
#time to first token
“ttft”: {
“min”: 0.4043765068054199,
“max”: 0.5446293354034424,
“median”: 0.46433258056640625,
“mean”: 0.4660029411315918,
“percentile_80”: 0.51033935546875,
“percentile_90”: 0.5210948467254639,
“percentile_95”: 0.5295632600784301,
“percentile_99”: 0.54161612033844
},
#input token latency
“itl”: {
“min”: 0.008117493672586566,
“max”: 0.01664590356337964,
“median”: 0.009861880810416522,
“mean”: 0.010531313198552402,
“percentile_80”: 0.010261738599844314,
“percentile_90”: 0.013813444118403915,
“percentile_95”: 0.015781731761280615,
“percentile_99”: 0.016473069202959836
},
#time to ack
“tt_ack”: {
“min”: 0.404374361038208,
“max”: 0.544623851776123,
“median”: 0.464330792427063,
“mean”: 0.46600091457366943,
“percentile_80”: 0.5103373527526855,
“percentile_90”: 0.5210925340652466,
“percentile_95”: 0.5295597910881042,
“percentile_99”: 0.5416110396385193
},
“response_time”: {
“min”: 2.102457046508789,
“max”: 3.7387688159942627,
“median”: 2.3843793869018555,
“mean”: 2.5091602653265,
“percentile_80”: 2.4795608520507812,
“percentile_90”: 2.992232322692871,
“percentile_95”: 3.541854977607727,
“percentile_99”: 3.6993860483169554
},
“output_tokens”: {
“min”: 200,
“max”: 200,
“median”: 200.0,
“mean”: 200.0,
“percentile_80”: 200.0,
“percentile_90”: 200.0,
“percentile_95”: 200.0,
“percentile_99”: 200.0
},
“input_tokens”: {
“min”: 2000,
“max”: 2000,
“median”: 2000.0,
“mean”: 2000.0,
“percentile_80”: 2000.0,
“percentile_90”: 2000.0,
“percentile_95”: 2000.0,
“percentile_99”: 2000.0
},

}
}

Conclusion

llm-load-test-azure is a powerful and versatile tool that simplifies the process of load testing large language models on Azure. Whether you’re a developer or AI enthusiast, this repository provides the tools you need to ensure that your models perform optimally under various conditions. Check out the repository on GitHub and start optimizing your LLMs today! 

Bookmark this Github link: maljazaery/llm-load-test-azure (github.com)

Acknowledgments

Special thanks to Zack Soenen for code contributions, Vlad Feigin for feedback and reviews, and Andrew Thomas, Gunjan Shah and my manager Joel Borellis for ideation and discussions.
llm-load-test-azure tool is derived from the original load test tool [openshift-psap/llm-load-test (github.com)]. Thanks to the creators.

Disclaimer

This tool is unofficial and not a Microsoft product. It is still under development, so feedback and bug reports are welcome.

​Microsoft Tech Community – Latest Blogs –Read More