The Future of AI: LLM Distillation just got easier

Part 4 – Maximize your fine-tuned model performance with the new Azure AI Evaluation SDK

By Cedric Vidal, Principal AI Advocate, Microsoft

Part of the Future of AI 🚀 series initiated by Marco Casalaina with his Exploring Multi-Agent AI Systems blog post.

Generated using Azure OpenAI DALL-E 3

In earlier posts of this distillation series, we detailed the process of distilling a Llama 3.1 405B model into a more compact Llama 3.1 8B model. This journey included generating a synthetic dataset using RAFT, as well as fine-tuning and deploying our student model on Azure AI Serverless.

But how can we confirm that our distilled model performs optimally? The crucial final step is evaluating the model.

Effective model evaluation is key to ensuring that our AI systems function as expected and meet the desired standards. With the introduction of the Azure AI Evaluation Python SDK, we now have a powerful toolkit for assessing AI models through advanced metrics. In this blog post, we’ll look at evaluating a distilled student model, which was trained with data generated by RAFT, and compare it against a baseline model.

In our setup, Llama 3.1 405B functions as the teacher, Llama 3.1 8B serves as the student model and GPT-4 serves as the judge.

Why evaluate?

Evaluating distilled student models is crucial because it allows us to assess how effectively knowledge has been transferred from the teacher model to the student model. Distillation aims to compress a larger, more complex model into a smaller, more efficient one without significantly sacrificing performance. By thoroughly evaluating the distilled models, we ensure they not only mimic the teacher model’s outputs but also maintain high levels of accuracy, coherence, and relevance. This evaluation process helps identify areas where the student model may need further fine-tuning and ensures that the distilled models are ready for deployment in resource-constrained environments where computational efficiency is paramount.

Process Overview

Evaluating the performance of our models involves several key steps, which can be broadly categorized under Testing and Scoring.

Testing

Run the Baseline Model on the Evaluation Split: Our first step is to run the teacher model (Llama 3.1 405B) on the evaluation split to generate its predictions.
Run the Student Model on the Evaluation Split: Next, we run the student model on the same evaluation dataset to generate its predictions.

Scoring

Calculate Metrics for the Baseline Model: Using the predictions from the baseline model, we calculate various performance metrics.
Calculate Metrics for the Student Model: Similarly, we calculate the performance metrics for the student model’s predictions.
Compare Metrics: Finally, we compare the performance of both models, highlighting the results through visuals and diagrams.

Testing the baseline and student models

Installing the SDK

First, you need to install the Azure AI Evaluation SDK:

pip install openai azure-ai-evaluation azure-identity promptflow-azure

Note on SDK Availability: It’s important to highlight that the Azure AI Evaluation SDK is currently in beta. This means that while the SDK offers a comprehensive suite of tools and features for evaluating AI models, it may still undergo changes and improvements. Users should stay updated with any modifications or enhancements introduced by Azure, and consider providing feedback to help refine and optimize the SDK for wider use in its official release.

Baseline Model Testing

This will generate answers to the questions in the evaluation dataset using the baseline model:

env $(cat .env .env.state) python .gorilla/raft/eval.py
–question-file $dataset_path_hf_eval
–answer-file $dataset_path_hf_eval_answer_baseline
–model $BASELINE_OPENAI_DEPLOYMENT
–env-prefix BASELINE
–mode $BASELINE_MODEL_API

Note: JSONL file format needs to be further converted to a format suitable for testing, see eval notebook for details.

Student Model Testing

This will generate answers to the questions in the evaluation dataset using the student model:

env $(cat .env .env.state) python .gorilla/raft/eval.py
–question-file $dataset_path_hf_eval
–answer-file $dataset_path_hf_eval_answer
–model $STUDENT_DEPLOYMENT_NAME
–env-prefix STUDENT
–mode $STUDENT_MODEL_API

Note: JSONL file format needs to be further converted to a format suitable for testing, see eval notebook for details.

Let’s look at a sample

This sample is extracted from the evaluation split and shows the baseline and student answers:

question: What types of waves do strong direct offshore winds create?

gold_final_answer: plunging or large barrel waves

context <DOCUMENT>Lefts, Rights, and A-frames could be directed from this pump design providing forWave intensity Artificial reefs Artificial wavesSurfing a stationary, artificialwave in Southern California A surfer going for the tube Catching waves at a surfing conteston the North Shore of Oahu, Hawaiirippable surf and barrel rides. The Ocean Dome cost about $2 billion tobuild and was expensive to maintain.[31] The Ocean Dome was closed in2007.</DOCUMENT> <DOCUMENT>However, thewaves that are produced by reef breaks are some of the best in the world. Famous reef breaks arepresent in Padang Padang (Indonesia), Pipeline (Hawaii), Uluwatu (Bali), and Teahupo’o(Tahiti).[49][52]A ledge break is formed by steep rocks ledges that make intense waves because the waves travelthrough deeper water then abruptly reach shallower water at the ledge. Shark Island, Australia is alocation with a ledge break.</DOCUMENT>

baseline_answer: Strong direct offshore winds create plunging or large barrel waves. These waves are characterized by their increased height and intensity due to the shallow water depth when they break.

student_answer: plunging or large barrel waves

This sample was chosen randomly and in this case, the student model answer is identical to the gold answer. This is not always the case.

Evaluating the baseline and student model responses

Built-in Evaluators

The Azure AI Evaluation SDK offers an extensive suite of built-in metrics, designed to facilitate comprehensive evaluation of AI models. In the following sections, we’ll highlight selected evaluators and provide detailed examples of their application, showcasing how they can enhance your model assessments.

They are categorized into two main groups: (1) metrics that leverage GPT models for scoring, providing advanced qualitative assessments, and (2) metrics that utilize straightforward mathematical calculations for evaluation.

GPT based metrics

Category

Evaluator Class

Notes

Quality

GroundednessEvaluator

Groundedness measures the extent to which the generated content is based on factual correctness and aligns with the provided data or context.

RelevanceEvaluator

Relevance assesses how pertinent the generated text is to the given input or prompt. Higher relevance scores indicate that the generated responses are more appropriate and closely aligned with the query or topic.

CoherenceEvaluator

Coherence measures how logically consistent and semantically meaningful the generated text is. Higher coherence indicates better understanding and logical consistency.

FluencyEvaluator

Fluency evaluates how naturally the generated text reads. Fluent text should be grammatically correct and smooth in its flow.

SimilarityEvaluator

Measures the similarity between the predicted answer and the correct answer

Content Safety

ViolenceEvaluator

SexualEvaluator

SelfHarmEvaluator

HateUnfairnessEvaluator

Composite

QAEvaluator

Built on top of individual quality evaluators.

ChatEvaluator

Similar to QAEvaluator but designed for evaluating chat messages.

ContentSafetyEvaluator

Built on top of individual content safety evaluators.

Math based metrics

Evaluator Class

Notes

BleuScoreEvaluator

BLEU (Bilingual Evaluation Understudy) is a widely-used metric for evaluating the quality of text generated by an AI by comparing it to one or more reference texts. It particularly looks at the precision of n-grams in the generated text.

RougeScoreEvaluator

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) primarily measures recall, comparing n-grams between the generated text and reference texts. It is commonly used for evaluation in summarization tasks.

F1ScoreEvaluator

A balance between precision and recall, the F1 score provides a single metric that combines both, offering a more comprehensive view of performance in classification problems.

Running metrics individually

The Azure AI Evaluation SDK enables the utilization of individual metrics. This feature is particularly useful for experimentation, gaining deeper insights, and incorporating metrics into bespoke evaluation workflows.

Tech Tip: This blog post is crafted using the Quarto writing system, a versatile tool for publishing with code. The Azure AI Evaluation metrics are seamlessly executed and displayed inline within this post.

Let’s look first at the F1 Score math metric

For a response that is accurate but includes additional information not found in the ground truth:

from azure.ai.evaluation import F1ScoreEvaluator
f1_score_evaluator = F1ScoreEvaluator()
f1_score = f1_score_evaluator(
ground_truth=”The capital of Japan is Tokyo.”,
response=”Tokyo is Japan’s capital, known for its blend of traditional culture”
)
print(f”The F1 Score is {round(f1_score[‘f1_score’], 2)}”)

The F1 Score is 0.5

For a response that is accurate but uses the same words turned differently:

from azure.ai.evaluation import F1ScoreEvaluator
f1_score_evaluator = F1ScoreEvaluator()
f1_score = f1_score_evaluator(
ground_truth=”The capital of Japan is Tokyo.”,
response=”Tokyo is Japan’s capital”
)
print(f”The F1 Score is {round(f1_score[‘f1_score’], 2)}”)

The F1 Score is 0.67

Let’s look first at the Similarity GPT metric

We first need to instantiate the Judge model client:

from os import getenv
from azure.ai.evaluation import AzureOpenAIModelConfiguration

model_config = AzureOpenAIModelConfiguration(
azure_endpoint = getenv(“JUDGE_AZURE_OPENAI_ENDPOINT”),
azure_deployment = getenv(“JUDGE_AZURE_OPENAI_DEPLOYMENT”),
api_version = getenv(“JUDGE_OPENAI_API_VERSION”),
)

Let’s now instantiate the Similarity score metric:

from azure.ai.evaluation import SimilarityEvaluator
similarity_evaluator = SimilarityEvaluator(model_config)

For a response that is accurate but includes additional information not found in the ground truth:

similarity = similarity_evaluator(
query=”What’s the capital of Japan?”,
ground_truth=”The capital of Japan is Tokyo.”,
response=”Tokyo is Japan’s capital, known for its blend of traditional culture”
)
print(f”The Similarity is {similarity[‘gpt_similarity’]}”)

The Similarity is 4.0

For a response that is accurate but uses the same words turned differently:

similarity = similarity_evaluator(
query=”What’s the capital of Japan?”,
ground_truth=”The capital of Japan is Tokyo.”,
response=”Tokyo is Japan’s capital”
)
print(f”The Similarity is {similarity[‘gpt_similarity’]}”)

The Similarity is 5.0

GPT-based similarity metrics demonstrate greater robustness in evaluating correct responses that are phrased differently compared to traditional F1 Scores.

Running metrics in bulk

While evaluating metrics individually helps in understanding their functionality, acquiring statistically significant results necessitates running them on a larger scale across an evaluation dataset.

The Azure AI Evaluation SDK provides a convenient bulk evaluation capability via the evaluate function.

To begin, we need to initialize the evaluators that will be used to assess the student and baseline models:

from azure.ai.evaluation import CoherenceEvaluator, F1ScoreEvaluator, FluencyEvaluator, GroundednessEvaluator, RelevanceEvaluator, SimilarityEvaluator, BleuScoreEvaluator, RougeScoreEvaluator, RougeType

# Initializing evaluators
evaluators = {
# GPT based metrics
“coherence” : CoherenceEvaluator(model_config),
“f1_score” : F1ScoreEvaluator(),
“fluency” : FluencyEvaluator(model_config),
“groundedness” : GroundednessEvaluator(model_config),
“relevance” : RelevanceEvaluator(model_config),
“similarity” : SimilarityEvaluator(model_config),

# Math metrics
“bleu” : BleuScoreEvaluator(),
“rouge_1” : RougeScoreEvaluator(RougeType.ROUGE_1),
“rouge_2” : RougeScoreEvaluator(RougeType.ROUGE_2),
}

Note that we have previously executed the baseline and student models on the evaluation dataset, which means the JSONL file provided to the evaluate function already includes their responses. Consequently, further model invocations are unnecessary at this stage.

Recommendation: It’s often beneficial to run the baseline and student models once initially. By doing so, you can execute the evaluate function multiple times with various metrics configurations without re-incurring the inference time and costs associated with model executions. Note that while this avoids repeated inference expenses, using GPT-based metrics will still incur costs and time for each evaluate execution, as the Judge model is utilized.

from azure.ai.evaluation import evaluate

result = evaluate(
data=”test-results-[baseline|student].jsonl”,
evaluators=evaluators,
evaluator_config={
“default”: {
“column_mapping”: {
“query”: “${data.question}”,
“response”: “${data.final_answer}”,
“ground_truth”: “${data.gold_final_answer}”,
“context”: “${data.context}”,
}
}
},
)

This command initiates a background process that hosts a user interface locally. Here is an example of its appearance:

The interface updates in real-time to display the progress of the scoring process on the evaluation dataset.

Additionally, you can click on each completed line to view the detailed trace of the calls. This feature is particularly useful for GPT-based metrics, as it reveals the system prompt used and provides insights into the underlying logic that contributed to the final score:

Comparing Metrics and Visualizing Results

Note: You can find the implementation details for generating the comparison figures of baseline and student metrics in the repository notebook. This resource provides comprehensive insights into how the metric comparisons were conducted, along with the code necessary to reproduce these visualizations.

Going further with continuous model evaluation and GenAIOps

This marks the beginning of a continuous improvement journey. It’s quite common to find that the student model’s initial performance does not meet expectations. Through our evaluation, we may uncover areas needing adjustment—whether it’s refining the synthetically generated dataset, optimizing fine-tuning parameters, or other elements. This initiates a cycle of iterative improvement and reassessment before the model is ready for deployment in production.

To effectively help you navigate this process, we came up with the GenAIOps Maturity Model, which serves as a comprehensive guide for evaluating your progress and maturity in operationalizing AI models.

Conclusion

By leveraging the Azure AI Evaluation Python SDK, we gain a detailed understanding of how our distilled student model compares to the baseline model across a spectrum of performance indicators. This structured evaluation framework not only helps in refining our models but also ensures that we are continuously improving and delivering robust AI solutions.

Explore, fork and clone the comprehensive 🚀🔥 GitHub Recipe Repository for complete code coverage on executing the full distillation process, including in-depth evaluations as detailed in this blog post. Discover step-by-step notebooks and resources to master the entire pipeline efficiently.

Stay tuned for more insights and tutorials on advanced AI topics and the latest tools available in the Azure ecosystem!

​Microsoft Tech Community – Latest Blogs –Read More 

Hello All,

We currently have some DLP policies to restrict Financial Data, HIPPA, and PII data from leaving our org.

However, is there a way to restrict this type of sensitive data from being sent into the org?  For example, an external address sends some sensitive data to a specific mailbox.  Can a DLP policy be created to block that data from reaching a specific mailbox and reply back the email was blocked due to the content?

Thanks for any info!

​Hello All,We currently have some DLP policies to restrict Financial Data, HIPPA, and PII data from leaving our org. However, is there a way to restrict this type of sensitive data from being sent into the org?  For example, an external address sends some sensitive data to a specific mailbox.  Can a DLP policy be created to block that data from reaching a specific mailbox and reply back the email was blocked due to the content? Thanks for any info!  Read More

​

Hi all

I’m trying to make my Power Queries more flexible. 

Starting position: I have three queries that have two xlsx files (SAP exports) as a base (connections only). The two files show Equipment data from different dates, which are compared using the queries. They show new equipments, removed equipments, and changes.

The files are named by date, so 01/08/2024, 01/09/2024 etc. And are all in the same folder.

My goal: I want the end user to be able to pick any two SAP exports within Excel and the queries to update accordingly. They should not use Power Query Editor to change the query source.

I tried parameters but failed when connecting it to the dedicated cell in Excel.

What are the steps using parameters? Or are there alternative solutions?

Thanks from a beginner!

​Hi all I’m trying to make my Power Queries more flexible. Starting position: I have three queries that have two xlsx files (SAP exports) as a base (connections only). The two files show Equipment data from different dates, which are compared using the queries. They show new equipments, removed equipments, and changes.The files are named by date, so 01/08/2024, 01/09/2024 etc. And are all in the same folder. My goal: I want the end user to be able to pick any two SAP exports within Excel and the queries to update accordingly. They should not use Power Query Editor to change the query source. I tried parameters but failed when connecting it to the dedicated cell in Excel. What are the steps using parameters? Or are there alternative solutions? Thanks from a beginner!  Read More

​

Hello!

In October of 2022, Microsoft took the Defender for Endpoint Server sku off the price file and replaced with with Defender for Endpoint plan 1 and plan 2 which are purchased through Azure. Our company started migrating customers from the old sku to the new azure model and all was well. For the last six months or more, Defender for Server is now showing up on our NCE price file once more. After Microsoft told me so start selling the consumption sku, I am now unsure what my next move should be. Are you selling both products? Is there a use case for each? Anyone had any experience wit this issue?

Thanks in advance!

​Hello! In October of 2022, Microsoft took the Defender for Endpoint Server sku off the price file and replaced with with Defender for Endpoint plan 1 and plan 2 which are purchased through Azure. Our company started migrating customers from the old sku to the new azure model and all was well. For the last six months or more, Defender for Server is now showing up on our NCE price file once more. After Microsoft told me so start selling the consumption sku, I am now unsure what my next move should be. Are you selling both products? Is there a use case for each? Anyone had any experience wit this issue? Thanks in advance!  Read More

​

Hi All

I have a table on Worksheet A, unimaginatively named “Table1”, with fields including “CODE”.

On Worksheet B, in cell C1, I have this formula:

=FILTER(Table1,(Table1[CODE]=”T1″) + (Table1[CODE]=”P1″))

This works fine i.e. it shows a new table with all the rows where CODE = “T1” or “P1”.  Lovely.

Also on Worksheet B, in cell A1, I have the following string:

(Table1[CODE]=”T1″) + (Table1[CODE]=”P1″)

I have created this string using logic based on the data in Table1.  On another day, the string literals might change and there may be more OR-ed elements, perhaps:

(Table1[CODE]=”ENG002″) + (Table1[CODE]=”BBBB”)  + (Table1[CODE]=”Z YW”)

Essentially this string is volatile and I don’t want to hard-code it as in the first example.

How can I use successfully the string in cell A1 as the ‘include’ parameter to the FILTER function? 

I tried:

=FILTER(Table1,A1)

but this gives #VALUE!

I thought INDIRECT might work but:

=FILTER(Table1,INDIRECT(A1))

gives a #REF!

I think I am missing something obvious but can’t see it.  Can you help, at all?  Thanks VM.

Peter

​Hi All I have a table on Worksheet A, unimaginatively named “Table1”, with fields including “CODE”.On Worksheet B, in cell C1, I have this formula:=FILTER(Table1,(Table1[CODE]=”T1″) + (Table1[CODE]=”P1″))This works fine i.e. it shows a new table with all the rows where CODE = “T1” or “P1″.  Lovely. Also on Worksheet B, in cell A1, I have the following string:(Table1[CODE]=”T1″) + (Table1[CODE]=”P1″) I have created this string using logic based on the data in Table1.  On another day, the string literals might change and there may be more OR-ed elements, perhaps:(Table1[CODE]=”ENG002″) + (Table1[CODE]=”BBBB”)  + (Table1[CODE]=”Z YW”)Essentially this string is volatile and I don’t want to hard-code it as in the first example. How can I use successfully the string in cell A1 as the ‘include’ parameter to the FILTER function?  I tried:=FILTER(Table1,A1)but this gives #VALUE!I thought INDIRECT might work but:=FILTER(Table1,INDIRECT(A1))gives a #REF! I think I am missing something obvious but can’t see it.  Can you help, at all?  Thanks VM.Peter  Read More

​

Overview

In this blog, we will cover the Azure solution and deployment approach to migrate very large Oracle databases (200 TB +) to Azure.

VM Solution

Azure Virtual Machine (VM) offers optimal vCPUs for managing Oracle license with high RAM ratio to accommodate large Oracle SGA, IO and network bandwidth to support transaction and batch workload. We tested both M192 and M176 SKUs with a 200 TB+ Oracle database.

In the below comparison, M176 is based on Intel Sapphire Rapids processor with DDR5 offers higher SAPS and 1.5 faster memory access than the M192 (Intel Cascade Lake based processor). M176 is also equipped with Azure Boost technology for improving both IO/Network throughput. In our testing, we found M176 offers higher SAPS, faster memory access, more IO & Network bandwidth.

VM SKU

Intel Chipset

vCPU

Memory GiB

IOPS/MBps

Network Bandwidth (Mbps)

M192idms_v2

Cascade Lake / DDR4

192

4096

80000/2000

30000

M176ds_4_v3

Sapphire Rapids / DDR5

176

3892

130000/4000

40000

System Global Area (SGA): Very Large Oracle databases benefit greatly from large SGA size. Customers with such sizeable Oracle workloads should deploy an Azure M-series with a minimum of 4 TB or more RAM size.  Specific parameter recommendations below:

Set Linux Huge Pages to 75-90% of Physical RAM size
Set System Global Area (SGA) to 90% of Huge Page size
Set the Oracle parameter USE_LARGE_PAGES = ONLY

Storage Solution and Configuration

Azure has multiple storage options: Premium SSD, Premium SSDv2, Ultra and Azure NetApp Files (ANF).   The chart below captures an overview of the storage characteristics for virtual machine Standard_M176ds_4_v3.

IO Metrics

Premium SSD

Premium SSDv2 (Pv2)

Azure NetApp Files (ANF)

IOPS

130K

130K

Millions

Throughput

4 GB/s

4 GB/s

>5 GB/s

Latency

Lower single digit

(in ms)

< 1 ms

< .4 ms

High Availability

Oracle Data Guard

Oracle Data Guard

Oracle Data Guard

Disaster Recovery

Oracle Data Guard

Oracle Data Guard

Oracle Data Guard and/or ANF Cross Region Replication

Storage Snapshot

Yes

No

Yes

Storage Manager

Automatic Storage Management (ASM)

ASM

dNFS

For 200 TB+ Oracle database workload, we tested the following storage configuration which optimally leverages both the network and IO channel from ANF and Premium SSDv2 (Pv2) respectively.  Leveraging both ANF & Pv2 helped to optimize available VM throughputs effectively to meet and exceed the required IO requirements of such a large Oracle database.

Component

Disk Type

Number of Volumes

Size (TiB)

Total Throughput

GiB/s

Volume

Stripe Size

Oracle Home

Pv2

1

1

250

LVM

sapdata1-6

ANF

6

40 per volume

3000-4000

Individual

Oracle redo1-4

ANF

4

.5 per volume

500-2000

Individual

Oracle Fra

ANF

1

5

500-1000

Individual

Oracle Archive

Pv2

4

10

1500

LVM

64KB

Oracle Temp

Pv2 or Ephemeral

4

10

1500

LVM

64KB

Storage Deployment Approach

Both NFSv3 and NFSv4.1 are supported with Oracle Direct NFS (dNFS), we ultimately went with the combination of NFSv3 and Oracle Direct NFS. NFSv3 has been proven more reliable, more robust and is much less bug sensitive to dNFS than the newer NFS Version 4.1.

Application volume group for Oracle (AVG for Oracle) deploys all volumes required to install and operate the Oracle databases at enterprise scale, with optimal performance and according to best practices in a single step with optimized workflow. AVG for Oracle shortens Oracle database deployment time and ensures volume performance and stability, including the use of multiple storage endpoints (multiple IPs).

Oracle Database with Azure NetApp Files – Azure Example Scenarios | Microsoft Learn

Understand Azure NetApp Files application volume group for Oracle | Microsoft Learn

The Oracle data files can be distributed across sapdata volumes in round robins to avoid individual filesystem IO pressure.

High Availability Architecture

Azure offers a High Availability option by leveraging availability zones with SLA of 99.99. Most of the Azure regions provide VM SKU and low latency between the zones to deploy active-active HA setup across zones. However not every zone has got the required VM SKU so it is important to find out required VM SKU availability by running SAP-on-Azure-Scripts-and-Utilities/Get-VM-by-Zones at main · Azure/SAP-on-Azure-Scripts-and-Utilities (github.com) from your subscription. You can find out low latency zones by running SAP-on-Azure-Scripts-and-Utilities/AvZone-Latency-Test at main · Azure/SAP-on-Azure-Scripts-and-Utilities (github.com) . Combination of SKU availability and low latency script can guide you to identify zones that can offer active-active zone pair for HA deployment.

It is important to note that each subscription may be mapped to different physical zones.  You can find out physical zone mapping using Azure API Subscriptions – List Locations – REST API (Azure Resource Management) | Microsoft Learn.

Below picture provides HA architecture.

Data Protection Strategy

Customers can leverage a combination of ANF snapshot on the primary VM and weekly Oracle streaming backups on HA stand-by. We recommend the ANF snapshot tool provided by Microsoft known as the application consistent snapshot. Both snapshot and cloning can be executed in minutes, regardless of database size. Cloned volumes can be leveraged for system copy, but it is critical that production and QA VMs be on the same physical zone to ensure low latency between them. 

Technically, ANF does not prevent you from mounting NFS volumes across zones, so it is important that operational procedure established to keep both zone & ANF storage on same side.

Backup & Snapshot Approach

Domain

Backup Component

Backup Options

Frequency

Ran against

Load on DB VM

Primary Region

DB

snapshot (azacsnap)

4 hours

HA Primary VM

Low

RMAN Backup 

Daily incremental and weekly full

HA Stand-by VM

Low

Log

Archive Log Backup

15 minutes

HA Primary VM

Low

DR Region

DB

Oracle Data Guard

Current

n/a

Low

Database Restore

Failure

Recovery Option

Recovery Time

Comment

DB Level

Snapshot

Log (roll-forward)

In Minutes

1st Option

RMAN Restore

Log (roll-forward)

In Hours

2nd Option

Region Wide

Oracle Data Guard

In Minutes

1st Option

RMAN Restore

Log (roll-forward)

In Hours

2nd Option

Migration Approach

Depending on on-prem HW, OS/DB and SAP software levels, migration falls into either Homogeneous or Heterogeneous migration category.

We will cover a heterogeneous migration approach in a separate blog and discuss about how to reduce downtime and improved benefits for very large databases.

In the homogenous migration approach, smaller databases can be migrated using backup and restore. Larger database can be migrated by setting up Oracle Data Guard (ODG) replication.

Customer should run Azure Quality Check against deployed solution to identify and address any Azure best practices deviation.

Testing Approaches

Customers have leveraged Oracle Real Application Testing (RAT) option to perform real-world testing of the Oracle Database. By capturing production workloads during the peak period and replaying on Azure can help identify the required VM SKU and storage solution. Customer leveraged Azure Monitoring Dashboards and RAT generated outputs to analyze and conclude the test results and move forward confidently to migrate the Oracle on SAP system to Azure.

The RAT test covers Oracle database performance requirements. It is highly recommended to run SAP level volume and performance testing to ensure that end-to-end SAP processing meets and exceeds performance KPIs.

System Performance

Azure innovations such as Mv3 (Intel sapphire rapids /DDR5), Azure Boost for improving IO & Network Throughput, ANF storage solutions with sub-milli second latency with DNFS combined with Oracle advanced compression has resulted in 30-50% of SAP processing improvement on Azure.

Conclusion

Azure has led SAP on Azure solutions over the years and reached new heights every year by bringing over advanced VM SKU, Storage/Network solution, end to end architecture and deployment approaches to successfully deploying the largest Oracle database on SAP to Azure. Azure successfully hosts 200 TB+ SAP on Oracle database!

Useful Links

Below are key SAP Notes and Microsoft documentation for a successful Azure migration

2039619 – SAP Applications on Microsoft Azure using the Oracle Database: Supported Products and Versions – SAP for Me
1928533 – SAP Applications on Microsoft Azure: Supported Products and Azure VM types – SAP for Me
Oracle Azure Virtual Machines database deployment for SAP workload | Microsoft Learn
General performance considerations for Azure NetApp Files | Microsoft Learn
Understand Azure NetApp Files application volume group for Oracle | Microsoft Learn
SAP-on-Azure-Scripts-and-Utilities/QualityCheck/Readme.md at main · Azure/SAP-on-Azure-Scripts-and-Utilities · GitHub
1672954 – Oracle 11g, 12c, 18c and 19c: Usage of hugepages on Linux – SAP for Me

Co-Authors

Denny Koovakattu 

Ralf Klahr

Sathish Thirunethiram

​Microsoft Tech Community – Latest Blogs –Read More 

Hello,

I require some help with a sharepoint list that I have setup to track issues. I have a column that has a priority label: Low, Medium or High based on a number value. 

I have setup a conditional formatting see below in list view:

However, this conditional formatting does not show in the item view see below:

Is anyone able to tell me why and how do i correct this?

Many thanks

​Hello,I require some help with a sharepoint list that I have setup to track issues. I have a column that has a priority label: Low, Medium or High based on a number value.  I have setup a conditional formatting see below in list view: However, this conditional formatting does not show in the item view see below: Is anyone able to tell me why and how do i correct this? Many thanks  Read More

​

Simple bug that have been happening for years and is still not solved.

In my outlook I often switch to calendar and email. Sometime, after messing with calendar by toggling views in calendar, for some reason, my outlook email pan switch view from Compact to Preview? Why?

This has been occurring for at least 6 years, with other emails / computers, also to colleagues.

​Simple bug that have been happening for years and is still not solved. In my outlook I often switch to calendar and email. Sometime, after messing with calendar by toggling views in calendar, for some reason, my outlook email pan switch view from Compact to Preview? Why? This has been occurring for at least 6 years, with other emails / computers, also to colleagues.  Read More

​

Hi,

I have work email and my personal email on my Outlook (Windows computer). When in ToDo, I cannot switch email, I have to go in calendar and select only the calendar I want to pick the ToDo I want. On phone, I can switch email in ToDo that is used.

On Windows Outlook, I couldn’t find this option after a long time. Does it exist? I have to go through calendar every time.

​Hi, I have work email and my personal email on my Outlook (Windows computer). When in ToDo, I cannot switch email, I have to go in calendar and select only the calendar I want to pick the ToDo I want. On phone, I can switch email in ToDo that is used. On Windows Outlook, I couldn’t find this option after a long time. Does it exist? I have to go through calendar every time.  Read More

​

Has there ever been a Microsoft product that allows one to search by code syntax (in internal cloud) of certain specific extension files–like Gitlab or Bitbucket can do?

At my organization, the Sharepoint and even the Azure’s Microsoft graph is moving toward “””Content””” and AI-based search.  During this transition, I have found its often harder and harder to find certain files and emails in my organization based on symbols that document might contain.

These are examples of symbols that are really important to my business…

%m+% # perhaps the most important symbol to the accounting and finance
%w+% # the biweekly version of the same symbol
%>%
|>

Its come to the point, that I have a local OneDrive copy where thousands 1000 kb or less (1.2 Gb of storage space) file are saved just so I can use more simplistic search methods over these files.  But I feel like its such a waste…. if there was a better method.  

So I was curious if anyone at Microsoft has ever come up with a cloud based solution to the code search and exact search problem that has become worse as a result of reliance on OpenAI

​Has there ever been a Microsoft product that allows one to search by code syntax (in internal cloud) of certain specific extension files–like Gitlab or Bitbucket can do? At my organization, the Sharepoint and even the Azure’s Microsoft graph is moving toward “””Content””” and AI-based search.  During this transition, I have found its often harder and harder to find certain files and emails in my organization based on symbols that document might contain.These are examples of symbols that are really important to my business…%m+% # perhaps the most important symbol to the accounting and finance%w+% # the biweekly version of the same symbol%>%|> Its come to the point, that I have a local OneDrive copy where thousands 1000 kb or less (1.2 Gb of storage space) file are saved just so I can use more simplistic search methods over these files.  But I feel like its such a waste…. if there was a better method.   So I was curious if anyone at Microsoft has ever come up with a cloud based solution to the code search and exact search problem that has become worse as a result of reliance on OpenAI   Read More

​