I am trying too automatically fill a color in a cell when I type a certain word in it.

​I am trying too automatically fill a color in a cell when I type a certain word in it.  Read More

​

TiWorker.exe Windows Modules Installer Worker has a high occupancy rate, the laptop 13980HX is fully loaded, and the task manager shows disk activity and network activity as 0. In addition, the memory usage kept increasing (it was still increasing 15 minutes later). The system version was 24H2 26120.1542. Experience the Windows Feature Experience Pack 1000.26100.18.0 and the Windows update interface Cumulative Update for Windows 11 Insider Preview (10.0.26120.1542) (KB5041872) Download card at 8% (re-execute Windows update after finishing the program, display progress at 0% CPU usage is normal, when the card is at 8%, CPU full load)，The update history of Windows 11 Insider Preview (10.0.26120.1542) (KB5041872) shows a successful installation on 2024/8/20. But Windows Update is still pushing me updates for Windows 11 Insider Preview (10.0.26120.1542) (KB5041872), Shows up in history as a cumulative update of Windows 11 Insider Preview (10.0.26120.1542) (KB5041872) (2)/(3)

​TiWorker.exe Windows Modules Installer Worker has a high occupancy rate, the laptop 13980HX is fully loaded, and the task manager shows disk activity and network activity as 0. In addition, the memory usage kept increasing (it was still increasing 15 minutes later). The system version was 24H2 26120.1542. Experience the Windows Feature Experience Pack 1000.26100.18.0 and the Windows update interface Cumulative Update for Windows 11 Insider Preview (10.0.26120.1542) (KB5041872) Download card at 8% (re-execute Windows update after finishing the program, display progress at 0% CPU usage is normal, when the card is at 8%, CPU full load)，The update history of Windows 11 Insider Preview (10.0.26120.1542) (KB5041872) shows a successful installation on 2024/8/20. But Windows Update is still pushing me updates for Windows 11 Insider Preview (10.0.26120.1542) (KB5041872), Shows up in history as a cumulative update of Windows 11 Insider Preview (10.0.26120.1542) (KB5041872) (2)/(3)  Read More

​

Microsoft will soon be releasing a new security mitigation for the Common Log File System (CLFS) to the Windows Insiders Canary channel. In the past five years, 24 CVEs impacting CLFS have been identified and mitigated, making it one of the largest targets for vulnerability research in Windows. Rather than continuing to address single issues as they are discovered, the Microsoft Offensive Research & Security Engineering (MORSE) team has worked to add a new verification step to parsing CLFS logfiles, which aims to address a class of vulnerabilities all at once. This work will help protect our customers across the Windows ecosystem before they are impacted by potential security issues.

CLFS Overview

CLFS is a general-purpose logging service that can be used by software clients running in user-mode or kernel-mode. This service provides the transaction functionality for the Kernel Transaction Manager of the Windows kernel, which Transactional Registry (TxR) and Transactional NTFS (TxF) are built upon. While used in multiple places in the Windows kernel, a public user-mode API is also offered and can be utilized for any application wanting to store log records on the file system.

CLFS stores all log information and log records in a set of files, referred to as a “logfile”, which persists at a user-defined location on the file system. While the logfile is comprised of multiple files, the CLFS driver manages them as a single unit by creating a file handle for the whole set. The logfile is made up of one “Base Log File” (BLF), which holds the necessary metadata for the log, and two or more “container files”, which is where user-supplied log records are stored.

The custom file format used for the logfile is mostly undocumented, however, some high level information about the internal structures can be found at CLFS Stable Storage. Like many binary file formats, the internal data structures are read into memory, mapped to C/C++ structures, and later operated against by application code. For both the CLFS user-mode and kernel-mode API, it is the responsibility of the driver to read, parse, and ensure the validity of the data structures that make up this custom file format.

Attack Surface

It has proven to be a difficult task to validate all data read from the logfile due to the complexity of the data structures and how they are used. Out of the 24 CVE’s reported in the past 5 years, 19 have involved exploiting a logic bug in the CLFS driver caused by improper validation of one of its data structures. Included in these 19 CVEs are vulnerabilities with known exploits, such as CVE-2022-37969, CVE-2023-23376, and CVE-2023-28252. To trigger such a bug, an attacker can utilize the file system API (e.g. CreateFileW and WriteFile) to either craft a new malicious logfile or corrupt an existing logfile.

Mitigation Overview

Instead of trying to validate individual values in logfile data structures, this security mitigation provides CLFS the ability to detect when logfiles have been modified by anything other than the CLFS driver itself. This has been accomplished by adding Hash-based Message Authentication Codes (HMAC) to the end of the logfile. An HMAC is a special kind of hash that is produced by hashing input data (in this case, logfile data) with a secret cryptographic key. Because the secret key is part of the hashing algorithm, calculating the HMAC for the same file data with different cryptographic keys will result in different hashes.

Just as you would validate the integrity of a file you downloaded from the internet by checking its hash or checksum, CLFS can validate the integrity of its logfiles by calculating its HMAC and comparing it to the HMAC stored inside the logfile. As long as the cryptographic key is unknown to the attacker, they will not have the information needed to produce a valid HMAC that CLFS will accept. Currently, only CLFS (SYSTEM) and Administrators have access to this cryptographic key.

Anytime CLFS wants to make a modification to a logfile, such as adding a new log record to a container file or updating its metadata in the BLF, a new HMAC will need to be calculated using the contents of the entire file. Modifications to logfiles occur frequently, so it would be infeasible for CLFS to be repeatedly reading the file for HMAC calculation anytime a modification occurs, especially since CLFS container files can be upwards to 4GB in size. To reduce the overhead required for maintaining a HMAC, CLFS utilizes a Merkle tree (also known as a hash tree), which drastically lowers the amount of file reading needed to be done whenever a new HMAC needs to be calculated. While the Merkle tree makes HMAC maintenance feasible, it requires additional data to be stored on the file system. Refer to the “User Impact” section of this article for estimates on the storage overhead introduced by this mitigation.

Mitigation Adoption Period / Learning mode

A system receiving an update with this version of CLFS will likely have existing logfiles on the system that do not have authentication codes.  To ensure these logfiles get transitioned over to the new format, the system will place the CLFS driver in a “learning mode”, which will instruct CLFS to automatically add HMACs to logfiles that do not have them. The automatic addition of authentication codes will occur at logfile open and only if the calling thread has write access to the underlying files. Currently, the adoption period lasts for 90 days, starting from the time in which the system first booted with this version of CLFS. After this 90-day adoption period has lapsed, the driver will automatically transition into enforcement mode on its next boot, after which CLFS will expect all logfiles to contain valid HMAC. Note that this 90-day value may change in the future.

For new installs of Windows, CLFS will start in enforcement mode, as we do not expect there to be any existing logfiles that need to be transitioned over to the new format.

FSUTIL Command

The fsutil clfs authenticate command line utility can be used by Administrators to add or correct authentication codes for an existing logfile. This command will be useful for the following scenarios:

If a logfile is not opened during the mitigation adoption period, and therefore was not automatically transitioned over to the new format, this command can be used to add authentication codes to the logfile.
Since the authentication codes are created using a system-unique cryptographic key using the local system’s cryptographic key, allowing you to open the logfile that was created on another system.

Usage:

PS D:> fsutil.exe clfs authenticate
Usage: fsutil clfs authenticate <Logfile BLF path>
Eg: fsutil clfs authenticate “C:example_log.blf”

Add authentication support to a CLFS logfile that has invalid or
missing authentication codes. Authentication codes will be written
to the Base Logfile (.blf) and all containers associated with the logfile.

It is required that this command be executed with administrative
privileges.

Configuration

Settings for this mitigation can be configured in a couple of ways. No matter what approach you take, you’ll need to be an Administrator.

1. Registry settings

Settings for this mitigation are stored in the registry under the key HKLMSYSTEMCurrentControlSetServicesCLFSAuthentication. There are two registry values that can be viewed and modified by administrators:

Mode: The operating mode of the mitigation

0: The mitigation is enforced. CLFS will fail to open logfiles that have missing or invalid authentication codes. After 90 days of running the system with this version of the driver, CLFS will automatically transition into enforcement mode.
1: The mitigation is in learning mode. CLFS will always open logfiles. If a logfile is missing authentication codes, then CLFS will generate and write the codes to the file (assuming caller has write access).
2: The mitigation was disabled by an Administrator.

EnforcementTransitionPeriod: The amount of time, in seconds, that the system will spend in the adoption period. If this value is zero, then the system will not automatically transition into enforcement.

To disable the mitigation, an Administrator can run the following powershell command:

Set-ItemProperty -Path “HKLM:SYSTEMCurrentControlSetServicesCLFSAuthentication” -Name Mode -Value 2

To prolong the mitigation’s adoption period, an Administrator can run the following powershell command:

Set-ItemProperty -Path “HKLM:SYSTEMCurrentControlSetServicesCLFSAuthentication” -Name EnforcementTransitionPeriod -Value 2592000

2. Group Policy

The mitigation can be controlled using the ClfsAuthenticationChecking Group Policy setting (“Enable / disable CLFS logfile authentication”). This policy setting can be found under “Administrative TemplatesSystemFilesystem” in gpedit.exe.

Figure 1: Local group policy editor

Like all group policy settings, the CLFS logfile authentication setting can be in one of three states:

“Not Configured” (Default) – The mitigation is allowed to be enabled. CLFS will check its local registry mode (HKLMSYSTEMCurrentControlSetServicesCLFSAuthentication  [Mode]).
“Enabled” – The same as “Not Configured”. The mitigation is allowed to be enabled but CLFS will first check local registry settings.
“Disabled” – The mitigation is disabled. CLFS will not check for authentication codes and will attempt to open logfiles that may be corrupted.

Note that if the mitigation goes from a disabled to enabled state (via Group Policy), then the mitigation adoption period will automatically be repeated since there will likely be logfiles on the system that were created without authentication codes during the time the mitigation was disabled.

User Impact

This mitigation may impact consumers of the CLFS API in the following ways:

Because the cryptographic key used to make the authentication codes is system-unique, logfiles are no longer portable between systems. To open a logfile that was created on another system, an Administrator must first use the fsutil clfs authenticate utility to authenticate the logfile using the local system’s cryptographic key.
A new file, with the extension “.cnpf”, will be stored alongside the BLF and data containers. If the BLF for a logfile is located at “C:UsersUserexample.blf”, its “patch file” should be located at “C:UsersUserexample.blf.cnpf”. If a logfile is not cleanly closed, the patch file will hold data needed for CLFS to recover the logfile. The patch file will be created with the same security attributes as the file it provides recovery information for. This file will be around the same size as “FlushThreshold” (HKLMSYSTEMCurrentControlSetServicesCLFSParameters [FlushThreshold]).
Additional file space is required to store authentication codes. The amount of space needed for authentication codes depends on the size of the file. Refer to the list below for an estimate on how much additional data will be required for your logfiles:

512KB container files require an additional ~8192 bytes.
1024KB container files require an additional ~12288 bytes.
10MB container files require an additional ~90112 bytes.
100MB container files require an additional ~57344 bytes.
4GB container files require an additional ~2101248 bytes.

Due to the increase in I/O operations for maintaining authentication codes, the time it takes to create, open, and write records to logfiles has increased.  The increase in time for logfile creation and logfile open depends entirely on the size of the containers, with larger logfiles having a much more noticeable impact. On average, the amount of time it takes to write to a record to a logfile has doubled.

Changes to CLFS API

To avoid breaking changes to the CLFS API, existing error codes are used to report integrity check failures to the caller:

If CreateLogFile fails, then GetLastError will return the ERROR_LOG_METADATA_CORRUPT error code when CLFS fails to verify the integrity of the logfile.
For ClfsCreateLogFile, the STATUS_LOG_METADATA_CORRUPT status is returned when CLFS fails to verify the integrity of the logfile.

​Microsoft Tech Community – Latest Blogs –Read More 

We are excited to announce that we have a new and improved solution for syncing your form responses to Excel, with better reliability and performance! Previously, some Forms supported an older version of live data synchronization with Excel. This older solution will be gradually discontinued and completely replaced by the new solution by October 20th, 2024.

The older version of data sync exists in two types of forms:

Forms created from OneDrive for Business and Excel for the web

Form created from Excel for the web

Group forms created with SharePoint Online, Teams and M365 Group.

Group forms created with SharePoint Online

After the older solution for data syncing is replaced, the existing Excel workbooks and their data will still be retained, but they will not be updated with new responses. To ensure that your workbooks continue receiving new form responses, you must follow the steps below to update your workbook.

How can I upgrade to the new version of data sync via the Forms website?

Open a form using the older syncing solution, and you’ll see a pop-up reminding you to update your workbook. Simply click ‘Update sync in Excel’ to open your workbook and initiate the syncing process. 

Upgrading to the new Forms data sync

How do I upgrade my workbook in web to use the new version of data sync?

Open an Excel workbook that uses the older syncing solution, and you will see a pane on the right-side reminding you to update the workbook to continue syncing new responses. Click the “Update sync” button to begin the updating process. Please note that this process is not reversible.

Upgrading to the new Forms data sync via the web

A new sheet will be created in the same workbook that is connected to the form using the new syncing solution.

You should see all your previous responses being resynced to this new sheet, and any new responses will also be synced to this sheet. You will still be able to see the previous data in your workbook, but this will no longer be updated.

Once the update is complete, a green success bar will appear near the top of the page. The right-side pane will also change to confirm that the update has been successfully completed.

Updated to the new solution

FAQ

Q: What will happen if I don’t update to the new solution?
A: The old data sync service will stop on Oct 20th, 2024. After this point, any new form responses will still be saved in your form, but they will no longer sync to your workbook. As soon as you update your workbook to the new data sync solution, these new responses (and the older ones) will sync to a new sheet in your workbook.

Q: What’s the difference between the old version and new version?
A:

The new syncing solution is more reliable and has improved performance.
Currently, the new solution will only sync new responses to Excel for the web. But we are actively working on adding the new syncing solution to the Windows and Mac desktop apps.
With the new syncing solution, you must open the Excel app to receive new responses. Otherwise, they will not sync to the workbook.

Q: Can I receive new responses without opening my workbook? What if I have a Power Automate Flow based on the data sync?

A: With the new syncing solution, new form responses will only sync to the workbook when it is opened in Excel for the web (with desktop support coming soon). If you have a Power Automate Flow connected to the workbook, it will no longer receive new responses until the workbook is manually opened. In this case, we recommend updating your workflow to use the Forms Connector.  Create an automated workflow for Microsoft Forms – Microsoft Support

Q: How do I know if the form is using the older version or the new one?

A: In the new solution, we also updated the UI to open responses in Excel in Forms “responses” tab.

Open responses to Excel button in the old experience

Open responses to Excel button in the new solution – Excel file is not created

Open Excel button in the new solution – Excel file is created

Q: I’m not the form owner, should I do anything?
A: Contact the form owner if you know who that is and ask them to update the workbook. If you don’t know who the form owner is, contact your IT admin.

Q: If the upgrade failed, what should I do?
A: If the update fails, then restart the Excel app. You should see the option to try updating the workbook again. We only disconnect the older syncing solution once the new syncing solution has been successfully connected.

Q: Could I revert back after the update?
A: No. Once your workbook has been updated to the new syncing solution, it cannot be reverted back to the old syncing solution.

Q: How could I find the forms need to be updated?
A: You could check the forms or Excel file from Excel for the web which are still actively receiving responses. If there are notification shows, please follow the guidance to update.

Q: I received an email from Microsoft Forms, what should I do?
A: The email will contain links to each of the workbooks that use the older syncing solution with forms that you own. You need to go to each workbook and update to the new syncing solution individually.

Q: I have a form which has more than 50K responses, could I create new data sync for it?
A: The new syncing solution should sync all responses your form has ever received in the newly created sheet in your workbook. But if you have over 50,000 responses, then the new syncing solution will only sync the most recent 50,000. In this case, you can copy the missing responses from your original data and paste them into the table in the new sheet in the appropriate place. Please make sure to insert the appropriate number of blank rows before pasting the missing data. The table will continue adding new form responses as they are received.

​Microsoft Tech Community – Latest Blogs –Read More 

FabSoft increases document efficiency by 40% with Microsoft Azure AI

A small, nimble company at heart, FabSoft has been a Microsoft partner for 25 years and has integrated various Microsoft technologies into its document handling solutions. With over 150,000 successful installations, FabSoft solutions have been implemented into daily business operations worldwide. FabSoft uses Microsoft Azure AI Studio and Azure OpenAI for advanced analytics and AI capabilities in its DeskConnect product, which integrates with Outlook and multiple other Microsoft technologies to enhance document management operations and functionality.

Relying on optical character recognition alone slows document management

Many businesses, including those in banking, manufacturing, and education, struggle with traditional document management workflows. Handwritten notes and unstructured papers require manual review and categorization. The scale of documents and technical compatibility across platforms can be overwhelming. DeskConnect streamlines these processes.

DeskConnect provides a unified platform for company-wide information access. The solution addresses inefficiencies in document collaboration, reduces communication gaps, and increases productivity across different teams and departments within organizations of all sizes. By using Azure AI and Azure OpenAI combined with traditional optical character recognition (OCR), DeskConnect minimizes errors and automates the conversion of documents from any format into searchable data.

DeskConnect is customizable and scalable, enabling businesses to adjust data capture, document distribution, and system integrations to fit specific organizational needs. As your document volume grows and new systems are added to your environment, DeskConnect scales to meet the challenge.

Continue reading here

**Explore all case studies or submit your own**

​Microsoft Tech Community – Latest Blogs –Read More 

In today’s digital landscape, where security breaches are an unfortunate reality, safeguarding sensitive data and infrastructure has become more critical than ever. Recent cyberattacks have highlighted the importance of minimizing the blast radius and fortifying defenses against potential threats. Among the myriad security measures available, Azure Bastion provides a robust solution, offering a secure and seamless pathway for accessing virtual machines (VMs)—from Dev/Test environments to enterprise production workloads. 

Enhanced Security Measures 

Public ports exposed to the internet are often prime targets for malicious actors. Hackers leverage port-scanning on open RDP (Remote Desktop Protocol) and SSH (Secure Shell Protocol) entry points to gain unauthorized access to systems, potentially wreaking havoc on organizations’ operations and compromising sensitive data. Azure Bastion acts as a shield against such threats by hardening at one centrally managed gateway, closing RDP/SSH ports from the public internet while providing private connectivity to VMs. 

With Azure Bastion, the need to expose VMs to the public internet, along with the associated risks, is eliminated. Instead of relying on traditional methods like VPNs or on-premises jump servers, Bastion offers a simplified yet highly secure approach. By leveraging Transport Layer Security (TLS) encryption and integrating with Azure’s robust authentication mechanisms, Bastion provides seamless RDP/SSH connectivity to your VMs while hardening the attack surface to one point. 

The importance of securing RDP/SSH access cannot be overstated. These protocols are essential for remote management and troubleshooting, but they also represent significant vulnerabilities if not properly secured. Azure Bastion ensures that these critical access points are protected, reducing the risk of unauthorized access and potential breaches. By centralizing the management of RDP/SSH access, Bastion simplifies the security landscape, making it easier for organizations to enforce consistent security policies and monitor access activities. 

The above diagram shows connections to virtual machines via a Bastion dedicated deployment that uses a Basic or Standard SKU.  

The above diagram shows connections to virtual machines via a Bastion Premium SKU deployment using the private-only feature. 

Streamlined Management 

Azure Bastion not only enhances security but also simplifies management tasks. Since it is a fully managed service provided by Azure, users are relieved of the burden of setting up and maintaining infrastructure components. With just a few clicks, administrators can deploy Bastion and start securely accessing their VMs without worrying about infrastructure overhead, manual updates and patching, or configuration complexities. 

Moreover, Azure Bastion offers scalability and flexibility, allowing users to connect to multiple VMs across their Azure environment effortlessly. This centralized management approach streamlines operations and enhances productivity, especially in large-scale deployments where managing connectivity to numerous VMs can be challenging. 

The ease of deployment and management is a significant advantage for organizations of all sizes. Small and medium-sized businesses (SMBs) can benefit from the reduced complexity and lower operational overhead, while large enterprises can leverage Bastion’s scalability to manage extensive VM environments efficiently. By providing a consistent and reliable method for accessing VMs, Azure Bastion helps organizations maintain high levels of productivity and operational efficiency. 

Choosing the Right SKU 

Azure Bastion offers four distinct SKUs tailored to various needs and use cases, each with its unique advantages. 

Developer SKU: Generally available in 6 public regions, the new Developer SKU provides a cost-effective option for developers and testers seeking access to VMs. It offers one connection per VNET without the configuration, scaling, and features of more advanced VM solutions. Bastion Developer is an excellent choice for users looking to provide secure access to their development and testing environments without incurring expenses. Bastion Developer is estimated to be available in all other Bastion supported public regions within the year.  

Basic SKU: For small to medium-sized businesses (SMBs) and enterprises, the Basic SKU of Azure Bastion offers a well-rounded solution. This SKU offers a dedicated deployment within your Virtual Network (VNET), providing secure access for organizations seeking a comprehensive yet cost-effective option. With support for NSGs, peered VNets, and connectivity to 40-45 VMs at a time, Bastion Basic SKU is perfect for customers looking to secure their environment on a smaller scale. 

Standard SKU: For enterprises with production workloads demanding high availability, scalability, and advanced features, the Standard SKU of Azure Bastion is the top offering. With Bastion Standard SKU, customers can scale up to 50 instances and support up to 400 VM connections. Customers can also enable advanced features such as CLI support, IP-based connection to non-Azure VMs, and shared connections. 

Premium SKU: The Premium SKU is designed for customers with highly regulated workloads, such as financial services, government, and healthcare customers. Bastion Premium expands on the scalability of Standard SKU and adds advanced features such as graphical session recording of VM sessions and private-only connection on the Bastion host. Premium SKU provides enhanced audit and risk management for organizations that require the highest level of security and performance for their critical workloads. 

Conclusion 

Azure Bastion stands out as a vital tool in the arsenal of cloud security measures. By providing a secure, centrally managed gateway for RDP/SSH access to VMs, it significantly reduces the attack surface and enhances the overall security posture of organizations. Its ease of deployment, scalability, and integration with Azure’s authentication mechanisms make it an indispensable solution for modern enterprises looking to protect their digital assets. 

The ability to choose from different SKUs allows organizations to tailor their Bastion deployment to their specific needs, ensuring that they can achieve the right balance of security, cost, and functionality. Whether for development and testing, small-scale production environments, or large-scale enterprise deployments, Azure Bastion offers a flexible and robust solution that can adapt to a wide range of use cases. 

As cyber threats continue to evolve, the importance of robust security measures like Azure Bastion will only grow. By investing in secure and scalable solutions, organizations can better protect their sensitive data and infrastructure, ensuring that they remain resilient in the face of ever-changing security challenges. Azure Bastion represents a critical component of a comprehensive security strategy, providing the tools and capabilities needed to safeguard the modern digital landscape. 

​Microsoft Tech Community – Latest Blogs –Read More 

Designing and Operating a Generative AI Platform

Summary

Are you in a platform team who has been tasked with building an AI Platform to serve the Generative AI needs of your internal consumers? What does that mean? It’s a daunting challenge to be set, and even harder if you’re operating in a highly regulated environment.

As enterprises scale out usage of Generative AI past a few initial use-cases they will face into a new set of challenges – scaling, onboarding, security and compliance to name a few.

This article discusses such challenges and approaches to building an AI Platform to serve your internal consumers. 

Needs and more needs

To successfully run Generative AI at scale, organisations are utilising new features in API Management platforms such as Azure API Management’s AI Gateway (https://techcommunity.microsoft.com/t5/azure-integration-services-blog/introducing-genai-gateway-capabilities-in-azure-api-management/ba-p/4146525). The key to success for these platforms will be based on effective CI / CD and automation strategies. As we will see, an architecture to run Azure Open AI safely at scale involves safely deploying and managing lots of moving pieces, which together solve for scenarios such as:

How many Azure Open AI (AOAI) APIs should I create?
How do I version AOAI APIs?
How do I support consumers with different content-safety and model requirements?
How do I restrict throughput per Consumer, per deployment?
How do I scale out AOAI services?
How do I log all prompts and responses including streaming, without disruption?
What other value add services should a platform offer consumers?

Further, we need to understand how common services and libraries involved in building Generative AI Services fit into the architecture. We can build the best AI Platform in the world but if our consumers find they cannot use common Generative AI Libraries with it, have we really succeeded?

This document iterates through use-cases to build out a reference implementation that can safely run Azure API Management (AI Gateway) and Azure Open AI at scale, supporting most common libraries and services. You can find a reference implementation here:

https://github.com/graemefoster/APImAIPlatform

Target Azure Architecture

Decision Matrix

You might not need all the components. This matrix should help you understand what each iteration brings, allowing you to make an informed decision on when to stop.

AOAI

APIm / AOAI

Proxy /  APIm / AOAI

Proxy / APIm / AOAI / Defender

Chargeback

✗

✓

✓

✓

Prompt flow (JWT & key)

✗

✓

✓

✓

Advanced Logging (PII redaction / streaming)

✗

✗

✓

✓

SIEM

✗

✗

✗

✓

Out of scope

We are focusing on the requirements for running Azure Open AI and Generative AI services, rather than specific application stacks. A Generative AI Orchestrator may involve multiple services such an Azure App Service, Azure AI Search and Storage. These will vary between applications and are considered out-of-scope for this document.

Roles

We have identified the following roles involved in running Generative AI Applications. NB: these may not map one to one to people. We leave team structure to your own personal choice.

Role

Responsibilities

Gen AI Developers

Building and testing AI Orchestrators including promptflow

GEN AI Operators

Understanding consumer usage, prompt performance, overall system response.

AI Platform

Managing Azure Open AI services, access, security, and audit logging

AI Governance

Monitoring AI safety / abuse / groundedness

The remainder of the document will introduce fictitious use-cases and map them to the above roles. We will iterate on the platform and show how it provides for the requirements. It can be overwhelming to see a target architecture without insight into why individual services exist. My hope is that by the end of the document you see why each service is there. Also, this is an evolving space. As services gain new features, knowing why something is there will enable you to consolidate accordingly.

Gen AI Application Engineers

Let’s start with a simple use-case. Ultimately, we want our feature teams to deliver applications that deliver real business value. There is no application or business value without the Application Engineers.

Use Case:

So that I can build Generative AI Applications
As a Gen AI Application Engineer
I need access to Azure Open AI API models

Use Case:

So that I can iterate on a Generative AI Application
As a Gen AI Application Engineer
I want to see diagnostics information like latency and prompt response time of my prompts

These two use-cases can be delivered with a single Azure Open AI resource:

It’s simple, but it might be all you need. We run the Orchestrator in a PaaS like Azure App Service, send telemetry to Application Insights, and use a single Azure Open AI service.

We can use Managed Identities for secure authentication, and content-filters inside Azure Open AI to keep us safe. They can do things like detect Jailbreaks and moderate responses.
We can provision PTUs (Provisioned Throughput Units https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/provisioned-throughput-onboarding) to guarantee capacity for Generative AI.

If you can stop now, then great. But most organisations will soon hit the next set of challenges.

I need to fairly share my PTU amongst multiple consumers

My consumers need access to embedding models which are not part of my PTU, and one deployment isn’t enough

I want more control over the identities accessing Azure Open AI such my own IdP

I need to log all prompts and responses for a central AI safety team

If this sounds like you, then read on as we build a platform!

Gen AI Platform Engineers

Use Case:

So that my organisation can run many AI applications on a PTU
As a Gen AI Platform Engineer
I need to share a PTU fairly across consumers

Use Case:

So that I can charge-back my PTU deployments
As a Gen AI Platform Engineer
I need to report over token usage metrics

Use Case:

So that we can leverage existing security controls
As a Gen AI Platform Engineer
I need to use identities from an existing IdP

To deliver this feature we’re going to reach for an AI Gateway. These are becoming commonplace in APIm products. In our case let’s drop Azure API Management with its AI Gateway components into the architecture and see what happens:

This is nice – we’ve now introduced an AI gateway which will give us lots of extra functionality

We’ll expose an API from Azure APIm that looks and feels like the Azure Open AI API. This will make it easier for our consumers using frameworks that ‘expect’ the AOAI surface area. It also makes it simpler for our platform team as there’s only one API for them to maintain.
We can create new versions of this API as Open AI releases new versions.
We can also restrict the surface area of the Azure Open AI API.

Let’s say we don’t want to expose Assistants, or the ability to upload files. With this model we can just choose not to expose those APIs.

We can use APIm’s load balanced pools to balance requests.
We can get it to prioritise our PTU and fail over to PAYG if we run out of capacity.

This will reduce the likelihood of 429’s getting back to our consumers.

We can use APIm to authorise incoming requests using our own IdP or a Platform Entra Application.

This lets us use custom roles and have more control over the JWT’s our consumers need.

We can use APIm policies like token-rate-limiting to fairly share our PTU stopping greedy consumers.

We’ll use “products” to lower the blast radius of breaking our consumers.
When (not if!) the platform gets popular we will need to support many consumers with different token limits. Products let us model this as lots of small policy files. These will be easier to manage than one big one.

We can use policies that emit token count metrics (including for streaming endpoints) allowing chargeback scenarios.

GEN AI Engineers… part II

This is looking great… it’s an amazing platform but the Engineers are not happy… What we’ve built is a brilliant launchpad for engineers who want to build Generative AI Applications, but it doesn’t cater for all tooling they might want.

Use Case:

So that I can support libraries like Prompt flow
As a Gen AI Platform Engineer
I need to cater for assumptions the libraries make on authentication

Some AI Libraries were built before the idea of AI Platforms existed and have taken tight dependencies on the way the ‘old world’ worked. These will be fixed over time but for now our platform is going to have to deal with them.

Let’s take Promptflow as an example. Our platform makes heavy use of API Management’s AI Gateway products to reduce the blast radius of change.

As of August 2024, APIm requires a Subscription Key to trigger Product behaviour. Our fictitious security team has mandated OAuth2 for all API calls. If you use Prompt flow’s OAuth flow it acquires tokens against Azure Open AI’s scope. And it’s tricky to attach a subscription key to its calls. It’s low-level but it’s causing friction between the application teams and the security team. 

Prompt flow makes our authentication life a bit more difficult… We’re faced with a risk-based decision. Do we:

Allow Generative AI applications built with Prompt flow to take a less secure approach sending an APIm subscription-key
Allow Gen AI Apps to have direction permissions against Azure Open AI introducing a risk they could bypass our gateway protections and call Azure Open AI directly.
Introduce something ‘in the middle’ to adapt Prompt flow’s requests to meet our security requirements.

There are a few ways to approach this. Let us start with an approach that doesn’t add any new services. We are going to introduce a new product to our APIm AI Gateway which will:

Authorise the incoming JWT provided by Prompt flow and map the caller to a subscription key.
Make a new call back into APIm with the subscription key appended as a header to the original request

This will incur an impact on APIm’s performance so it’s not a free lunch. But it will enable us to reduce the risk of Prompt flow using just Api-Keys.

Great – what we can do now is:

Let our AI Applications use tokens acquired for Azure Open AI without giving them permissions to Azure Open AI

This is a nice trick here as we getting a token for AOAI doesn’t authorize us. Auth is done at the AOAI service.

The APIm product can:

Check the incoming claims on the JWT to authenticate the caller
Acquire a new token on-behalf-of the original (optional step)
Append a subscription key (using a pre-configured lookup)
Make the outbound call back to APIm where it will trigger the expected product behaviour.

For many organisations this might be enough. But there are still a few things we might care about. The OWASP LLM Top 10 identifies common vulnerabilities for LLM usage. Wouldn’t it be great if we can help our consumers detect some of them?

Bigger organisations tend to have AI Safety teams who need to report over all LLM consumption. They are asking questions like:

Can I get confidence prompts aren’t being jail-broken?

Can we be confident prompts are using grounding data, and not hallucinating?

Can we maintain a redacted audit log of all prompts and responses?

Let’s step into the shoes of our Responsible AI team…

Responsible AI team

Use Case:

So that my organisation stays within responsible AI boundaries
As a Responsible AI team
We want to spot check redacted prompt inputs, and outputs

Use Case:

So that my organisation stays within responsible AI boundaries
As a Responsible AI team
We want centralised alerting on attempted attacks on our Gen AI applications

Use Case:

So that my organisation stays within responsible AI boundaries
As a Responsible AI team
We want to know about CVE’s from LLM models and images our applications are using

We are going to reach for a few new tools to achieve these use-cases. AI Threat Detection from Azure Defender (https://learn.microsoft.com/en-us/azure/defender-for-cloud/ai-threat-protection) will help us with centralised alerts and CVE detection. Great news if we’re using Azure Open AI as it can take signals directly from the Content Safety layer meaning we don’t have to build anything to integrate the services.

For logging those prompts and responses we’re going to have to think outside the box. Most API Management platforms offer support for logging the bodies of requests and responses but there’s a caveat. They often don’t support streaming (Sever Side Events) which are used to provide better response latency to callers. If you’ve ever used Chat GPT you’ll have seen a streaming response in action. That ticker-tacker typewriter like experience when your response is returned is a Streaming Response.

API Management solutions currently buffer these responses in-order to log them which reduces the user-experience.

I’ll avoid the question of ‘do you need to use Server-Side Events’. It’s safe to say like all questions in IT, the answer is ’It Depends’. But what if you do need them?

To handle logging, I’ll introduce a second Gen AI Gateway into the architecture. It’s called AI Central https://github.com/microsoft/aicentral (there are lots of these out there – disclaimer, I am the primary maintainer of AI Central), and it runs as a Docker Container / Web API sitting in-front of APIm. Another good option is AI Sentry – https://github.com/microsoft/ai-sentry 

AI Central will drop prompts and responses into a queue for PII redaction and logging. It works with streaming responses and doesn’t buffer so won’t degrade the end user experience. It currently uses the Azure Language service, but we are investigating using the PTU overnight when it’s not used as much, or PHI-3 models running in a sidecar.

AI Central logs PII Redacted prompts and responses to a Cosmos Database that a Responsible AI Team can use.

We’ve also enabled Azure Defender for AI. The AI Content Filter built into AOAI is transmitting data to Azure Defender looking for attacks on their LLM. This is all surfaced in the standard Azure Defender dashboards.

But before we call ‘done’, let’s face into our last hurdle. How do or Gen AI Application Engineers onboard to this platform?

Gen AI Platform Engineers… part II

Use Case:

So that I can simplify onboarding
As a Gen AI Platform Engineer
I want to streamline consumer onboarding

Use Case:

So that I can manage consumer demand
As a Gen AI Platform Engineer
I want a tool to simplify managing multiple Azure Open AI deployments

Use Case:

So that I can deploy on a Friday
As a Gen AI Platform Engineer
I want to deploy daily

How does a feature team express their requirements to a platform team? We are using a JSON document which could be added via a Pull Request into the Platform team’s repository. Something like this should suffice. It will capture enough information about the consumer to:

Understand their token / model requirements
Understand their content safety requirements
Understand when they want to promote into environments
Get in touch with them
Support Chargeback

{
“consumerName”: “consumer-1”,
“requestName”: “my-amazing-service”,
“contactEmail”: “engineer.name@myorg.com”,
“costCentre”: “92304”,
“constantAppIdIdentifiers”: [],
“models”: [
{
“deploymentName”: “embeddings-for-my-purpose”,
“modelName”: “text-embedding-ada-002”,
“contentSafety”: “high”,
“environments”: {
“dev”: {
“thousandsOfTokens”: 1,
“deployAt”: “2024-07-02T00:00:0000”
},
“test”: {
“thousandsOfTokens”: 1,
“deployAt”: “2024-07-02T00:00:0000”
},
“prod”: {
“thousandsOfTokens”: 15,
“deployAt”: “2024-07-02T00:00:0000”
}
}
},
{
“deploymentName”: “gpt35-for-my-purpose”,
“modelName”: “gpt-35-turbo”,
“contentSafety” : “high”,
“environments”: {
“dev”: {
“thousandsOfTokens”: 1,
“deployAt”: “2024-07-02T00:00:0000”
},
“test”: {
“thousandsOfTokens”: 1,
“deployAt”: “2024-07-02T00:00:0000”
},
“prod”: {
“thousandsOfTokens”: 15,
“deployAt”: “2024-07-02T00:00:0000”
}
}
}
]
}

This will be a conversation. When a consumer opens a Pull Request to add / update this information into the Platform repository use the Pull Request to query / suggest alternate deployment approaches (maybe gpt-4o is a better fit for their requirement then gpt-4).

When you are comfortable with a request, merge the Pull Request into the platform repository.

Platform Team Mapping

Now you have the feature teams’ requirements, as a Platform team you need to express these as Azure Open AI deployments and API Management Products. This is unlikely to be a one-to-one mapping.

For example, to maximise use of PTU you might want to consolidate multiple consumer demands into a single deployment. To maximise PTU further you will need to consolidate Content Filter policies into ‘low’, ‘medium’, or ‘high’ (as content filter policies have one-to-one affinity to a deployment).

These are all backend implementation decisions. Your contract to the consumer is to provide them access to Azure Open AI with the agreed deployment names, throughput, and content-filters, securing using their provided Entra Application Ids.

Platform Team Decisions

How many AOAI services will you need? AOAI has limits on quotas per region. For example, as of August 2024 Australia East allows you 350k tokens per minute for Text-Embedding-Ada-002. https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits

If you have more demand than available capacity for a region / subscription pair, then you will need to deploy more Azure Open AI resources to either different subscriptions, or different regions. The code sample provided uses a single subscription, scaling out over multiple regions.

We’ll leave the science of sizing PTU deployments out-of-scope for here. There’s lots of good documentation out there for sizing PTUs, e.g. https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/right-size-your-ptu-deployment-and-save-big/ba-p/4053857. 

Back to our platform let’s start simple – we will define a deployment to AOAI using this JSON. Our platform will be built using a list of these.

{
“aoaiName”: “graemeopenai”,
“deploymentName”: “testdeploy2”,
“enableDynamicQuota”: false,
“deploymentType”: “PAYG”,
“model”: “gpt-35-turbo”,
“modelVersion”: “0613”,
“thousandsOfTokensPerMinute”: 5
}

Finally, we need a mapping between the demands of our feature team to the platform deployments. We will use this JSON:

{
“consumerName”: “consumer-1”,
“requirements”: [
{
“outsideDeploymentName”: “text-embedding-ada-002”,
“platformTeamDeploymentMapping”: “text-embedding-ada-002”,
“platformTeamPoolMapping”: “graemeopenai-embedding-pool”
},
{
“outsideDeploymentName”: “gpt35”,
“platformTeamDeploymentMapping”: “testdeploy2”,
“platformTeamPoolMapping”: “graemeopenai-pool”
}
]
}

The platform uses APIm policies to rewrite consumer requests to the ‘outside’ deployment name to the actual deployment names, allowing multiple consumers to share single deployments. This mapping lets the Platform team introduce new deployments potentially with different names without affecting the consumers.

In future iterations we want to write a simple User Interface to help manage this mapping exercise.

Deployment

Deployment of a platform should be no different to deployment of a feature. The more you do it, the more confident you will be. The Bicep templates that form the platform (https://github.com/graemefoster/APImAIPlatform) are designed to deploy everything – Azure Open AI services, deployments, APIs, Products, AI Central, etc in a single “line-of-sight”.

Our entire platform is deployed using a single ‘az deployment sub create‘ command. If we had to rebuild the entire thing from scratch it would be the same single deployment command.

My recommendation is to deploy daily at least and potentially more frequently.

Optional Extras

In building this document some other ideas popped up that I think would be helpful in making it even easier and quicker for your internal customers to get onboard:

Providing platform endpoints to test your Prompts against well-known jail-breaks and other OWASP LLM threats
Running random checks against audited prompts and responses to check the grounded-ness of the responses
CI / CD Pipelines that ‘collect’ an application’s prompts and responses and pro-actively run guardrail evaluations over them

Think about all the requirements that your consumers need to ‘tick’ before they can deploy into production. Platforms succeed when their customers fall into the proverbial pit-of-success. The more general requirements you can automate, the more your engineers are going to love your platform.

Final thoughts

And that’s it. We’ve covered a lot of detail but have setup:

Azure Open AI Deployments
An API Management AI Gateway capable of authenticating, load balancing and enforcing consumer quota
An AI proxy that can provide advanced logging as-well as bridging libraries like Prompt flow which cannot easily send Entra JWT’s and APIm Subscription Keys
Discussed AI Threat Detection from Azure Defender to assist SIEM monitoring
Thought about future enhancements that could make your platform even more valuable.

What do you think? How are you approaching this problem? Try the sample. Make it better! We’d love to hear from you so please leave some feedback.

​Microsoft Tech Community – Latest Blogs –Read More 

Hello ALL,
I need to retrieve values a cost, a rate to a table, which are stock in another table.
It is table fs_sro_labor_mst and
Those tables have a mutual column. It is work_code.

Total count the work codes is 2 934.

How can I figure out this issue?

Thanks for all solutions

​Hello ALL,I need to retrieve values a cost, a rate to a table, which are stock in another table.It is table fs_sro_labor_mst andThose tables have a mutual column. It is work_code. Total count the work codes is 2 934.How can I figure out this issue?Thanks for all solutions   Read More

​

Hi

Is the source code for the Azure Document Intelligence Studio frontend available? If so, where can I access it? If it’s not available, are there any example projects or resources that showcase similar functionality or UI components that I can refer to?

Thank you for your assistance!

​HiIs the source code for the Azure Document Intelligence Studio frontend available? If so, where can I access it? If it’s not available, are there any example projects or resources that showcase similar functionality or UI components that I can refer to?Thank you for your assistance!  Read More

​

I tried to sign in Outlook app android using work or school account tenant M365 A and tenant M365 B using personal android. There is no problem when I use account from tenant A, but when using account tenant M365 B it is always asked to “activate device administrator”.

I also tested using Outlook iOS/iPhone using account tenant M365 B, the result is the prompt does not appear and can log in to use email service.

I think this is a policy applied to Intune, but I also confirm that tenant M365 B does not have intune policy and does not subscribe to Intune. I do not know why this happens and I do not want to do “activate device administrator”.

Is there a way to overcome this, maybe it can be turned off for “activate device administrator”, so that the prompt does not appear again?

​I tried to sign in Outlook app android using work or school account tenant M365 A and tenant M365 B using personal android. There is no problem when I use account from tenant A, but when using account tenant M365 B it is always asked to “activate device administrator”. I also tested using Outlook iOS/iPhone using account tenant M365 B, the result is the prompt does not appear and can log in to use email service. I think this is a policy applied to Intune, but I also confirm that tenant M365 B does not have intune policy and does not subscribe to Intune. I do not know why this happens and I do not want to do “activate device administrator”. Is there a way to overcome this, maybe it can be turned off for “activate device administrator”, so that the prompt does not appear again?  Read More

​