Category: Microsoft
Category Archives: Microsoft
Limit on email recipients
I have a large group (approx. 700) email addresses for retirees from my former workplace, and I keep them informed of retiree events and noteworthy happenings (obituaries, new retirements, etc.). I have the retiree email addresses organized into groups of 100 (the limit from my Comcast email account) and send a message out to everyone by sending to the first group, then simply forwarding that message to the remaining 6 groups. Starting last week, my emails started being throttled down to the point where I can only send out to 2 groups, then I have to wait an hour before being able to send to the next 2 groups (otherwise I get an error message of a “policy violation”). Comcast support tells me they’re not throttling me down, that it’s Outlook doing that. Has anyone else had a similar experience? How do I convince Outlook (Microsoft) that I’m not a spammer? I tried calling a help line but the automated message sends me to online support, and I’ve found no help there.
I have a large group (approx. 700) email addresses for retirees from my former workplace, and I keep them informed of retiree events and noteworthy happenings (obituaries, new retirements, etc.). I have the retiree email addresses organized into groups of 100 (the limit from my Comcast email account) and send a message out to everyone by sending to the first group, then simply forwarding that message to the remaining 6 groups. Starting last week, my emails started being throttled down to the point where I can only send out to 2 groups, then I have to wait an hour before being able to send to the next 2 groups (otherwise I get an error message of a “policy violation”). Comcast support tells me they’re not throttling me down, that it’s Outlook doing that. Has anyone else had a similar experience? How do I convince Outlook (Microsoft) that I’m not a spammer? I tried calling a help line but the automated message sends me to online support, and I’ve found no help there. Read More
Best Practices for Tracking Changes on a SharePoint List Accessed by a Service Account
Hello Microsoft Community,
I am reaching out to seek advice and best practices for monitoring and tracking changes on a SharePoint list that is accessed via a service account. In our organization, multiple individuals have access to this service account, which raises concerns about maintaining accountability and transparency regarding the modifications made to the list.
The SharePoint list in question contains various columns, including calculated columns, which are crucial for our operations. However, these calculated columns present a challenge as they do not inherently track who made changes or when these changes were made. This lack of traceability is problematic, especially when trying to audit or review historical data.
I am looking for a method or tool that can help us keep a detailed log of all activities on the list. Ideally, this solution would capture:
Timestamps of when changes are made.Identification of the specific elements that were altered (from metadata to formulas).The nature of the changes (what columns was added, removed, or modified).
We want to ensure that our SharePoint list remains a reliable and accurate source of information for all team members. Therefore, having a system that can provide a comprehensive audit trail would be extremely beneficial.
I would greatly appreciate any recommendations on features within SharePoint, Power Automate, or custom solutions that could assist us in achieving this level of oversight. Additionally, if there are any best practices for managing shared service account access to SharePoint lists, I would be eager to learn about those as well.
Thank you in advance for your time and assistance.
Hello Microsoft Community, I am reaching out to seek advice and best practices for monitoring and tracking changes on a SharePoint list that is accessed via a service account. In our organization, multiple individuals have access to this service account, which raises concerns about maintaining accountability and transparency regarding the modifications made to the list. The SharePoint list in question contains various columns, including calculated columns, which are crucial for our operations. However, these calculated columns present a challenge as they do not inherently track who made changes or when these changes were made. This lack of traceability is problematic, especially when trying to audit or review historical data. I am looking for a method or tool that can help us keep a detailed log of all activities on the list. Ideally, this solution would capture:Timestamps of when changes are made.Identification of the specific elements that were altered (from metadata to formulas).The nature of the changes (what columns was added, removed, or modified).We want to ensure that our SharePoint list remains a reliable and accurate source of information for all team members. Therefore, having a system that can provide a comprehensive audit trail would be extremely beneficial. I would greatly appreciate any recommendations on features within SharePoint, Power Automate, or custom solutions that could assist us in achieving this level of oversight. Additionally, if there are any best practices for managing shared service account access to SharePoint lists, I would be eager to learn about those as well. Thank you in advance for your time and assistance. Read More
Resource room calendar not updating meeting times: Intermittent
I’ve encountered a situation where a user creates a meeting with a resource room attached. Sometimes, when she updates the meeting times and forwards the invitation to all users, their times update accordingly, but the meeting room calendar does not.
Although she receives the accepted time email from the meeting room email, the time does not actually update on the calendar.
When testing with her account on another meeting made the exact same way it worked without issue.
Deleting and recreating the meeting worked fine but she’s mentioned this multiple times and it continues to be dismissed because upon testing it works as expected.
I’ve encountered a situation where a user creates a meeting with a resource room attached. Sometimes, when she updates the meeting times and forwards the invitation to all users, their times update accordingly, but the meeting room calendar does not. Although she receives the accepted time email from the meeting room email, the time does not actually update on the calendar.When testing with her account on another meeting made the exact same way it worked without issue.Deleting and recreating the meeting worked fine but she’s mentioned this multiple times and it continues to be dismissed because upon testing it works as expected. Read More
trying to edit entire column
how do I apply this formula to the entire column B3 to B1873?
=LEFT(B3, LEN(B3) – 10)
Also, I have a column with first and last name together, is there a way to separate these into 2 columns?
how do I apply this formula to the entire column B3 to B1873? =LEFT(B3, LEN(B3) – 10) Also, I have a column with first and last name together, is there a way to separate these into 2 columns? Read More
Re-importing Calendar items from In-Place Archive, back into live Exchange mailbox?
I have a user whose primary Exchange Online mailbox had hit the 50GB limit, so In-Place Archiving needed to be turned on in order to make room in live mailbox.
However, the user wants to be able to have ALL the Calendar items accessible in the live mailbox. They are fine with the Archived email items remaining in the In-Place Archive. So the questions I have are:
If I set the user’s MRM policy on the Archive Mailbox to ‘No Policy’ but leave In-Place Archving active, would I be able to export the Calendar items from the In-Place Archive to a PST (via Outlook) and then re-import them into the live mailbox? If the above were possible would all the Calendar items import back into the live mailbox’s calendar in the same state they were in, in the In-Place Archive (i.e accepted, etc.)If the items reimport, will Outlook Reminders get triggered for all the re-imported items, in that the user would get a whole list of Calendar items showing up as Overdue?
I have a user whose primary Exchange Online mailbox had hit the 50GB limit, so In-Place Archiving needed to be turned on in order to make room in live mailbox.However, the user wants to be able to have ALL the Calendar items accessible in the live mailbox. They are fine with the Archived email items remaining in the In-Place Archive. So the questions I have are: If I set the user’s MRM policy on the Archive Mailbox to ‘No Policy’ but leave In-Place Archving active, would I be able to export the Calendar items from the In-Place Archive to a PST (via Outlook) and then re-import them into the live mailbox? If the above were possible would all the Calendar items import back into the live mailbox’s calendar in the same state they were in, in the In-Place Archive (i.e accepted, etc.)If the items reimport, will Outlook Reminders get triggered for all the re-imported items, in that the user would get a whole list of Calendar items showing up as Overdue? Read More
selecting a value based on criteria from three columns
I have a table with 13 columns. Two additional columns need to be calculated based on values in columns 1-13. Specifically as follows:
If G2=”C” and if B2=”Y” and E2=””, then the column M1 Value should = “HIG”
If G2=”C” and if B2=”” and E2=””, then the column M1 Value should = “TP”
If G2=”C” and if B2=”Y” and E2=”CTIM”, then the column M1 Value should = “TP”
If G2=”T” and if B2=”Y” and E2=””, then the column M1 Value should = “GFP”
If G2=”T” and if B2=”” and E2=””, then the column M1 Value should = “SP”
If G2=”T” and if B2=”Y” and E2=”BDSF”, then the column M1 Value should = “SP”
If G2=”O” then the column M1 Value should = “C”
If G2=”C” and if B2=”Y” and E2=””, then the column M1 Value should =
If G2=”” and if A2=not Blank, then the column M1 Value should = Error
A1B1C1D1E1F1G1H1I1J1K1L1M1ABC-DEF-GHIJ-0YSTARTRED NT N5/17/202406:21POUT BCD-EFG-HIJK-0YSTOP
GREEN
BDSF C CDE-FGH-IJKL-0YSTARTBLUE
I am thinking that a VLOOKUP would probably work best, but my brain just can’t close the loop. Thanks
I have a table with 13 columns. Two additional columns need to be calculated based on values in columns 1-13. Specifically as follows:If G2=”C” and if B2=”Y” and E2=””, then the column M1 Value should = “HIG”If G2=”C” and if B2=”” and E2=””, then the column M1 Value should = “TP”If G2=”C” and if B2=”Y” and E2=”CTIM”, then the column M1 Value should = “TP”If G2=”T” and if B2=”Y” and E2=””, then the column M1 Value should = “GFP”If G2=”T” and if B2=”” and E2=””, then the column M1 Value should = “SP”If G2=”T” and if B2=”Y” and E2=”BDSF”, then the column M1 Value should = “SP”If G2=”O” then the column M1 Value should = “C”If G2=”C” and if B2=”Y” and E2=””, then the column M1 Value should =If G2=”” and if A2=not Blank, then the column M1 Value should = ErrorA1B1C1D1E1F1G1H1I1J1K1L1M1ABC-DEF-GHIJ-0YSTARTRED NT N5/17/202406:21POUT BCD-EFG-HIJK-0YSTOPGREENBDSF C CDE-FGH-IJKL-0YSTARTBLUE I am thinking that a VLOOKUP would probably work best, but my brain just can’t close the loop. Thanks Read More
All the locations where you can find Sensitivity labels
Here are the locations where you can find the sensitivity label of a document (if there are any that I’ve missed, please feel free to add it here)
Sensitivity Label Button in the Document:
In Office applications such as Word, Excel, and PowerPoint, you can find the Sensitivity label button on the Home tab. This button allows users to apply or view sensitivity labels directly within the document interface.
(Sensitivity label app on the upper right)
(the bottom left will show the label applied to the document)
Document Properties > Advanced Properties
Sensitivity labels can also be found in the document properties. To access this, go to File > Info > Properties > Advanced Properties. Here, you can see detailed metadata, including any applied sensitivity labels.
Sensitivity Label Column in SharePoint:
In SharePoint, sensitivity labels are displayed in a dedicated column. This allows users to quickly see the sensitivity level of documents stored within SharePoint libraries
Windows File Explorer:
Sensitivity labels can be extended to Windows File Explorer, allowing users to apply and view labels directly from their file management interface.
Mobile Applications:
Office mobile apps for iOS and Android also support sensitivity labels, enabling users to apply and view labels on the go.
Microsoft Purview Compliance Portal:
Administrators can manage and view sensitivity labels applied across the organization through the Microsoft Purview Compliance Portal. This portal is only accessible to IT admins who has the right Purview role.
Here are the locations where you can find the sensitivity label of a document (if there are any that I’ve missed, please feel free to add it here) Sensitivity Label Button in the Document:In Office applications such as Word, Excel, and PowerPoint, you can find the Sensitivity label button on the Home tab. This button allows users to apply or view sensitivity labels directly within the document interface. (Sensitivity label app on the upper right)(the bottom left will show the label applied to the document) Document Properties > Advanced PropertiesSensitivity labels can also be found in the document properties. To access this, go to File > Info > Properties > Advanced Properties. Here, you can see detailed metadata, including any applied sensitivity labels. Sensitivity Label Column in SharePoint:In SharePoint, sensitivity labels are displayed in a dedicated column. This allows users to quickly see the sensitivity level of documents stored within SharePoint libraries Windows File Explorer:Sensitivity labels can be extended to Windows File Explorer, allowing users to apply and view labels directly from their file management interface. Mobile Applications:Office mobile apps for iOS and Android also support sensitivity labels, enabling users to apply and view labels on the go. Microsoft Purview Compliance Portal:Administrators can manage and view sensitivity labels applied across the organization through the Microsoft Purview Compliance Portal. This portal is only accessible to IT admins who has the right Purview role. Read More
Look up a name based on multiple date
Hi,
I am looking for a formula to return the person’s name if they are managing a location.
So on the below table I would like to show in Column A the following
A2 – Andrea
A3 – Bill
A4 – No match
A5 – Dave
A6 – Elliott
A7 – No match
Example
Hi, I am looking for a formula to return the person’s name if they are managing a location. So on the below table I would like to show in Column A the following A2 – Andrea A3 – BillA4 – No matchA5 – DaveA6 – Elliott A7 – No match Example Read More
Formatting order numbers
I need help with the ROW function: I am trying to set up a document to keep track of orders where each row represents one item in the order. some orders are for only one item, some are for several, so I need the “Order Number” column to format the order number as “24-[order number]” but only in cells where a new order is entered (the first row in a multi-item order). I can’t figure out a formula that lets me both enter a sequence (automatically format as “24-0001” “24-0002″ etc) and skip rows while keeping track of skipped rows. What I have now is =IF(ISBLANK(J2),” “,TEXT(ROW(A1),”24-0000”)) – this keeps multi-item rows except the first blank but resumes the count as though the blank rows count.
I need help with the ROW function: I am trying to set up a document to keep track of orders where each row represents one item in the order. some orders are for only one item, some are for several, so I need the “Order Number” column to format the order number as “24-[order number]” but only in cells where a new order is entered (the first row in a multi-item order). I can’t figure out a formula that lets me both enter a sequence (automatically format as “24-0001” “24-0002″ etc) and skip rows while keeping track of skipped rows. What I have now is =IF(ISBLANK(J2),” “,TEXT(ROW(A1),”24-0000”)) – this keeps multi-item rows except the first blank but resumes the count as though the blank rows count. Read More
I believe all owners of i7-7700k and any other processor that meets minimums but not supported…
Should sue microsoft, this has to be some kind of anti trust violation or something. It is extremely unethical to force people with a perfectly capable, extremely solid and well running processor that meets the minimum requirements to be blacklisted from access to something all other customers are getting. We need to sue their pants off until they allow all processors that check all of their stupid little boxes. They should not be allowed to just blacklist some processors due to a backroom deal with intel to try to force us to buy a new processor.
Requiring secure boot and tpm 2.0 is fine, that is understandable. if the software needs those features fine, but the i7-7700k and other processors that are blacklisted have those things, and therefore should not be blocked from normal update proceedures. Either that or a court should force them to continue to support windows 10 for all processors that meet the 11 minimum requirements but are blacklisted for absolutely no good reason.
Should sue microsoft, this has to be some kind of anti trust violation or something. It is extremely unethical to force people with a perfectly capable, extremely solid and well running processor that meets the minimum requirements to be blacklisted from access to something all other customers are getting. We need to sue their pants off until they allow all processors that check all of their stupid little boxes. They should not be allowed to just blacklist some processors due to a backroom deal with intel to try to force us to buy a new processor.Requiring secure boot and tpm 2.0 is fine, that is understandable. if the software needs those features fine, but the i7-7700k and other processors that are blacklisted have those things, and therefore should not be blocked from normal update proceedures. Either that or a court should force them to continue to support windows 10 for all processors that meet the 11 minimum requirements but are blacklisted for absolutely no good reason. Read More
Have I been hacked?
Hello,
Someone send me an email telling me I have been hacked..
It all seems nonsense
A (He hacked my PC, he says, but that seems not true, as he would have used ransomware
B he told me my old password was my password but it is an old password from long ago)
C he didn’t change any of my details in my hotmail account (or any other accounts).. no recovery email change, no password change, and so on
Only thing made me believe him was that the email he sent from is exactly identical to mine.. even when I try to click to have the real info.. it stills shows mine
I had a nice idea to download eml.. put it in viewer to say what the real value is.. ‘ from ‘ part is identical my email exactly.
Note I tried the feature which view actual account sent from when clicking the first one.. other emails show normally
This persons email, however, still shows mine
Someone told me this spoofing can work against outlook it self without detecting actual address.
What can I do to confirm that this is a non harmful claim of hacking..
Please note I am securing everything on my PC, accounts,.. and already started using 1pssword
Hello,Someone send me an email telling me I have been hacked..It all seems nonsenseA (He hacked my PC, he says, but that seems not true, as he would have used ransomwareB he told me my old password was my password but it is an old password from long ago)C he didn’t change any of my details in my hotmail account (or any other accounts).. no recovery email change, no password change, and so on Only thing made me believe him was that the email he sent from is exactly identical to mine.. even when I try to click to have the real info.. it stills shows mineI had a nice idea to download eml.. put it in viewer to say what the real value is.. ‘ from ‘ part is identical my email exactly. Note I tried the feature which view actual account sent from when clicking the first one.. other emails show normally This persons email, however, still shows mine Someone told me this spoofing can work against outlook it self without detecting actual address. What can I do to confirm that this is a non harmful claim of hacking.. Please note I am securing everything on my PC, accounts,.. and already started using 1pssword Read More
Simple Calculation
I am simply trying to subtract one cell from another. Doing it manually. Click the + and it highlights 2 cells including the one I am trying to select. I have cleared Content, reformatted alignment – the tricks that usually work. Does anyone have the secret solution?
I am simply trying to subtract one cell from another. Doing it manually. Click the + and it highlights 2 cells including the one I am trying to select. I have cleared Content, reformatted alignment – the tricks that usually work. Does anyone have the secret solution? Read More
Não consigo me manter conectada na conta do Microsoft Learn
“Me conecto” com minha conta no Microsoft Learn e não consigo acessar nada além da página “você está conectado como”, aparece meu e-mail e pede para eu avançar para fazer meu perfil, quando avanço tudo e faço meu perfil ele volta para essa página do “você está conectado como” e volta tudo de novo.
Já tentei acessar todas as páginas atrás desse login mas sempre volta a página do “você está conectado como”. Tinha um prazo para terminar uma trilha de aprendizagem mas acabei perdendo por conta disso e queria saber como faço para recuperar esse prazo já que o problema não está no meu computador, celular e nem em minhas contas, já que tentei fazer login com 3 contas diferentes…Fiz a trilha de aprendizagem sem estar ligada na conta e consegui, quando fui emitir o certificado e pediu meu email, não consegui acessar de novo e voltou para a página do “você está conectado como” e aparece meu email em baixo. Não sai disso e não sei mais o que fazer!
”Me conecto” com minha conta no Microsoft Learn e não consigo acessar nada além da página “você está conectado como”, aparece meu e-mail e pede para eu avançar para fazer meu perfil, quando avanço tudo e faço meu perfil ele volta para essa página do “você está conectado como” e volta tudo de novo.Já tentei acessar todas as páginas atrás desse login mas sempre volta a página do “você está conectado como”. Tinha um prazo para terminar uma trilha de aprendizagem mas acabei perdendo por conta disso e queria saber como faço para recuperar esse prazo já que o problema não está no meu computador, celular e nem em minhas contas, já que tentei fazer login com 3 contas diferentes…Fiz a trilha de aprendizagem sem estar ligada na conta e consegui, quando fui emitir o certificado e pediu meu email, não consegui acessar de novo e voltou para a página do “você está conectado como” e aparece meu email em baixo. Não sai disso e não sei mais o que fazer! Read More
Link certain data from one tab to another based on condition
Hello,
I have one sheet and on the first tab I have the main data of leads. This data is in order of thousands and I want to pull the information of a particular lead in another tab only if for example in one cell I put the word hot or warm (I use a dropdown list for this). In the other sheet it will be approximately 20% of them, just the ones who are potentials for my business. How to create a formula to put them in order in the next available empty row based on the condition that I selected hot or warm in a particular cell in the main sheet.
thank you
Hello, I have one sheet and on the first tab I have the main data of leads. This data is in order of thousands and I want to pull the information of a particular lead in another tab only if for example in one cell I put the word hot or warm (I use a dropdown list for this). In the other sheet it will be approximately 20% of them, just the ones who are potentials for my business. How to create a formula to put them in order in the next available empty row based on the condition that I selected hot or warm in a particular cell in the main sheet. thank you Read More
Missing information for AZ-500 exam preparation
Hello everybody.
What’s going on with the Microsoft Certified: Azure Security Engineer Associate certification? The “Prepare for the exam” box in the certification’s page is empty, like there’s no exam preparation resource.
Thank you.
Hello everybody.What’s going on with the Microsoft Certified: Azure Security Engineer Associate certification? The “Prepare for the exam” box in the certification’s page is empty, like there’s no exam preparation resource.Thank you. Read More
How to Install a Copy Hook shell extension for a UWP app
I’m packaging my classic Win32 application as a MSIX package to be distributed on MS Store.
The application includes a shell extension that implements `ICopyHook`. With the traditional installer, the shell extension DLL is registered under the `CopyHookHandlers` registry key.
With MSIX, the extension has to be declared in the AppManifest.xml file. However, per documentation [1], there is no way to add a copy hooker handler extension. The only handler types available are context menu and drag and drop.
This is an excerpt of my AppManifest.xml:
<com:Extension Category=”windows.comServer”>
<com:ComServer>
<com:SurrogateServer>
<com:Class Id=”cf1cbb8d-897c-45dc-b1a9-925201981d67″ Path=”VFSProgramFilesX64MyApplicationshellext.dll” ThreadingModel=”STA” />
</com:SurrogateServer>
</com:ComServer>
</com:Extension>
<desktop9:Extension Category=”windows.fileExplorerClassicDragDropContextMenuHandler”>
<desktop9:FileExplorerClassicDragDropContextMenuHandler>
<desktop9:ExtensionHandler Type=”*” Clsid=”cf1cbb8d-897c-45dc-b1a9-925201981d67″ />
</desktop9:FileExplorerClassicDragDropContextMenuHandler>
</desktop9:Extension>
I tried to manually register the shell extension DLL from the main app code, eg with `LoadLibrary`, `GetProcAddress` for the `DllRegisterServer` function and calling `DllRegisterServer`.
I get no error but the registry entries are not created.
Is there support for copy hooker shell extensions with AppManifest.xml? Is there a workaround?
I’m packaging my classic Win32 application as a MSIX package to be distributed on MS Store. The application includes a shell extension that implements `ICopyHook`. With the traditional installer, the shell extension DLL is registered under the `CopyHookHandlers` registry key. With MSIX, the extension has to be declared in the AppManifest.xml file. However, per documentation [1], there is no way to add a copy hooker handler extension. The only handler types available are context menu and drag and drop. This is an excerpt of my AppManifest.xml: <com:Extension Category=”windows.comServer”><com:ComServer><com:SurrogateServer><com:Class Id=”cf1cbb8d-897c-45dc-b1a9-925201981d67″ Path=”VFSProgramFilesX64MyApplicationshellext.dll” ThreadingModel=”STA” /></com:SurrogateServer></com:ComServer></com:Extension><desktop9:Extension Category=”windows.fileExplorerClassicDragDropContextMenuHandler”><desktop9:FileExplorerClassicDragDropContextMenuHandler><desktop9:ExtensionHandler Type=”*” Clsid=”cf1cbb8d-897c-45dc-b1a9-925201981d67″ /></desktop9:FileExplorerClassicDragDropContextMenuHandler></desktop9:Extension> I tried to manually register the shell extension DLL from the main app code, eg with `LoadLibrary`, `GetProcAddress` for the `DllRegisterServer` function and calling `DllRegisterServer`.I get no error but the registry entries are not created. Is there support for copy hooker shell extensions with AppManifest.xml? Is there a workaround? [1] https://learn.microsoft.com/en-us/uwp/schemas/appxpackage/uapmanifestschema/element-desktop9-extension Read More
Load Testing RAG based Generative AI Applications
Building an Effective Strategy
Mastering Evaluation Techniques
How-To Guides
Building an Effective Strategy
Identifying What to Evaluate
The user interacts with the frontend UI to pose a question.
The frontend service forwards the user’s question to the Orchestrator.
The Orchestrator retrieves the user’s conversation history from the database.
The Orchestrator accesses the AI Search key stored in the Key Vault.
The Orchestrator retrieves relevant documents from the AI Search index.
The Orchestrator uses Azure OpenAI to generate a user response.
The connection from the App Service to Storage Account indicates the scenario when the user wants to view the document that grounds the provided answer.
The connection from the App Service to Speech Services indicates the cases when the user wishes to interact with the application through audio.
Test Scenario
RPM = (u * p * s * i) / n / 60
u=10000 (total users)
p=0.1 (percentage of active users during peaktime)
s=1 (sessions per user)
i=2 (interactions per session)
n=1 (peaktime duration in hours)
Test Data
Test Measurements
Client metrics
Metric
Description
Number of Virtual Users
This metric shows the virtual user count during a load test, helping assess application performance under different user loads.
Requests per Second
This is the rate at which requests are sent to the LLM App during the load test. It’s a measure of the load your application can handle.
Response Time
This refers to the duration between sending a request and receiving the full response. It does not include any time spent on client-side response processing or rendering.
Latency
The latency of an individual request is the total time from just before sending the request to just after the first response is received.
Number of Failed Requests
This is the count of requests that failed during the load test. It helps identify the reliability of your application under stress.
Simplified example of the breakdown of request response time.
Performance Metrics for a LLM
Metric
Description
Number Prompt Tokens per Minute
Rate at which the client sends prompts to the OpenAI model.
Number Generated Tokens per Min
Rate at which the OpenAI model generates response tokens.
Time to First Token (TTFT)
The time interval between the start of the client’s request and the arrival of the first response token.
Time Between Tokens (TBT)
Time interval between consecutive response tokens being generated.
Server metrics
Service Name
Metric
Description
Azure OpenAI
Azure OpenAI Requests
Total calls to Azure OpenAI API.
Azure OpenAI
Generated Completion Tokens
Output tokens from Azure OpenAI model.
Azure OpenAI
Processed Inference Tokens
The number of input and output tokens that are processed by the Azure OpenAI model.
Azure OpenAI
Provision-managed Utilization V2
The percentage of the provisioned-managed deployment that is currently being used.
Azure App Service
CPU Percentage
The percentage of CPU used by the App backend services.
Azure App Service
Memory Percentage
The percentage of memory used by the App backend services.
Azure Cosmos DB
Total Requests
Number of requests made to Cosmos DB.
Azure Cosmos DB
Provisioned Throughput
The amount of throughput that has been provisioned for a container or database.
Azure Cosmos DB
Normalized RU Consumption
The normalized request unit consumption based on the provisioned throughput.
Azure API Management
Total Requests
Total number of requests made to APIM.
Azure API Management
Capacity
Percentage of resource and network queue usage in APIM instance.
When should I evaluate performance?
Enterprise LLM Lifecycle.
Mastering Evaluation Techniques
Great job on your journey so far in learning the essentials of your testing strategy! As we proceed in this section, we will be examining two distinct evaluation techniques. The first technique will concentrate on the performance testing of the entire LLM application, while the second will be primarily focused on testing the deployed LLM. It’s important to remember that these are just two popular instances from a wide-ranging list. Depending on your unique performance requirements, integrating other techniques into your testing strategy may prove beneficial.
LLM App Load Testing
Concept
Description
Test
Refers to a performance evaluation setup that assesses system behavior under simulated loads by configuring load parameters, test scripts, and target environments.
Test Run
Represents the execution of a Test.
Test Engine
Engine that runs the JMeter test scripts. Adjust load test scale by configuring test engine instances.
Threads
Are parallel threads in JMeter that represent virtual users. They are limited to a maximum of 250.
Virtual Users (VUs)
Simulate concurrent users. Calculated as threads * engine instances.
Ramp-up Time
Is the time required to reach the maximum number of VUs for the load test.
Latency
The latency of an individual request is the total time from just before sending the request to just after the first response is received.
Response Time
This refers to the duration between sending a request and receiving the full response. It does not include any time spent on client-side response processing or rendering.
You can securely store keys and credentials used during the test as Azure Key Vault secrets, and Azure Load Testing can also have its managed identity for access to Azure resources. When deployed within your virtual network, it can generate load directed at your application’s private endpoint. Application authentication through access tokens, user credentials, or client certificates is also supported, depending on your application’s requirements.
Monitoring Application Resources
Load Testing Automation
az loadtest create
–name $loadTestResource
–resource-group $resourceGroup
–location $location
–test-file @path-to-your-jmeter-test-file.jmx
–configuration-file @path-to-your-load-test-config.yaml
az loadtest run
–name $loadTestResource
–resource-group $resourceGroup
–test-id $testId
Key Metrics to Monitor During Load Tests
Request Rate: Monitor the request rate during load testing. Ensure that the LLM application can handle the expected number of requests per second.
Response Time: Analyze response times under different loads. Identify bottlenecks and optimize slow components.
Throughput: Measure the number of successful requests per unit of time. Optimize for higher throughput.
Resource Utilization: Monitor CPU, memory, and disk usage. Ensure efficient resource utilization.
Best Practices for Executing Load Tests
Test Scenarios: Create realistic test scenarios that mimic actual user behavior
Ramp-Up Strategy: Gradually increase the load to simulate real-world traffic patterns. The warm-up period typically lasts between 20 to 60 seconds. After the warm-up, the actual load test begins
Think Time: Include think time between requests to simulate user interactions.
Geographical Distribution: Test from different Azure regions to assess global performance.
Performance Tuning Strategies for LLM Apps
Application Design
Optimize Application Code: Examine and refine the algorithms and backend systems of your LLM application to increase efficiency. Utilize asynchronous processing methods, such as Python’s async/await, to elevate application performance. This method allows data processing without interrupting other tasks.
Batch Processing: Batch LLM requests whenever possible to reduce overhead. Grouping multiple requests for simultaneous processing improves throughput and efficiency by allowing the model to better leverage parallel processing capabilities, thereby optimizing overall performance.
Implement Caching: Use caching for repetitive queries to reduce the application’s load and speed up response times. This is especially beneficial in LLM applications where similar questions are frequently asked. Caching answers to common questions minimizes the need to run the model repeatedly for the same inputs, saving both time and computational resources. Some examples of how you can implement this include using Redis as a semantic cache or Azure APIM policies.
Revisit your Retry Logic: LLM model deployments might start to operate at their capacity, which can lead to 429 errors. A well-designed retry mechanism can help maintain application responsiveness. With the OpenAI Python SDK, you can opt for an exponential backoff algorithm. This algorithm gradually increases the wait time between retries, helping to prevent service overload. Additionally, consider the option of falling back on another model deployment. For more information, refer to the load balance item in the Solution Architecture section.
Prompt Design
Generate Less Tokens: To reduce model latency, create concise prompts and limit token output. According to the OpenAI latency optimization guide, cutting 50% of your output tokens can reduce latency by approximately 50%. Utilizing the ‘max_tokens’ parameter can also expedite response time.
Optimize Your Prompt: If dealing with large amounts of context data, consider prompt compression methods. Approaches like those offered by LLMLingua-2, fine-tuning the model to reduce lengthy prompts, eliminating superfluous RAG responses, and removing extraneous HTML can be efficient. Trimming your prompt by 50% might only yield a latency reduction of 1-5%, but these strategies can lead to more substantial improvements in performance.
Refine Your Prompt: Optimize the prompt text by placing dynamic elements, such as RAG results or historical data, toward the end of your prompt. This enhances compatibility with the KV cache system commonly used by most large language model providers. As a result, fewer input tokens need processing with each request, increasing efficiency.
Use Smaller Models: Whenever possible, pick smaller models because they are faster and more cost-effective. You can improve their responses by using detailed prompts, a few examples, or by fine-tuning.
Solution Architecture
Provisioned Throughput Deployments: When using Azure OpenAI use provisioned throughput in scenarios requiring stable latency and predictable performance, avoiding the ‘noisy neighbor’ issue in regular standard deployments.
Load Balancing LLM Endpoints: Implement load balancing for LLM deployment endpoints. Distribute the workload dynamically to enhance performance based on endpoint latency. Establish suitable rate limits to prevent resource exhaustion and ensure stable latency.
Resource Scaling: If services show strain under increased load, consider scaling up resources. Azure allows seamless scaling of CPU, RAM, and storage to meet growing demands.
Network Latency: Position Azure resources, like the Azure OpenAI service, near your users geographically to minimize network latency during data transmission to and from the service.
Azure OpenAI Benchmarking
Test Parameters
The benchmarking tool contains a number of configuration parameters to configure the test, as well as two script entry points. The benchmark.bench entry point is the basic script point, while the benchmark.contrib.batch_runner entry point can run batches of multiple workload configurations, and will automatically warm up the model endpoint prior to each test workload. It is recommended to use the batch_runner entry point to ensure accurate results and a much simpler testing process, especially when running tests for multiple workload profiles or when testing with PTU model deployments.
Parameter
Description
rate
Controls the frequency of requests in Requests Per Minute (RPM), allowing for detailed management of test intensity.
clients
Enables you to specify the number of parallel clients that will send requests simultaneously, providing a way to simulate varying levels of user interaction.
context-generation-method
Allows you to select whether to automatically generate the context data for the test (–context-generation-method generate), or whether to use existing messages data for the test (–context-generation-method replay)
shape-profile
Adjusts the request characteristics based on the number of context and generated tokens, enabling precise testing scenarios that reflect different usage patterns. Options include “balanced”, “context”, “custom” or “generation”.
context-tokens (for custom shape-profile)
When context-generation-method = generate and shape-profile = custom, this allows you to specify the number of context tokens in the request.
max-tokens (for custom shape-profile)
This allows you to specify the maximum number of tokens that should be generated in the response.
aggregation-window
Defines the duration, in seconds, for which the data aggregation window spans. Before the test hits the aggregation-window duration, all stats are computed over a flexible window, equivalent to the elapsed time. This ensures accurate RPM/TPM stats even if the test ends early due to hitting the request limit. A value of 60 seconds or more is recommended.
log-save-dir
If provided, the test log will be automatically saved to the directory, making analysing and comparing different benchmarking runs simple.
Warming up PTU endpoints
Retry Strategy
Output Metrics
measure
description
ttft
Time to First Token. Time in seconds from the beginning of the request until the first token was received.
tbt
Time Between Tokens. Time in seconds between two consecutive generated tokens.
e2e
End to end response time.
context_tpr
Number of context tokens per request.
gen_tpr
Number of generated tokens per request.
util
Azure OpenAI deployment utilization percentage as reported by the service (only for PTU deployments).
Sample Scenarios
1. Using the benchmark.bench entrypoint
–deployment gpt-4
–rate 60
–retry none
–log-save-dir logs/
https://myaccount.openai.azure.com
2023-10-19 18:21:06 INFO using shape profile balanced: context tokens: 500, max tokens: 500
2023-10-19 18:21:06 INFO warming up prompt cache
2023-10-19 18:21:06 INFO starting load…
2023-10-19 18:21:06 rpm: 1.0 requests: 1 failures: 0 throttled: 0 ctx tpm: 501.0 gen tpm: 103.0 ttft avg: 0.736 ttft 95th: n/a tbt avg: 0.088 tbt 95th: n/a e2e avg: 1.845 e2e 95th: n/a util avg: 0.0% util 95th: n/a
2023-10-19 18:21:07 rpm: 5.0 requests: 5 failures: 0 throttled: 0 ctx tpm: 2505.0 gen tpm: 515.0 ttft avg: 0.937 ttft 95th: 1.321 tbt avg: 0.042 tbt 95th: 0.043 e2e avg: 1.223 e2e 95th: 1.658 util avg: 0.8% util 95th: 1.6%
2023-10-19 18:21:08 rpm: 8.0 requests: 8 failures: 0 throttled: 0 ctx tpm: 4008.0 gen tpm: 824.0 ttft avg: 0.913 ttft 95th: 1.304 tbt avg: 0.042 tbt 95th: 0.043 e2e avg: 1.241 e2e 95th: 1.663 util avg: 1.3% util 95th: 2.6%
2. Using the benchmark.contrib.batch_runner entrypoint
context_tokens=500, max_tokens=100, rate=20
context_tokens=3500, max_tokens=300, rate=7.5
With the num-batches and batch-start-interval parameters, it will also run the same batch of tests every hour over the next 4 hours:
–deployment gpt-4-1106-ptu –context-generation-method generate
–token-rate-workload-list 500-100-20,3500-300-7.5 –duration 130
–aggregation-window 120 –log-save-dir logs/
–start-ptum-runs-at-full-utilization true –log-request-content true
–num-batches 5 –batch-start-interval 3600
For more detailed examples, refer to the README within the repository.
Processing and Analyzing the Log Files
After running the tests, the separate logs can be automatically processed and combined into a single output CSV. This CSV will contain all configuration parameters, aggregate performance metrics, and the timestamps, call status and content of every individual request.
With the combined CSV file, the runs can now easily be compared to each other, and with the individual request data, more detailed graphs that plot all request activity over time can be generated.
Monitoring AOAI Resource
| where TimeGenerated between(datetime(2024-04-26T15:30:00) .. datetime(2024-04-26T16:30:00))
| where OperationName == “ChatCompletions_Create”
| project TimeGenerated, _ResourceId, Category, OperationName, DurationMs, ResultSignature, properties_s
How-To Guides
LLM RAG application testing with Azure Load Testing.
Model deployment testing with AOAI Benchmarking Tool.
Wrapping Up
In conclusion, performance evaluation is crucial in optimizing LLM applications. By understanding your application’s specifics, creating an efficient strategy, and utilizing appropriate tools, you can tackle performance issues effectively. This boosts user experience and ensures that your application can handle real-world demands. Regular performance evaluations using methods such as load testing, benchmarking, and continuous monitoring can lead to your LLM application’s ultimate success.
Microsoft Tech Community – Latest Blogs –Read More
deleted emails on Mac
Hello.
When I delete an email or put it in a folder, it reappears again the next day. How can I stop this?
Thank you!
Hello. When I delete an email or put it in a folder, it reappears again the next day. How can I stop this?Thank you! Read More
Bookings: Category is not saving
Hi Community,
I am running a personal bookings page with different types of bookings. For administraton I use the online-site. When I created them, I added a category to them to make it visible in my calender “this has been made via bookings page”. Now I wanted to change the color and it does not save. Even worse, the old color category was deleted while editing and I cannot add a new one.
Suprisingly, everything else can be changed AND saved. There is no error message, the online site even reminds me to save after altering the category. But it does not do it. Using the teams-app (as suggested in other cases) did not solve it.
Does anyone have any ideas..?
Thanks and best regards – Markus
Hi Community, I am running a personal bookings page with different types of bookings. For administraton I use the online-site. When I created them, I added a category to them to make it visible in my calender “this has been made via bookings page”. Now I wanted to change the color and it does not save. Even worse, the old color category was deleted while editing and I cannot add a new one. Suprisingly, everything else can be changed AND saved. There is no error message, the online site even reminds me to save after altering the category. But it does not do it. Using the teams-app (as suggested in other cases) did not solve it. Does anyone have any ideas..? Thanks and best regards – Markus Read More
Bookings
Hi there,
I am trying to launch MS Bookings in my workplace. I am having real trouble matching the calendar to the booking page. No matter how I enter the office hours, assign staff availability, clear the Outlook calendar, the booking slots which are available don’t show correctly on the booking page?
Could anyone help me? I am presenting to the board in a week so don’t have long to figure this out?!
Thank you in advance,
Emma
Hi there,I am trying to launch MS Bookings in my workplace. I am having real trouble matching the calendar to the booking page. No matter how I enter the office hours, assign staff availability, clear the Outlook calendar, the booking slots which are available don’t show correctly on the booking page?Could anyone help me? I am presenting to the board in a week so don’t have long to figure this out?!Thank you in advance,Emma Read More
