Tag Archives: microsoft
Copilot Studio – Questions about Knowledge
Hello!
We’re testing copilot studio and we’re having questions about the knowledge we’re uploading.
We’re using it for employees to ask internal questions.
Is there anyone who is using it and performing well?
Do you have any recommendations on what the file should look like?
Pdf, Word, Excel, with or without topics
I would appreciate it if you could support us
Thank you for your attention
Hello!We’re testing copilot studio and we’re having questions about the knowledge we’re uploading.We’re using it for employees to ask internal questions.Is there anyone who is using it and performing well?Do you have any recommendations on what the file should look like?Pdf, Word, Excel, with or without topicsI would appreciate it if you could support usThank you for your attention Read More
Bell Notifications doesn’t work. Again.
and above link forwards on
But I definitely shall have them.
and above link forwards on
But I definitely shall have them. Read More
Leveraging phi-3 for an Enhanced Semantic Cache in RAG Applications
The field of Generative AI (GenAI) is rapidly evolving, with Large Language Models (LLMs) playing a central role. Building responsive and efficient applications using these models is crucial. Retrieval-Augmented Generation (RAG) applications, which combine retrieval and generation techniques, have emerged as a powerful solution for generating high-quality responses. However, a key challenge arises in handling repeat queries efficiently while maintaining contextually accurate and diverse responses. This blog post explores a solution that addresses this challenge. We propose a multi-layered approach that utilizes a semantic cache layer and phi-3, a Small Language Model (SLM) from Microsoft, to rewrite responses. This approach enhances both performance and user experience.
Demystifying RAG: Retrieval Meets Generation
Retrieval-Augmented Generation (RAG) is a cutting-edge framework that extends the capabilities of natural language generation models by incorporating information retrieval.
Here’s how it works:
User Query: This is the initial input from the user.
App service: Central component that orchestrates the entire RAG workflow, managing user queries, interacting with the cache and search service, and delivering final responses.
Vectorize Query: Leverage OpenAI Embedding models to vectorize the user query into numerical representations. These vectors, similar to fingerprints, allow for efficient comparison and retrieval of relevant information from the vector store and semantic cache.
Semantic Cache Store: This component acts as a storage unit for responses to previously encountered queries, and to check if the current user query aligns with any queries stored in the semantic cache. If a response is found in the cache (cache-hit), the response is fetched and sent to the user.
Vector Store: If no matching query is found in the cache (cache-miss), leverage Azure AI Search service to scour the vast corpus of text to identify relevant documents or snippets based on the user’s query.
Azure OpenAI LLM (GPT 3.5/4/4o): The retrieved documents from AI Search are fed to these LLMs to craft a response, in-context to the user’s query.
Logs: These are used to monitor and analyze system performance.
What is Semantic Cache?
Semantic caching plays a pivotal role in enhancing the efficiency and responsiveness of Retrieval-Augmented Generation (RAG) applications. This section delves into its significance and functionality within the broader architecture:
Understanding Semantic Cache
Storage and Retrieval: The semantic cache acts as a specialized storage unit that stores responses to previously encountered queries. It indexes these responses based on the semantic content of the queries, allowing for efficient retrieval when similar queries are encountered in the future.
Query Matching: When a user query is received, it undergoes vectorization using embedding models to create a numerical representation. This representation is compared against stored queries in the semantic cache. If a match is found (cache-hit), the corresponding response is fetched without the need for additional computation.
Benefits of Semantic Cache:
Speed: Responses retrieved from the semantic cache are delivered almost instantaneously, significantly reducing latency compared to generating responses from scratch.
Resource Efficiency: By reusing pre-computed responses, semantic caching optimizes resource utilization, allowing computational resources to be allocated more effectively.
Consistency: Cached responses ensure consistency in answers to frequently asked questions or similar queries, maintaining a coherent user experience.
Scalability: As the volume of queries increases, semantic caching scales efficiently by storing and retrieving responses based on semantic similarities rather than raw text matching.
Implementing Semantic Cache in RAG
Integration with RAG Workflow: The semantic cache is seamlessly integrated into the RAG workflow, typically managed by the application service. Upon receiving a user query, the application service first checks the semantic cache for a matching response.
Update and Refresh: Regular updates and maintenance of the semantic cache are essential to ensure that responses remain relevant and up to date. This may involve periodic pruning of outdated entries and adding new responses based on recent user interactions.
Performance Monitoring: Monitoring tools track the performance of the semantic cache, providing insights into cache-hit rates, response retrieval times, and overall system efficiency. These metrics guide optimization efforts and ensure continuous improvement.
Challenges in RAG with Semantic Caching
While RAG models are undeniably powerful, they encounter some hurdles:
Repetitive Queries: When users pose similar or identical queries repeatedly, it can lead to redundant processing, resulting in slower response times.
Response Consistency: Ensuring responses maintain contextual accuracy and relevance, especially for similar queries, is crucial.
Computational Burden: Generating responses from scratch for every query can be computationally expensive, impacting resource utilization.
Improving the Semantic Cache with phi-3
To address these challenges, we propose a multi-layered approach built on top of RAG architecture with semantic caching that leverages phi-3, a Small Language Model (SLM) from Microsoft, to dynamically rewrite cached responses retrieved from the semantic cache for similar repeat queries. This ensures responses remain contextually relevant and varied, even when served from the cache.
Major change in the architeture above is addition of phi-3, When a matching query is found in the cache, the retrieved cached response is routed through phi-3. This SLM analyzes the cached response and the current user query, dynamically rewriting the cached response to better suit the nuances of the new query.
By integrating phi-3 into the semantic cache layer, we can achieve the following:
Dynamic Rewriting: When a query matching a cached response is received, phi-3 steps in. It analyses the cached response and the user’s current query, identifying nuances and differences. Subsequently, phi-3 rewrites the cached response to seamlessly incorporate the specific context of the new query while preserving the core meaning. This ensures that even cached responses feel fresh, relevant, and up to date.
Reduced Computational Load: By leveraging phi-3 for rewriting cached responses, we significantly reduce the burden on the larger, computationally expensive LLMs (like GPT-3). This frees up resources for the LLM to handle complex or novel queries that require its full generative power.
Improved Response Diversity: Even for repetitive queries, phi-3 injects variation into the responses through rewriting. This prevents users from encountering identical responses repeatedly, enhancing the overall user experience.
Implementation Considerations
Integrating phi-3 into your RAG application requires careful planning and execution:
Semantic Cache Management: Efficient management of the semantic cache is crucial to ensure quick access to relevant cached responses. Regular updates and pruning of the cache can help maintain its effectiveness.
Fine-Tuning phi-3: Fine-tuning phi-3 to handle specific rewriting tasks can further enhance its performance and ensure it aligns well with the context of your application.
Monitoring and Analytics: Continuous monitoring and analytics can help identify patterns in user queries and optimize the caching strategy. Logs play a crucial role in this aspect, providing insights into the system’s performance and areas for improvement.
Conclusion
The integration of phi-3 into the semantic cache layer of a RAG application represents a significant advancement in handling repeat queries efficiently while maintaining contextually accurate and diverse responses. By leveraging the dynamic rewriting capabilities of phi-3, we can enhance both the performance and user experience of RAG applications.
This multi-layered approach not only addresses the challenges of repetitive queries and computational burden but also ensures that responses remain fresh and relevant, even when served from the cache. As Generative AI continues to evolve, such innovations will play a crucial role in building responsive and efficient applications that can meet the diverse needs of users.
Incorporating these strategies into your RAG application can help you stay ahead in the rapidly evolving field of Generative AI, delivering high-quality and contextually accurate responses that enhance user satisfaction and engagement.
Microsoft Tech Community – Latest Blogs –Read More
Entra ID Connect Download Issue
Good afternoon,
I’ve been trying all day to download the latest version of Entra ID Connect and it seems the download link is failing due to a server error. I’ve tried on different machines on different networks with the same result. Is anyone else seeing this or know the best way to open a ticket to let them know? Or is there another Microsoft mirror I could use?
Sends you to the redirect URL of:
https://go.microsoft.com/fwlink/?LinkId=615771
Which ultimately just sends you to the last URL I’ll post below.
This one, however:
https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/reference-connect-version-history#looking-for-the-latest-versions
Sends you directly to the actual download link of:
https://www.microsoft.com/en-us/download/details.aspx?id=47594
Which results in a server error.
Any help is greatly appreciated. Thank you so much.
Good afternoon, I’ve been trying all day to download the latest version of Entra ID Connect and it seems the download link is failing due to a server error. I’ve tried on different machines on different networks with the same result. Is anyone else seeing this or know the best way to open a ticket to let them know? Or is there another Microsoft mirror I could use? This link:https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/how-to-connect-install-roadmap#install-microsoft-entra-connectSends you to the redirect URL of:https://go.microsoft.com/fwlink/?LinkId=615771Which ultimately just sends you to the last URL I’ll post below.This one, however:https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/reference-connect-version-history#looking-for-the-latest-versionsSends you directly to the actual download link of:https://www.microsoft.com/en-us/download/details.aspx?id=47594Which results in a server error. Any help is greatly appreciated. Thank you so much. Read More
Subforms in Tabs: Can They Be Dragged-and-Dropped In, or Cut-and-Pasted?
Hello, I have a question or two about using tab controls and subforms. I am using Access 2021, and am a novice at database design.
I am designing a movie collection database, and I currently have three subforms. I’d like to have them in tabs, rather than all on one form.
My question: Is it possible to drag and drop the subforms into the tabs, or cut-and-paste them? Or do I have to build the subforms over again, from scratch?
Which method of placing subforms in the tabs do you prefer? Thank you!
Hello, I have a question or two about using tab controls and subforms. I am using Access 2021, and am a novice at database design. I am designing a movie collection database, and I currently have three subforms. I’d like to have them in tabs, rather than all on one form. My question: Is it possible to drag and drop the subforms into the tabs, or cut-and-paste them? Or do I have to build the subforms over again, from scratch? Which method of placing subforms in the tabs do you prefer? Thank you! Read More
Stacked Bar Chart as Office Timeline
I am using Microsoft 365, but I do not have access to Power BI.
I am trying to create a stacked bar chart as a replacement for our firm’s Office Timeline, which currently exists on our outdated, slowly dying intranet site. What I’m looking for is something similar to this:
Projects will be on the Y axis, with a perpetual calendar on the X axis – the color changes represent the end of each project phase, and the black boxes represent milestone dates within the phase.
I have brought in the data, added a perpetual calendar and formatted the dates as numbers. I established a ‘base date’, but I’m not sure that I need it. The resulting stacked bar chart is not even close to what I’m trying to do, and I think the reason is that I need to incorporate a formula to match the ending phase dates to the perpetual calendar (see the data example below).
Do any of you know of a tutorial I could study or a general method to do what I’m trying to accomplish? I have looked at dozens of articles and YouTube videos and I can’t find anything that addresses this.
Thanks,
Traci
I am using Microsoft 365, but I do not have access to Power BI.I am trying to create a stacked bar chart as a replacement for our firm’s Office Timeline, which currently exists on our outdated, slowly dying intranet site. What I’m looking for is something similar to this:Projects will be on the Y axis, with a perpetual calendar on the X axis – the color changes represent the end of each project phase, and the black boxes represent milestone dates within the phase. I have brought in the data, added a perpetual calendar and formatted the dates as numbers. I established a ‘base date’, but I’m not sure that I need it. The resulting stacked bar chart is not even close to what I’m trying to do, and I think the reason is that I need to incorporate a formula to match the ending phase dates to the perpetual calendar (see the data example below). Do any of you know of a tutorial I could study or a general method to do what I’m trying to accomplish? I have looked at dozens of articles and YouTube videos and I can’t find anything that addresses this. Thanks,Traci Read More
Search in Outlook 16.8 does not work on Mac Book Air Ventura 13.2.1
Hi there,
bought the a new mac and the search in Outlook does not work, is not existing. I work in Outlook 16.86 on Mac Book Air Ventura 13.2.1
I de-installed MS and re-installed = followed all steps from MS. Emptied the trash and shut my mac. Re- started it and started again with re-installing. I bought the MS package and thought, I am good. But I am not.
On my old Mac 15.8, all is working in Outlook.
Can you pls advise…. I tried to download the older version of Outlook, but failed here, too.
Thanks, Sabine
Hi there, bought the a new mac and the search in Outlook does not work, is not existing. I work in Outlook 16.86 on Mac Book Air Ventura 13.2.1I de-installed MS and re-installed = followed all steps from MS. Emptied the trash and shut my mac. Re- started it and started again with re-installing. I bought the MS package and thought, I am good. But I am not.On my old Mac 15.8, all is working in Outlook. Can you pls advise…. I tried to download the older version of Outlook, but failed here, too. Thanks, Sabine Read More
Want breaks in line graph
I’m trying to follow this to see gaps in my line graph, but the gaps option is grayed out. What to do?
Display empty cells, null (#N/A) values, and hidden worksheet data in a chart – Microsoft Support
Note: This is fake data, just testing out the line graph formatting.
I’m trying to follow this to see gaps in my line graph, but the gaps option is grayed out. What to do? Display empty cells, null (#N/A) values, and hidden worksheet data in a chart – Microsoft Support Note: This is fake data, just testing out the line graph formatting. Read More
Export all Tenant Team Chats
I have read several postings here and on other sites but its still not clear if it is possible or how to do get a list of all team chats.
Our CEO is asking and not providing much information to why or what the purpose is, what they are looking for. Not sure if they are looking for the group chat titles, I assume 1to1 chats are not required as there could be hundreds/thousands of these.
Some of the reading suggests there may be a way to see;
all the chats a specific person belongs to.eDiscovery might be able to be used but seems to imply this is more about reconstruction of a chat rather than just a simple list. But this would require a Premium license and I am not sure it actually gives a list.
It also not clear how external “Chats” from a meeting would play in to this, it seems a bit more complicated than what our CEO understands.
Can anyone confirm if we can pull a list of Team chat?
I have read several postings here and on other sites but its still not clear if it is possible or how to do get a list of all team chats.Our CEO is asking and not providing much information to why or what the purpose is, what they are looking for. Not sure if they are looking for the group chat titles, I assume 1to1 chats are not required as there could be hundreds/thousands of these. Some of the reading suggests there may be a way to see;all the chats a specific person belongs to.eDiscovery might be able to be used but seems to imply this is more about reconstruction of a chat rather than just a simple list. But this would require a Premium license and I am not sure it actually gives a list.It also not clear how external “Chats” from a meeting would play in to this, it seems a bit more complicated than what our CEO understands. Can anyone confirm if we can pull a list of Team chat? Read More
Tell us about yourself! What do you want to see in the community?
We want to know more about you, to make sure we are providing the right content and bringing in experts that you are looking for. Tell us a bit more about you and what is important. Post in the comments or fill out a short survey for what you would like to see more of in this community!
We want to know more about you, to make sure we are providing the right content and bringing in experts that you are looking for. Tell us a bit more about you and what is important. Post in the comments or fill out a short survey for what you would like to see more of in this community!
Read More
Creating a sharepoint list programatically
Hi,
I have an excel file with 100~ columns which was exported from another SharePoint list (that I don’t have access to). I would like to create a list on my SharePoint site from this file but the simple ‘create list from xlsx’ doesn’t allow me to use many of the column types that I need (lookup, choice etc.).
Initially I tried using the PowerShell connector for SharePoint but my org doesn’t have permissions enabled for it. So I would I like to explore alternative solutions.
The one I have in mind is to acquire an .stp template from somewhere (my org has also disabled built-in templates so I cant seem to download a copy) and modify its JSON definition to add columns with python. Another option is to modify the xml of a list and add the columns that way.
Any help would be appreciated.
Hi, I have an excel file with 100~ columns which was exported from another SharePoint list (that I don’t have access to). I would like to create a list on my SharePoint site from this file but the simple ‘create list from xlsx’ doesn’t allow me to use many of the column types that I need (lookup, choice etc.). Initially I tried using the PowerShell connector for SharePoint but my org doesn’t have permissions enabled for it. So I would I like to explore alternative solutions. The one I have in mind is to acquire an .stp template from somewhere (my org has also disabled built-in templates so I cant seem to download a copy) and modify its JSON definition to add columns with python. Another option is to modify the xml of a list and add the columns that way. Any help would be appreciated. Read More
Introducing the podcast – No Slides Attached: AI Stories with Microsoft Partners
On July 10, we launched an exciting new podcast series crafted for Microsoft partners who are eager to gain insights, learnings, and inspiration from industry leaders and peers. Click here to learn more!
Why tune in?
No Slides Attached features meaningful conversations that combine the expertise of Microsoft leaders and inspiring stories from our partners.
Explore the latest technologies. Season 1 is all about Microsoft Copilot—your everyday AI companion.
Listen to success stories directly from other Microsoft partners.
Join the conversation through our LinkedIn Community.
Take a break
Need a breather from all the emails, meetings, and presentations? Our hosts, Helane Cohen and Fede Pacheco, will look to connect with you on the go.
Subscribe today to listen wherever you get your favorite podcasts:
YouTube* | Spotify | Apple Podcasts | Amazon Music | RSS
*With closed captions in Spanish, Portuguese, and French.
On July 10, we launched an exciting new podcast series crafted for Microsoft partners who are eager to gain insights, learnings, and inspiration from industry leaders and peers. Click here to learn more!
Why tune in?
No Slides Attached features meaningful conversations that combine the expertise of Microsoft leaders and inspiring stories from our partners.
Explore the latest technologies. Season 1 is all about Microsoft Copilot—your everyday AI companion.
Listen to success stories directly from other Microsoft partners.
Join the conversation through our LinkedIn Community.
Take a break
Need a breather from all the emails, meetings, and presentations? Our hosts, Helane Cohen and Fede Pacheco, will look to connect with you on the go.
Subscribe today to listen wherever you get your favorite podcasts:
YouTube* | Spotify | Apple Podcasts | Amazon Music | RSS
*With closed captions in Spanish, Portuguese, and French. Read More
Using GitHub Copilot for Microsoft 365 PowerShell Development
GitHub Copilot costs $10/month. At that price level, anyone working with PowerShell should try out the AI assistance to see if it helps. But will GitHub Copilot work well with Microsoft 365 PowerShell modules like Exchange Online and the Microsoft Graph PowerShell SDK. That’s what we tested and report on here.
https://practical365.com/github-copilot-powershell/
GitHub Copilot costs $10/month. At that price level, anyone working with PowerShell should try out the AI assistance to see if it helps. But will GitHub Copilot work well with Microsoft 365 PowerShell modules like Exchange Online and the Microsoft Graph PowerShell SDK. That’s what we tested and report on here.
https://practical365.com/github-copilot-powershell/ Read More
Creating a Data Package Tier List
Hi, I’m trying to build a model in Excel that displays different data sets. The data sets in Column A (1,2,3 etc.) each have different levels (A,B,C etc.). If I were to choose 1E I would obtain all the other levels (ABCD).
I want to find a way that if the user selects 1A, 4D & 3F it would automatically calculate cost of the highest tier selected for each data set (1,2,3) and also list which ones they did.
Happy to insert columns to the left of each price tag and the user can simply place an X in the cell to identify which data set they are choosing. If I can do this with checkboxes it would be great.
Picture below, apologies for the poor quality.
Hi, I’m trying to build a model in Excel that displays different data sets. The data sets in Column A (1,2,3 etc.) each have different levels (A,B,C etc.). If I were to choose 1E I would obtain all the other levels (ABCD).I want to find a way that if the user selects 1A, 4D & 3F it would automatically calculate cost of the highest tier selected for each data set (1,2,3) and also list which ones they did. Happy to insert columns to the left of each price tag and the user can simply place an X in the cell to identify which data set they are choosing. If I can do this with checkboxes it would be great. Picture below, apologies for the poor quality. Read More
How to stop outlook from archiving with backspace button
Why is Outlook archiving my emails when I hit the backspace button while using the editor functions? How do I turn off that setting? I’m wasting too much time. Switching back to the older outlook.
Why is Outlook archiving my emails when I hit the backspace button while using the editor functions? How do I turn off that setting? I’m wasting too much time. Switching back to the older outlook. Read More
Outlook 2019 Pro will not start
My system rebooted and when it came back up, Outlook won’t start.
I’ve tried starting it in safe mode as well. No change.
The last time it did this it said my license key was already used (DUH!) and I had to contact support…which is a colossal pain in the butt.
Any other methods to get this software back up and running?
System is WIN11 all the latest updates…
My system rebooted and when it came back up, Outlook won’t start.I’ve tried starting it in safe mode as well. No change.The last time it did this it said my license key was already used (DUH!) and I had to contact support…which is a colossal pain in the butt.Any other methods to get this software back up and running?System is WIN11 all the latest updates… Read More
Migrating SLURM Job Accounting from Azure Database for MariaDB to MySQL Flexible Server
Overview
Azure CycleCloud (CC) is an enterprise-friendly tool for orchestrating and managing High-Performance Computing (HPC) environments on Azure. With CycleCloud, users can provision infrastructure for HPC systems, deploy familiar HPC schedulers, deploy/mount filesystems and automatically scale the infrastructure to run jobs efficiently at any scale. Azure CycleCloud is targeted at HPC administrators and users who want to deploy an HPC environment with a specific scheduler in mind.
One of the supported schedulers is SLURM, which stands for Simple Linux Utility for Resource Management. SLURM is an open source, scalable and fault-tolerant cluster management and job scheduling system for Linux clusters of any size. SLURM can allocate resources to users for a duration of time, manage workloads, provide accounting and monitoring, and support parallel and distributed computing applications.
SLURM job accounting is a feature that allows users to collect and report various metrics about the jobs that run on the cluster. SLURM job accounting can help users optimize their resource utilization, monitor their quotas, and track their costs. To enable job accounting, you need to configure and run a SLURM database daemon (slurmdbd) that can store the job information in a database.
One of the common choices for the database backend for SLURM job accounting is Azure Database for MariaDB. However, Azure Database for MariaDB will be retired on September 19, 2025, and users are encouraged to migrate their data and applications to Azure Database for MySQL Flexible Server.
In this blog, we will walk you through the steps to migrate your SLURM job accounting data from Azure Database for MariaDB to Azure Database for MySQL Flexible Server in Azure CycleCloud.
Requirements/Versions
CycleCloud Server (CC version used is 8.6.2)
Slurm Cluster
CycleCloud project used is 3.0.5
Slurm version used is 23.02.6-1
A source instance of Azure Database for MariaDB
A target instance of Azure Database for MySQL Flexible Server
A Linux VM with access to both the MariaDB and MySQL instances
We used the CycleCloud VM
Running on Alma Linux 8.7
Migration Procedure
Install required packages
Back up your SLURM Accounting DB in Azure Database for MariaDB
Restore the backup to your Azure Database for MySQL Flexible Server
Create SLURM user in Azure Database for MySQL and grant privileges
Update your CycleCloud cluster configuration for SLURM Job Accounting
To perform the migration, you will need a VM that can connect to both the source and target databases. In this blog post, we will use the CycleCloud VM as an example, but you can use any VM that meets this requirement.
Step 1: Install required packages
MySQL Shell (mysqlsh) is an advanced command-line client for MySQL that supports various modes of operation, such as SQL, JavaScript, and Python, and enables interactive and batch execution of queries and scripts. This utility will be used to perform the transfer of the SLURM job accounting DB from MariaDB to MySQL, and requires the installation of the following additional packages:
mysql
mysql-shell
To install the required packages, you will need to run the following commands on the migration VM:
sudo yum install -y msql
wget https://dev.mysql.com/get/mysql84-community-release-el8-1.noarch.rpm /tmp
sudo yum localinstall /tmp/mysql84-community-release-el8-1.noarch.rpm
sudo yum install -y mysql-shell
Step 2: Back up your SLURM Accounting DB in Azure Database for MariaDB
At this point, it is recommended for your SLURM cluster to be terminated to ensure that the accounting DB is no longer being updated.
Connect to MariaDB and check the size of the SLURM accounting DB:
mariadbname=jmslurmmariadbeus #update with your specific db name
mariadbusername=themorey #update with your specific db user name
mysqlsh –uri ${mariadbusername}%40${mariadbname}@${mariadbname}.mariadb.database.azure.com:3306
OPTIONAL – to check the size of the DB:
SELECT table_schema AS “Database”,
ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS “Size (MB)”
FROM information_schema.TABLES
GROUP BY table_schema;
Sample Output:
Check free space on VM to determine where backup can be stored:
df -h
The output of the df -h command shows that we have enough space for us to dump our backup on our local disk:
Create a directory for your MariaDB backup. Note, the target directory for the backup must be empty:
sudo mkdir -p /backup/mysql/mariadb_backup/
sudo chown -R $(whoami) /backup
Log back in to MariaDB using mysqlsh:
mariadbname=jmslurmmariadbeus #update with your specific db name
mariadbusername=themorey #update with your specific db user name
mysqlsh –uri ${mariadbusername}%40${mariadbname}@${mariadbname}.mariadb.database.azure.com:3306
Switch to Javascript mode on the mysql shell:
js
Run the dumpUtil to take a full backup:
util.dumpInstance(“/backup/mysqlsh/mariadb_backup”,{threads: 16, showProgress: true, users: false})
Sample Output:
After the dumpUtil command is complete, you can exit out of the MySQL shell ( q; ) and view the backup on your VM by listing the contents of the backup directory. The backup consists of a series of JSON files that contain the schema and data of the instance.
Step 3: Restore the backup to your Azure Database for MySQL Flexible Server
Now that you have a backup of your accounting data from MariaDB, you can restore it to your Azure Database for MySQL Flexible Server.
To do this, first login to your Azure Database for MySQL Flexible Server using MySQL shell. Note, the syntax to establish the connection is slightly different than the one used to connect to MariaDB:
mysqldbname=jmmysqlfrommariadb #update with your specific db name
mysqldbusername=themorey #update with your specific db user name
mysqlsh –uri ${mysqldbusername}@${mariadbname}.mysqldb.database.azure.com:3306
List the databases in your MySQL server to confirm that the slurm_acct_db does not exist:
show databases;
Sample Output:
Switch to javascript mode and run the loadDump utility to import the MariaDB dump files:
js
util.loadDump(“/backup/mysqlsh/mariadb_backup”, {threads: 16, showProgress: true, ignoreVersion: true})
Sample Output:
Step 4: Create SLURM user in Azure Database for MySQL and grant privileges
When we used the util.loadDump() function to restore the data from the MariaDB backup, we only restored the SLURM accounting database and not the user accounts. This means that the SLURM user account that was used to access the database in MariaDB does not exist in the Azure Database for MySQL instance.
To fix this, we need to switch back to SQL mode, create a new user account with the same name and password as the SLURM user in MariaDB, and grant privileges:
sql
create user slurm@’%’;
ALTER USER slurm IDENTIFIED BY ‘P@ssw0rd!@#’;
grant usage on *.* to slurm@’%’;
grant all privileges on slurm_acct_db.* to slurm@’%’;
flush privileges;
SHOW GRANTS FOR ‘slurm’@’%’;
Sample Output:
Step 5: Update your CycleCloud cluster configuration for Slurm Job Accounting
After creating the user and granting the privileges, we are now ready to connect our cluster to the new MySQL server instance. To do this, we need to navigate to the advanced settings of the CycleCloud SLURM cluster and update following details:
Slurm DBD URL = URL for MySQL Server instance
Slurm DBD User = SLURM accounting DB user
Slurm DBD Password = SLURM accounting DB password
SSL Cert URL = https://dl.cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem
Sample Output:
After updating the required details for your SLURM Cluster, save the configuration and start the cluster.
Once the cluster is operational, run “sacct” to verify that you can view the historical information about the jobs that have been ran in the cluster before the migration took place:
NOTE: Slurm accounting command (sacct) defaults to the same day. You may need to expand the search criteria to see jobs older than current day. For example, “sacct -S 060124“ will show jobs starting from 6/1/2024 until the current day.
SUMMARY
The impending retirement date for MariaDB does not require you to abandon your Slurm accounting history. An Azure MySQL Flexible Server can be the solution moving forward while also loading the historical data from MariaDB.
REFERENCES
Migrating from Azure Database for MariaDB to Azure Database for MySQL – Microsoft Community Hub
Microsoft Tech Community – Latest Blogs –Read More
From Paper to Pixels: Azure AI in Historical Document Digitization and Translation
v:* {behavior:url(#default#VML);}
o:* {behavior:url(#default#VML);}
w:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
Heather MacKinnon-Miller
Normal
Heather MacKinnon-Miller
2
5923
2024-07-16T17:12:00Z
2024-07-16T17:12:00Z
5
1127
6427
53
15
7539
16.00
Clean
Clean
false
false
false
false
EN-US
X-NONE
<w:LidThemeComplexscript>X-NONE</w:LidThemeComplexscript>
For 64 years, a stack of letters lay unread in my grandfather’s trunk. These letters, written by relatives of my great-grandfather who immigrated from Poland in 1906, represent the last remnants of Polish language in my family. Over a decade ago, I promised my mother I would have them translated, but life got in the way. Recently, inspired by the success I’ve seen using OCR, digitization, and translation with various customer documents, I decided to tackle this personal project myself using the Microsoft AI services I am familiar with.
Iterating through Process and Technology
I gathered the letters, took images of each page with my phone, and uploaded them to Azure Storage. Sending the image files directly to Azure OpenAI’s GPT-4o model resulted in a confusing mix of English and Polish, so I converted them to .pdf files and took advantage of Azure’s Document Intelligence Service, specifically the Read model, to identify the language and extract the text. The Document Analysis feature can recognize different styles, including handwritten text, and identify the language, reassuring me that the text would be extracted accurately. I used the Document Intelligence code samples repo to get started, and verified that the text was handwritten and correctly identified as Polish (P1). The best workflow involved pre-processing the documents with Azure Document Intelligence’s OCR capabilities, then passing the extracted Polish text to the GPT-4 model for translation into English. I used this git repo as a quick start, changed the model endpoint, and processed the letters in chunks of two pages per call to the model. This combination provided a reliable method to digitize and translate the documents effectively. Success! I had a general idea of the contents of the letters, so once my initial attempt proved successful, I felt motivated to continue. Within about an hour, I had processed all the letters and was ready to validate the results.
Figure 1. Excerpt from an original letter in Polish
The English translation of the above letter excerpt:
“Dear Joseph, I ask you to please ask your brothers Kazimierz and Jan to write to me. I deeply regret the loss of Piotr who died on the front in France. Tell me how your father is doing and if he is healthy. My dearest nephews, please come to Poland to visit us and improve our lives. If you cannot come, please send my family a parcel of clothes. We need both winter and summer clothing, as well as footwear—shoes and clogs. From your abundance, please collect some money and send it to us, poor farmers. I wish you happiness and success in the future. I send my warmest greetings to Joseph, Kazimierz, Jan, your father, and all our relatives.”
Although the translations seemed accurate, I needed confirmation from a native speaker before sharing with family. Luckily, a colleague who speaks Polish fluently offered to review a sample page. He confirmed that, while there were some nuances and minor errors in the translation, the overall message and sentiment were accurate. For example, “cannot replace me, an old man, in hard work” was interpreted by GPT-4o as “too young to replace me in heavy labor.” While this analysis method was sufficient for my small project, more sophisticated metrics would be best used at scale. While this small personal project used only a subset of Azure services and was not intended for public use, this pattern could easily be integrated into a search service, such as Azure AI Search for discovery, or broadened to a “Chat with Your Data” pattern, for scalability.
Extending to cursive handwriting
After completing this project, I became curious about applying the same process to more complex documents, such as those written in cursive. Since cursive has not been taught as part of the common core at most U.S. schools since 2010, there is a real risk that future generations may lose the ability to read and preserve these historical documents. Institutions like the Smithsonian and the National Archives rely on volunteers to transcribe and preserve these records, but the pace is insufficient to capture written history before these skills disappear. The time has never been better to introduce AI for preservation of handwritten history.
To test this idea, I found a few family recipes written in cursive. This time, I uploaded image files of the recipes directly to GPT-4o. The results were impressive and did not require OCR as an initial step. Here’s a sample recipe from my grandmother:
Figure 2. Recipe handwritten in cursive
And here are the results from GPT-4o:
Sandies
Ingredients:
1 cup butter
1/3 cup granulated sugar
2 tablespoons water
2 teaspoons vanilla
2 cups flour
1 cup chopped nuts
Instructions:
Cream butter and sugar.
Add water and vanilla.
Then add flour and nuts.
Chill 4 hours.
Roll in cookie sheet, shape into fingers.
Bake at 325°F for 20 minutes.
Cool and roll in powdered sugar.
Amazingly, the model was even able to identify “cookie sheet,” which was written in very small superscript text.
Prompt nuances matter
While I didn’t have to do much on the prompt engineering side, I realized how small changes in prompts can affect the quality of the result. In the case of the Polish letters, I initially prompted the GPT model to “translate this text from Polish to English”. I iterated a bit and found “translate these family letters from Polish to English” made the result a bit more readable and self-corrected on some misspellings. In the case of the recipes, I specifically prompted the GPT model to “read this family recipe” rather than just read what was in the image. This resulted in not only a very accurate result, but the model output separated the ingredients and the instructions without being explicitly written in the original recipe.
The journey from paper to pixels has never been more accessible or efficient, thanks to Azure Document Intelligence and Azure OpenAI. These powerful tools have proven their capability in digitizing and translating handwritten historical documents, preserving invaluable cultural and personal histories. My experience with translating my great-grandfather’s letters and digitizing family recipes demonstrates the transformative potential of these technologies. By leveraging Azure’s AI tools, we can ensure that the stories and knowledge contained in historical documents are not lost to time but are instead accessible to future generations.
Microsoft Tech Community – Latest Blogs –Read More
ACTION REQUIRED: Users of HDC API may need to update code for deprecated API by August 28th
On August 28th, 2024, Hardware Dev Center APIs will change the type of blob urls that are returned in various API calls from service sas urls to user delegated sas urls. Partners using HDC Hardware dashboard API to automate driver submission process may be affected. The specific area affected is the upload and downloads through the download links in the submission response. All other aspects of the submission process that can be accessed through the API are not affected. Partners are recommended to update to latest version of the libraries they use to upload and download from Azure Blobs during the driver submission process.
Partners using C# are encouraged to move to Azure.Storage.Blobs library to ensure that driver submissions are not interrupted. Any partner already managing blob interactions through Azure.Storage.Blobs library are not affected by this change. Please reference this documentation that covers how to upload to a blob with .NET: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-upload.
If you experience any issues, please open a support ticket with HDC support. For details about how to contact the support team, visit https://aka.ms/GetHardwareSupport .
Microsoft Tech Community – Latest Blogs –Read More
Microsoft Power BI and Microsoft Defender for Cloud
Introduction
As cloud environments grow more complex and threats increase, organizations need robust tools to monitor, analyze, and respond to security issues effectively. Microsoft Defender for Cloud (MDC) offers robust security management, but to unlock its full potential, organizations need powerful visualization and analysis tools.
While Azure Workbooks provide valuable visualizations for MDC data, integrating Microsoft Power BI offers an enhanced approach to data analysis and visualization. Power BI’s advanced features, such as customizable dashboards, interactive elements, and seamless integration with various data sources, make it ideal for enhancing the value derived from MDC data.
This article is the first in a series of correlated blogs that will explore scenarios and applicability in depth. As an introduction to the series, this article provides the foundation on how to start leveraging Power BI to report and dashboard MDC insights.
Benefits of Using Power BI with Microsoft Defender for Cloud
Advanced Data Visualization: Power BI provides a wide array of visualization options, allowing security teams to create highly customized and visually rich dashboards that effectively communicate insights to different stakeholders.
Enhanced Data Analysis: Power BI’s robust analytical tools, including DAX (Data Analysis Expressions) and built-in AI capabilities, enable security teams to perform complex data analysis and uncover deeper insights.
Seamless Integration: Power BI integrates with various data sources, including Azure Resource Graph, allowing you to consolidate data from multiple platforms into a single, unified view.
Collaborative Features: Power BI facilitates collaboration by enabling teams to share dashboards and reports easily, with role-based access controls ensuring data security.
Ease of Use: Power BI’s intuitive drag-and-drop functionality makes it simple for users to create and customize visualizations without extensive technical knowledge, making it accessible to users of all skill levels.
Step-by-Step Guide to Integrating MDC Data into Power BI
To integrate MDC data into Power BI, follow these steps:
Step 1: Set Up Power BI and Azure Resource Graph
Install Power BI Desktop: Download Power BI Desktop.
Enable Azure Resource Graph: Ensure that you have the necessary permissions to access Azure Resource Graph.
Step 2: Connect Power BI to Azure Resource Graph
Open Power BI Desktop: Launch Power BI Desktop on your computer.
Get Data: Click on Get Data on the Home tab.
Select Azure Resource Graph: In the Get Data window, search for Azure Resource Graph and select it.
Connect: Click Connect and sign in with your Azure credentials.
Step 3: Load MDC Data into Power BI
Once you’ve connected Power BI to Azure Resource Graph, you can begin loading MDC data.
Here, we’ll provide a few example queries to retrieve data for recommendations, attack paths, secure scores, and governance. Note that these are just a few examples; you can retrieve any data available in Azure Resource Graph (ARG) according to your needs.
Enter ARG Queries: Write or paste the ARG KQL query and click OK
Load Data: After entering the queries, click Load to import the data into Power BI. The imported data will appear in the Fields pane, ready for you to create visualizations and reports.
Use the following ARG queries to pull the main MDC data points:
Recommendations (by risk):
This query retrieves security recommendations by risk from MDC, allowing you to analyze assessments and identify areas that need attention.
securityresources
| where type =~ “microsoft.security/assessments”
| extend assessmentType = iff(type == “microsoft.security/assessments”, tostring(properties.metadata.assessmentType), dynamic(null))
| where (type == “microsoft.security/assessments” and (assessmentType in~ (“BuiltIn”, “CustomerManaged”)))
| extend assessmentTypeSkimmed = iff(type == “microsoft.security/assessments”, case(
tostring(properties.metadata.assessmentType) == “BuiltIn”, “BuiltIn”,
tostring(properties.metadata.assessmentType) == “BuiltInPolicy”, “BuiltIn”,
tostring(properties.metadata.assessmentType) == “CustomPolicy”, “Custom”,
tostring(properties.metadata.assessmentType) == “CustomerManaged”, “Custom”,
tostring(properties.metadata.assessmentType) == “ManualCustomPolicy”, “Custom”,
tostring(properties.metadata.assessmentType) == “ManualBuiltInPolicy”, “BuiltIn”,
dynamic(null)
), dynamic(null))
| extend assessmentId = tolower(id)
| extend assessmentKey = iff(type == “microsoft.security/assessments”, name, dynamic(null))
| extend source = iff(type == “microsoft.security/assessments”, trim(‘ ‘, tolower(tostring(properties.resourceDetails.Source))), dynamic(null))
| extend statusCode = iff(type == “microsoft.security/assessments”, tostring(properties.status.code), dynamic(null))
| extend resourceId = iff(type == “microsoft.security/assessments”, trim(” “, tolower(tostring(case(source =~ “azure”, properties.resourceDetails.Id,
(type == “microsoft.security/assessments” and (source =~ “aws” and isnotempty(tostring(properties.resourceDetails.ConnectorId)))), properties.resourceDetails.Id,
(type == “microsoft.security/assessments” and (source =~ “gcp” and isnotempty(tostring(properties.resourceDetails.ConnectorId)))), properties.resourceDetails.Id,
source =~ “aws”, properties.resourceDetails.AzureResourceId,
source =~ “gcp”, properties.resourceDetails.AzureResourceId,
extract(“^(?i)(.+)/providers/Microsoft.Security/assessments/.+$”,1,id)
)))), dynamic(null))
| extend resourceName = iff(type == “microsoft.security/assessments”, tostring(coalesce(properties.resourceDetails.ResourceName, properties.additionalData.CloudNativeResourceName, properties.additionalData.ResourceName, properties.additionalData.resourceName, split(resourceId, ‘/’)[-1], extract(@”(.+)/(.+)”, 2, resourceId))), dynamic(null))
| extend resourceType = iff(type == “microsoft.security/assessments”, tolower(properties.resourceDetails.ResourceType), dynamic(null))
| extend riskLevelText = iff(type == “microsoft.security/assessments”, tostring(properties.risk.level), dynamic(null))
| extend riskLevel = iff(type == “microsoft.security/assessments”, case(riskLevelText =~ “Critical”, 4,
riskLevelText =~ “High”, 3,
riskLevelText =~ “Medium”, 2,
riskLevelText =~ “Low”, 1,
0), dynamic(null))
| extend riskFactors = iff(type == “microsoft.security/assessments”, iff(isnull(properties.risk.riskFactors), dynamic([]), properties.risk.riskFactors), dynamic(null))
| extend attackPaths = array_length(iff(type == “microsoft.security/assessments”, iff(isnull(properties.risk.attackPathsReferences), dynamic([]), properties.risk.attackPathsReferences), dynamic(null)))
| extend displayName = iff(type == “microsoft.security/assessments”, tostring(properties.displayName), dynamic(null))
| extend statusCause = iff(type == “microsoft.security/assessments”, tostring(properties.status.cause), dynamic(null))
| extend isExempt = iff(type == “microsoft.security/assessments”, iff(statusCause == “Exempt”, tobool(1), tobool(0)), dynamic(null))
| extend statusChangeDate = tostring(iff(type == “microsoft.security/assessments”, todatetime(properties.status.statusChangeDate), dynamic(null)))
| project assessmentId,
statusChangeDate,
isExempt,
riskLevel,
riskFactors,
attackPaths,
statusCode,
displayName,
resourceId,
assessmentKey,
resourceType,
resourceName,
assessmentTypeSkimmed
| join kind=leftouter (
securityresources
| where type == ‘microsoft.security/assessments/governanceassignments’
| extend assignedResourceId = tolower(iff(type == “microsoft.security/assessments/governanceassignments”, tostring(properties.assignedResourceId), dynamic(null)))
| extend dueDate = iff(type == “microsoft.security/assessments/governanceassignments”, todatetime(properties.remediationDueDate), dynamic(null))
| extend owner = iff(type == “microsoft.security/assessments/governanceassignments”, iff(isempty(tostring(properties.owner)), “unspecified”, tostring(properties.owner)), dynamic(null))
| extend governanceStatus = iff(type == “microsoft.security/assessments/governanceassignments”, case(
isnull(todatetime(properties.remediationDueDate)), “NoDueDate”,
todatetime(properties.remediationDueDate) >= bin(now(), 1d), “OnTime”,
“Overdue”
), dynamic(null))
| project assignedResourceId, dueDate, owner, governanceStatus
) on $left.assessmentId == $right.assignedResourceId
| extend completionStatusNumber = case(governanceStatus == “Overdue”, 5,
governanceStatus == “OnTime”, 4,
statusCode == “Unhealthy”, 3,
isExempt, 7,
1)
| extend completionStatus = case(completionStatusNumber == 5, “Overdue”,
completionStatusNumber == 4, “OnTime”,
completionStatusNumber == 3, “Unassigned”,
completionStatusNumber == 7, “Exempted”,
“Completed”)
| where completionStatus in~ (“OnTime”,”Overdue”,”Unassigned”)
| project-away assignedResourceId, governanceStatus, isExempt
| order by riskLevel desc, attackPaths desc, displayName
Attack Paths:
Use this query to fetch attack path data, providing insights into potential attack vectors within your cloud environment.
securityresources
| where type == “microsoft.security/attackpaths”
| extend riskCategories = tostring(properties.riskCategories)
| extend riskCategories = tostring(split(riskCategories, “[“)[1])
| extend riskCategories = tostring(split(riskCategories, “]”)[0])
| extend riskCategory = iff(‘{riskCategories}’ == “All”, riskCategories, ‘{riskCategories}’)
| where riskCategories has(riskCategory)
| project apId = name, apTemplate = tostring(properties.displayName), riskCategories
| summarize Path_Count = count() by Attack_Path = apTemplate, riskCategories
| project Attack_Path, Path_Count, riskCategories
Secure Score:
This query retrieves secure score data, helping you understand your overall security posture and prioritize remediation efforts.
securityresources
| where type == “microsoft.security/securescores”
| where name == “ascScore”
| extend environment = tostring(properties.environment)
| extend scopeMaxScore = toint(properties.score.max)
| extend scopeWeight = toint(properties.weight)
| extend scopeScorePerc = round(todouble(properties.score.percentage), 0)
Governance:
Use this query to get data on governance rules, enabling you to manage compliance and governance policies effectively.
securityresources
| where type == “microsoft.security/assessments”
| where isnull(properties.resourceDetails.AwsResourceId) and isnull(properties.resourceDetails.GcpResourceId)
| extend DisplayName = tostring(properties.displayName)
| where isempty(DisplayName) == false
| join kind=leftouter (securityresources
| where type == “microsoft.security/assessments/governanceassignments”
| extend assignedResourceId = tostring(todynamic(properties).assignedResourceId)
| extend remediationDueDate = todatetime(properties.remediationDueDate)
| project id = assignedResourceId, governanceassignmentsProperties = todynamic(properties), remediationDueDate) on id
| extend hasAssignment = isempty( governanceassignmentsProperties) == false and isnull( governanceassignmentsProperties) == false
| extend assignmentStatus = iif(tostring(properties.status.code) == “Unhealthy”,iif(hasAssignment == true, iif(bin(remediationDueDate, 1d) < bin(now(), 1d), “Overdue”, “Ontime”), “Unassigned”) , “Completed”)
| summarize count() by assignmentStatus
Compliance:
This query retrieves compliance data from MDC, which is essential for maintaining and demonstrating adherence to various regulatory requirements.
securityresources
| where type == “microsoft.security/regulatorycompliancestandards/regulatorycompliancecontrols/regulatorycomplianceassessments” | extend scope = properties.scope
| where isempty(scope) or scope in~(“Subscription”, “MultiCloudAggregation”)
| parse id with * “regulatoryComplianceStandards/” complianceStandardId “/regulatoryComplianceControls/” complianceControlId “/regulatoryComplianceAssessments” *
| extend complianceStandardId = replace( “-“, ” “, complianceStandardId)
| extend Status = properties.state
Remember, the queries provided above are just examples. ARG allows you to query a wide range of data, so feel free to customize and create queries that suit your specific requirements. With ARG, you have the flexibility to retrieve and analyze any data available within your MDC environment, ensuring comprehensive and tailored insights.
Step 4: Create Visualizations in Power BI
Select Visualization Type: Choose from various visualizations such as charts, graphs, and maps to represent your data.
Customize Visualizations: Use the drag-and-drop functionality to customize your visualizations.
Create Dashboards: Arrange your visualizations into dashboards to provide a comprehensive view of your security data.
Perhaps you can build a report similar to the one shown in the picture below.
If you prefer, you can also use a predefined sample report available for download from the Defender for Cloud GitHub.
This sample report provides a great starting point and can be customized further to meet your specific needs, ensuring you get the most out of your MDC data.
Step 5: Share and Collaborate
Publish Reports: Publish your reports to the Power BI service to share with your team.
Set Permissions: Use role-based access controls to manage who can view or edit the reports.
Conclusion
By leveraging Power BI’s advanced features alongside Azure Workbooks, organizations can unlock deeper insights, create more customized and interactive reports, and improve collaboration across teams. This approach provides a more comprehensive and flexible solution for visualizing and analyzing MDC data, enhancing security posture management and decision-making.
Microsoft Defender for Cloud Additional Resources
Watch a demonstration on how to use Governance Rule in this episode of Defender for Coud in the Field
Download the new Microsoft CNAPP eBook at aka.ms/MSCNAPP
Become a Defender for Cloud Ninja by taking the assessment at aka.ms/MDCNinja
Reviewers
Yuri Diogenes, Principal PM Manager, CxE Defender for Cloud
Tal Rosler, Senior PM lead, Microsoft Defender for Cloud
Microsoft Tech Community – Latest Blogs –Read More
