Introduction

In this article I will guide you on how to build a Generative AI application in Microsoft Fabric.
This guide will walk you through implementing a RAG (Retrieval Augmented Generation) system in Microsoft Fabric using Azure OpenAI and Microsoft Fabric Eventhouse as your vector store.

Why MS Fabric Eventhouse?

Fabric Eventhouse is built using the Kusto Engine that delivers top-notch performance for similarity search at high scale.

If you are looking to build a RAG application with a large number of embeddings vectors, look no more, using MS Fabric you can leverage the processing power for building the Vector Database and the high performant engine powering Fabric Eventhouse DB.

If you want to know more about using Fabric Eventhouse as a Vector store here are some links.

Azure Data Explorer for Vector Similarity Search 

Optimizing Vector Similarity Search on Azure Data Explorer – Performance Update 

Optimizing Vector Similarity Searches at Scale 

What is RAG – Retrieval Augmented Generation?

Large Language Models (LLMs) excel in creating text that resembles human writing.
Initially, LLMs are equipped with a broad spectrum of knowledge from extensive datasets used for their training. This grants them flexibility but may not provide the specialized focus or knowledge necessary in certain topics.

Retrieval Augmented Generation (RAG) is a technique that improves the pertinence and precision of LLMs by incorporating real-time, relevant information into their responses. With RAG, an LLM is boosted by a search system that sifts through unstructured text to find information, which then refines the LLM’s replies.

What is a Vector Database?

The Vector Database is a vital component in the retrieval process in RAG, facilitating the quick and effective identification of relevant text sections in response to a query, based on how closely they match the search terms.

Vector DBs are data stores optimized for storing and processing vector data. Vector data can refer to data types such as geometric shapes, spatial data, or more abstract high-dimensional data used in machine learning applications, such as embeddings.

These databases are designed to efficiently handle operations such as similarity search, nearest neighbour search, and other operations that are common when dealing with high-dimensional vector spaces.

For example, in machine learning, it’s common to convert text, images, or other complex data into high-dimensional vectors using models like word embeddings, image embeddings, etc. To efficiently search and compare these vectors, a vector database or vector store with specialized indexing and search algorithms would be used.

In our case we will use Azure OpenAI Ada Embeddings model to create embeddings, which are vector representations of the text we are indexing and storing in Microsoft Fabric Eventhouse DB.

The code

The code can be found here.

We will use the Moby Dick book from the Gutenberg project in PDF format as our knowledge base.

We will read the PDF file, cut the text into chunks of 1000 characters and calculate the embeddings for each chunk, then we will store the text and the embeddings in our Vector Database (Fabric Eventhouse)

We will then ask questions and get answers from our Vector DB and send the question and answers to Azure OpenAI GPT4 to get a response in natural language.

Processing the files and indexing the embeddings

We will do this once – only to create the embeddings and then save them into our Vector Database – Fabric Eventhouse

Read files from Fabric Lakehouse
Create embeddings from the text using Azure OpenAI ada Embeddings model
Save the text and embeddings in our Fabric Eventhouse DB

RAG – Getting answers

Every time we want to search for answers from our knowledge base, we will:

Create the embeddings for the question and search our Fabric Eventhouse for the answers, using Similarity search
Combining the question and the retrieved answers from our Vector Database, we will call Azure OpenAI GPT4 model to get “natural language” answer.

Prerequisites

To follow this guide, you will need to ensure that you have access to the following services and have the necessary credentials and keys set up.

Microsoft Fabric.
Azure OpenAI Studio to manage and deploy OpenAI models.

Setup

Create a Fabric Workspace

Create a Lakehouse

Upload the moby dick pdf file

Create an Eventhouse DB called “GenAI_eventhouse”

Click on the DB name and then “Explore your data” on the top-right side

Create the “bookEmbeddings” table

Paste the following command and run it

.create table bookEmbeddings (document_name:string, content:string, embedding:dynamic)

Import our notebook

Grab your Azure openAI endpoint and secret key and paste it in the notebook, replace your models deployment names if needed.

Get the Eventhouse URI and paste it as “KUSTO_URI” in the notebook

Connect the notebook to the Lakehouse

Let’s run our notebook

This will install all the python libraries we need

%pip install openai==1.12.0 azure-kusto-data langchain tenacity langchain-openai pypdf

Run cell 2 after configuring the environment variables for:

OPENAI_GPT4_DEPLOYMENT_NAME=”gpt-4″
OPENAI_DEPLOYMENT_ENDPOINT=”<your-azure openai endpoint>”
OPENAI_API_KEY=”<your-azure openai api key>”
OPENAI_ADA_EMBEDDING_DEPLOYMENT_NAME = “text-embedding-ada-002”
KUSTO_URI = “<your-eventhouse cluster-uri>”

Run cell 3

Here we create an Azure OpenAI client and define a function to calculate embeddings

client = AzureOpenAI(
azure_endpoint=OPENAI_DEPLOYMENT_ENDPOINT,
api_key=OPENAI_API_KEY,
api_version=”2023-09-01-preview”
)

#we use the tenacity library to create delays and retries when calling openAI embeddings to avoid hitting throttling limits
@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6))

def generate_embeddings(text):
# replace newlines, which can negatively affect performance.
txt = text.replace(“n”, ” “)
return client.embeddings.create(input = [txt], model=OPENAI_ADA_EMBEDDING_DEPLOYMENT_NAME).data[0].embedding

Run cell 4

Read the file, divide it into 1000 chars chunks

# splitting into 1000 char long chunks with 30 char overlap
# split [“nn”, “n”, ” “, “”]
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=30,
)

documentName = “moby dick book”
#Copy File API path
fileName = “/lakehouse/default/Files/moby dick.pdf”
loader = PyPDFLoader(fileName)
pages = loader.load_and_split(text_splitter=splitter)
print(“Number of pages: “, len(pages))

Run cell 5

Save the text chunks to a pandas dataframe

#save all the pages into a pandas dataframe
import pandas as pd
df = pd.DataFrame(columns=[‘document_name’, ‘content’, ’embedding’])
for page in pages:
df.loc[len(df.index)] = [documentName, page.page_content, “”]
df.head()

Run cell 6

Calculate embeddings

# calculate the embeddings using openAI ada
df[“embedding”] = df.content.apply(lambda x: generate_embeddings(x))
print(df.head(2))

Run cell 7

Write the data to MS Fabric Eventhouse

df_sp = spark.createDataFrame(df)
df_sp.write.
format(“com.microsoft.kusto.spark.synapse.datasource”).
option(“kustoCluster”,KUSTO_URI).
option(“kustoDatabase”,KUSTO_DATABASE).
option(“kustoTable”, KUSTO_TABLE).
option(“accessToken”, accessToken ).
mode(“Append”).save()

Let’s check the data was saved to our Vector Database

Go to the Eventhouse and run this query

bookEmbeddings
| take 10

Go back to the notebook and run the rest of the cells

Creates a function to call GPT4 for a NL answer

def call_openAI(text):
response = client.chat.completions.create(
model=OPENAI_GPT4_DEPLOYMENT_NAME,
messages = text,
temperature=0
)
return response.choices[0].message.content

Creates a function to retrieve answers using embeddings with similarity search

def get_answer_from_eventhouse(question, nr_of_answers=1):
searchedEmbedding = generate_embeddings(question)
kusto_query = KUSTO_TABLE + ” | extend similarity = series_cosine_similarity(dynamic(“+str(searchedEmbedding)+”), embedding) | top ” + str(nr_of_answers) + ” by similarity desc “
kustoDf = spark.read
.format(“com.microsoft.kusto.spark.synapse.datasource”)
.option(“kustoCluster”,KUSTO_URI)
.option(“kustoDatabase”,KUSTO_DATABASE)
.option(“accessToken”, accessToken)
.option(“kustoQuery”, kusto_query).load()
return kustoDf

Retrieves 2 answers from Eventhouse

nr_of_answers = 2
question = “Why does the coffin prepared for Queequeg become Ishmael’s life buoy once the Pequod sinks?”
answers_df = get_answer_from_eventhouse(question, nr_of_answers)

Concatenates the answers

answer = “”
for row in answers_df.rdd.toLocalIterator():
answer = answer + ” ” + row[‘content’]

Creates a prompt for GPT4 with the question and the 2 answers

prompt = ‘Question: {}’.format(question) + ‘n’ + ‘Information: {}’.format(answer)
# prepare prompt
messages = [{“role”: “system”, “content”: “You are a HELPFUL assistant answering users questions. Answer the question using the provided information and do not add anything else.”},
{“role”: “user”, “content”: prompt}]
result = call_openAI(messages)
display(result)

That’s it, you have built your very first RAG app using MS Fabric

All the code can be found here.

Thanks

Denise

​Microsoft Tech Community – Latest Blogs –Read More 

As we saw in previous article, the binary drift alert gives you information about where the activity happened like the object namespace, image, cluster, etc.

This might or might not be enough information for you to act. Say, if you want to identify “how” this drift came to be for example, did a user logged on to container and downloaded the said binary. To supplement the information provided by the alert we can then use Defender XDR portal (https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-defender-portal)

Harnessing the power of the Microsoft Ecosystem

If you are an E5 customer, your security teams most likely are very familiar with Advance Hunting on security.microsoft.com portal. Here we will extend that hunting capability to add context to your Kubernetes alerts. This is a huge time saver and cost advantage for you as you don’t need to teach your Red Team or SOC analysts Level 400 Kubernetes concepts. To jump start your Kubernetes hunting, you can leverage the developer knowledge of your Platform teams to provide most common Kubernetes actions like exec (access to a container), debug (access to node). The hunting team can then leverage these in the hunting queries in a data structure and format they already know using KQL.

Enhancing the hunt using XDR portal

Defender for Cloud sends incidents and alerts to the Defender portal (https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-security-center-defender-cloud#investigation-experience-in-the-microsoft-defender-portal).

Similarly, Microsoft Sentinel also send the data to Defender portal (https://learn.microsoft.com/en-us/azure/sentinel/microsoft-365-defender-sentinel-integration?toc=%2Fdefender-xdr%2Ftoc.json&bc=%2Fdefender-xdr%2Fbreadcrumb%2Ftoc.json&tabs=defender-portal)

Since your SOC and Red Teams are already proficient in using XDR portal, Kubernetes hunts can now easily become part of their playbook.

By looking at the Alerts and Incidents in the XDR portal you can see the birds eye view of what, who, and where. This will equip you to further narrow down your search in Hunting Queries.

                   Fig. Attack Story for Binary Drift

The Evidence and Response tab shows all the relevant evidence. Most likely, you will create your hunting queries using these fields.

                Fig. Kubernetes objects related to the incident

This integration allows us to run Advance Hunting queries using CloudAuditEvents table that has Defender for Cloud data.

The query below looks for exec in a Pod named ubuntu (assumption here being that this pod is also running a container ubuntu where the drift happened and alert generated)

CloudAuditEvents
| where DataSource == “Azure Kubernetes Service”
| where OperationName == “create”
| where RawEventData.ObjectRef.resource == “pods” and RawEventData.ResponseStatus.code == 101
| where RawEventData.ObjectRef.subresource == “exec”
| where RawEventData.ResponseStatus.code == 101
| extend RequestURI = tostring(RawEventData.RequestURI)
| extend PodName = tostring(RawEventData.ObjectRef.name)
| extend PodNamespace = tostring(RawEventData.ObjectRef.namespace)
| extend Username = tostring(RawEventData.User.username)
| where PodName startswith “ubuntu”
| extend Commands = extract_all(@”command=([^&]*)”, RequestURI)
| extend ParsedCommand = url_decode(strcat_array(Commands, ” “))
| project Timestamp, AzureResourceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, Username, ParsedCommand

The query above will produce the following, here you can see who executed the command, when was the command executed and many other fields that are helpful for the context

                          Fig. Query output

Another scenario you would want to look for are activities performed by a managed identity that might have been related to an alert.

CloudAuditEvents
| where RawEventData.principaloid == <managed identity id>

This query will provide you the actions that this managed identity performed. These might be enumerating key vaults, storage accounts, etc. As before by reviewing the output such as timestamp, geo etc. you can make a determination if this action is malicious.

You can then look at the activities that were performed during this timeframe, like so,

CloudAuditEvents
| where AzureResourceId == <AKS Cluster ID>
| where TimeGenerated > datetime(<start time>) and TimeGenerated < datetime(<end time>)
| where OperationName == “create”
| where UserAgent has “kubectl”

If the attacked conducted any “exec” during this time frame you will be able to see it under “ObjectRef” and “RequestURI” will show you the exact command that was executed.

Important to note that Query results are presented in your local time zone as per settings. Kusto filters, however, work in UTC.

Automating the process and Extending the out of the box detections

Now that you have queries defined that provide you additional context behind the drifts, you can convert them into a custom detection rule that you can run at a defined frequency.

For example, say you want to get alerted on privileged pods:

These pods run with an elevated set of privileges required to do their job, but that could conceivably be used as a jumping off point to gain escalated privileges.

Note: You can also use Azure Policy built-in definitions to prevent this:

https://portal.azure.com/#blade/Microsoft_Azure_Policy/PolicyDetailBlade/definitionId/%2Fproviders%2FMicrosoft.Authorization%2FpolicyDefinitions%2F95edb821-ddaf-4404-9732-666045e056b4

CloudAuditEvents
| where Timestamp > ago(1d)
| where DataSource == “Azure Kubernetes Service”
| where OperationName == “create”
| where RawEventData.ObjectRef.resource == “pods” and isnull(RawEventData.ObjectRef.subresource)
| where RawEventData.ResponseStatus.code startswith “20”
| extend PodName = RawEventData.RequestObject.metadata.name
| extend PodNamespace = RawEventData.ObjectRef.namespace
| mv-expand Container = RawEventData.RequestObject.spec.containers
| extend ContainerName = Container.name
| where Container.securityContext.privileged == “true”
| extend Username = RawEventData.User.username
| extend DeviceId = RawEventData.AzureResourceId
| project Timestamp, ReportId, DeviceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, ContainerName, Username

Note that we will be using “Timestamp, ReportId, DeviceId” to turn the query into a custom detection rule. This again can be a starter pattern for your rules.

To create a custom detection/rule you will need to include the columns mentioned under https://learn.microsoft.com/en-us/defender-xdr/custom-detection-rules#required-columns-in-the-query-results

You can now easily convert this query into a custom detection rule

            Fig. Creating custom detection rule

Complete the fields that will show up in Alerts

               Fig. Privileged pod alert details

Select the impacted object in our case the DeviceId (aka AzureResourceId)

             Fig. Identity impacted by the alert

Choose the action

                 Fig. Action to be taken upon alert

Lastly, create your alert

                               Fig. Alert created

Once created the rule will appear in your custom detection rules in XDR portal

                Fig. Custom detection rules

Call to action

As you saw in this two-part series that unlike many third party solutions, you can maximize your existing investment in Microsoft’s security ecosystem to conduct deeper hunts using the tools that you already know.

Our suggestion is to:

Engage your Platform Engineering team to identify most risky Kubernetes actions that are relevant to your environment. For example, if you are using distroless images exec won’t make much sense
Provide your SOC and Red Team the XDR Portal and Defender for Cloud integration documentation so they can start leveraging the starter queries and customize to your environment
Review your third-party Kubernetes security solutions to see if the value you are getting for features that are not available in Native is worth the investment. In our discussions with dozens of customers, Native proves to be most direct, cost efficient, easier to use, and manage in the longer term
Moreover, if you have not thought about going Native-First in your security strategy reevaluate the choice and talk to your Security Seller about the latest roadmap of Native Security Services  
Remember, a lot of on-premises security challenges were because of reliance on “best of breed”. That strategy does not translate well in Cloud. Share the following Native First security resources with your decision makers

https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/native-first-cloud-security-approach/ba-p/4102367
https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/unleashing-the-power-of-microsoft-defender-for-cloud-unique/ba-p/4102392

​Microsoft Tech Community – Latest Blogs –Read More