Building Custom Co-Pilot (PoC)

Azure OpenAI is a cloud service that allows developers to access the powerful capabilities of OpenAI, while enjoying the benefits of security, governance, and more of Azure. When moving from the first initial ideation phase, and starting to move towards a Proof of Concept(PoC)/ Proof of Value(PoV)/ Proof of Technology(PoT), there are a number of considerations that need to be made to ensure this phase is of success.

One of the most common applications of a PoC on Azure OpenAI is to build a custom co-pilot, a tool that can assist both internal employee’s and external users with a wide range of activities, such as summarization, content generation, or more technical such as, suggesting relevant code snippets, completing code blocks; or explaining code logic. This “co-pilot” approach is a tried and tested approach at all levels of maturity across enterprises, as it is a very low-hanging fruit, but a fruit that offers real benefits to those who are both developing and using the application at a PoC phase.

Given the wide scope of technologies that can encompass this phase, I have divided up and placed everything into four defined approaches; each with subsequent pro’s and con’s. To give a quick summary, essentially each path takes a level of code/no-code/low-code, or a combination of all; depending on the level of customization outside of the degrees of black-box that are found within each approach.

That is not to say that one approach is better than another one, but instead, that for a PoC, given simplicity is appreciated, one could look at the greatest abstractions (such as no-code), as the first option to test, given limited time-sink albeit at the cost of complexity, and work through the approaches one-by-one to find the perfect level of trade-offs that are acceptable. With a number of primary aims in a PoC, being to generate excitement between both business and technology, to prove hypothesises of technology and value, and drive towards the next phase; it is really important to be able to iterate quickly and decipher where the current iteration is succeeding or failing, which is where a level of complexity, found in the low/code first approaches provide more value; but again, at more of a time sink.

Let’s talk through some of the approaches

Code-First:

With the inclusion of various packages, such as Microsoft’s Semantic Kernel, or LangChain allow for the orchestration of Azure OpenAI and other microservices to create a copilot solution. This allows for the greatest level of complexity, through code, alias the greatest amount of time to set-up and run.

Usually, these frameworks would sit either in the backend of the code, or run as an orchestrator through some level of abstraction/serverless compute offering, such as a function app.

This deployment can be seen as robust and future-proofing but could be overcomplicating at an earlier stage than required. The newly launched Azure AI Studio is a trusted platform that enables users to create generative AI applications and custom Copilot experiences. Typical PoC’s at this stage implore typical use-cases, such as “Chat with Data”, or RAG (Retrieval Augmented Generation) patterns; which, given their tried and tested nature, can be comparatively easier to implement through our next pattern; being Low-Code.

Low-Code:

This approach takes advantage of some of the “black box” approaches and integrations of Azure, abstracting away some of the difficulty in orchestrating microservices that are found in the purely code-first approach. A number of these are, PromptFlow and Co-Pilot Studio. These offer a more streamlined approach towards that of a RAG-style co-pilot and allow for the goal of a PoC to be achieved, that much faster and more efficiently. A great example of this is found here.

Prompt Flow, as the orchestrator, offers special benefits through abstractions and prebuilt “nodes”, that can streamline and automate a large amount of the code that we would have to write, and even goes as far as one-click creation of complex data structures through automated embeddings and vectorization databases, massively speeding up this phase, and bringing us closer to real value.

No-Code:

Finally, we have a number of no code accelerators for typical co-pilot scenarios, that abstract everything through the GUI and allow us to very quickly adapt a predefined siloed dataset into the base of knowledge that we need for a co-pilot PoC. The typical one is called “Chat with your Data” from both the co-pilot studio and Azure OpenAI portals.

From a PoC point of view, this really allows for the speed and efficiency of this stage to be realised. Without complex code, or specific knowledge around GenAI, this method really allows us to drive and focus on value, before potentially including more complexity in a later stage.

Hybrid:

This approach involves using a combination of the above approaches, depending on the complexity of the co-pilot. For example, a developer in this phase can use the code first approach to write the core logic of the code, and then use the no-code approach to generate additional code features or functionalities. A great example of this is using Prompt Flow, first starting to work on the solution in either a no-code or low-code approach, and then iterating through code subsequently.

GenAI Application PoC Flow

The process depicted above shows how the MSFT team is actively involved in assisting our customers in choosing a PoC path, regardless of the PoC development methodology. We will support customers in assessing the strategy, considering factors such as the use case, skills, technology preference, viability, and timeline.

Summary

To summarize, the text describes three different approaches to developing a co-pilot using GenAI:

Code first: This approach involves writing the code manually and then using GenAI to improve it or add new features. This is suitable for developers who have prior experience with coding and want to have more control over the code quality and functionality.

No-code: This approach involves using a graphical interface or natural language to specify the requirements and then using GenAI to generate the code automatically. This is suitable for non-developers who want to create a co-pilot without writing any code and focus on the value proposition.

Hybrid: This approach involves using a combination of the above approaches, depending on the complexity of the co-pilot. For example, a developer can use the code first approach to write the core logic and then use the no-code approach to generate additional features or functionalities. This is suitable for developers who want to leverage the best of both worlds and iterate quickly.

Series: Next article will discuss about consideration and approach of moving from GenAI PoC to MVP

Author: Morgan Gladwell,

Co-Author: @arung 

@Paolo Colecchia @Stephen Rhoades @Taonga_Banda @renbafa @morgan Gladwell