The platform engineering journey requires abundant information to make informed decisions. Understanding how developers use the platform—how frequently and for how long—is invaluable. Since the internal developer portal (IDP) serves as the central hub for developers’ regular tasks, it becomes the ideal location for monitoring these activities and collecting essential data.

Backstage, a common IDP implementation, provides an excellent opportunity to demonstrate how to integrate monitoring. In this article, we’ll explore how to add monitoring to the Backstage portal using Azure Monitor, specifically Application Insights.

Upon reviewing the Backstage documentation, we find that the portal is already instrumented using OpenTelemetry. Furthermore, Application Insights is a supported provider for OpenTelemetry.

This is excellent news, as it streamlines the process and saves time that would otherwise be spent manually adding tracing calls to the Backstage code.

To enable OpenTelemetry instrumentation, follow the steps in the official documentation(https://backstage.io/docs/tutorials/setup-opentelemetry/). Start by adding the required packages using the following command:

yarn –cwd packages/backend add

@opentelemetry/sdk-node

@opentelemetry/auto-instrumentations-node

/monitor-opentelemetry-exporter

Next, create a new file named instrumentation.js inside the backend/src folder.

The content of the file should be: 

const { NodeSDK } = require(‘@opentelemetry/sdk-node’);
const {
getNodeAutoInstrumentations,
} = require(‘@opentelemetry/auto-instrumentations-node’);
const { AzureMonitorTraceExporter } = require(“@azure/monitor-opentelemetry-exporter”);
const { AzureMonitorMetricExporter } = require(“@azure/monitor-opentelemetry-exporter”);

// Create an exporter instance
const azTraceExporter = new AzureMonitorTraceExporter({
connectionString:
process.env[“APPLICATIONINSIGHTS_CONNECTION_STRING”] || “<YourAppInsightsConnectionString>”
});

const sdk = new NodeSDK({
traceExporter: azTraceExporter,
instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

And with that…we are done!

Browse some pages within the Backstage portal and in a few minutes, telemetry will be available in Application Insights.

Here are some samples of information that has been gathered. All this information is out-of-the-box reports, so without any other customization, we can obtain a wealth of information.

We can check if the portal is returning errors, if the response time is within acceptable parameters, how many requests are made and the usage pattern…

Examine the details of a request to Backstage, the time taken to serve the request and the time spent on all dependencies…

Retrieve information from the accessed URLs, for example, to identify the most common searches and ascertain developers’ interests.

​Microsoft Tech Community – Latest Blogs –Read More 

The Modern Workplace Conference Paris 2024, primarily aimed at French-speaking users, has been held. In its sixth edition, the conference featured various speakers, including Microsoft MVPs, Regional Directors, industry technology experts, and Microsoft employees. They delivered more than 75 sessions over two days, providing attendees with the latest technical information on Microsoft 365, Power Platform, including Microsoft AI and Copilot.

In this blog, we introduce the insights and future community activities of two MVPs who contributed significantly to the success of this conference.

Microsoft MVP, Chloé Moreau, describes her own role at the conference as “a big project manager or a conductor,” working alongside talented MVPs and passionate community leaders to ensure the best planning and execution. Although they are highly motivated community leaders rather than professional event organizers, they tackled challenges head-on and were proud of delivering a high-quality event. “Our best asset is that we are passionate people loving to share content and knowledge on our preferred technologies, for MWCP it is Microsoft 365 and Power Platform,” says Chloé.

Two years ago, the fourth event was held entirely online due to the limitations of pandemic, which over the following years, transformed into a hybrid format. Hybrid events enable extensive information sharing about new technologies not only to on-site attendees but also to remote participants. However, managing multiple communication channels makes the operation more complex. For this event, in addition to the on-site venue, organizer members managed live streaming on YouTube and used Teams for sharing slides and facilitating participant interaction. Reflecting on these efforts, Chloé states, “We were able to offer a great quality of interactions both online and in person by really proving all hybrid options to our attendees.”

Regarding concerns that offering more participation options might reduce the number of people attending in person, Chloé mentions, “This year we had a very strong level of attendees participating in person despite all content being accessible on YouTube. So, we are convinced that hybrid is the way to go for the future as Microsoft showed this for Microsoft Ignite for example.”

Chloé expresses a strong desire to continue supporting community members through various online channels. In addition to sharing the latest Microsoft news on LinkedIn, Facebook, X, and Instagram, the community hosts groups with over 6,500 professionals worldwide, monthly podcasts, and various community events that have attracted over 10,000 participants across five continents in the past eight years. By continuously supporting participants who meet at events, community leaders continue to demonstrate exemplary leadership that contributes to the growth of both individuals and the community as a whole.

Another key organizer, Patrick Guimonet, also an MVP, played a significant role in coordinating numerous aspects of the event from planning to execution, greatly contributing to its success. He expresses his joy, saying, “I am very proud to contribute to the success of this event and to be part of such a passionate, motivated, and committed team. Each edition is a new adventure that allows us to unite people and share our passion.”

Through interactions with participants, Patrick learned the importance of session diversity and innovation. He discovered that participants are not just looking for various presentations but are interested in practical case studies that provide specific knowledge. This feedback, offering practical insights into both business and technical challenges, was highly valuable.

Like Chloé, Patrick also reflects on the hybrid nature of the event, emphasizing that the YouTube session videos met participant expectations, stating, “This flexibility allows them to review sessions at their leisure, catch up on any content they might have missed, or simply refresh their memory on the information presented.”

Looking ahead to future community activities, Patrick mentioned the regular events hosted by aMP (formerly aMS) community. These events offer opportunities for direct interaction at local events and include online streaming, allowing participation regardless of location. They provide a great hybrid event experience where the community can learn about the latest updates of Microsoft 365, Power Platform, and more. Community events are planned in various cities worldwide, so we encourage you to join these events.

– June 25, 2024: aMS Brazzaville

– August 1, 2024: aMP Colombo

– August 24, 2024: aMP Pune

– August 27, 2024: aMP Salem

– September 12, 2024: aMP Tunis

– September 19, 2024: aMP Bordeaux

– September 27, 2024: aMS Leipzig

– October 3, 2024: aMP Montpellier

– October 18, 2024: aMP Seoul

– October 23, 2024: aMP Kuala Lumpur

– October 25, 2024: aMP Manila

​Microsoft Tech Community – Latest Blogs –Read More 

Keyword or metadata values (keyword query language)Previously identified patterns of sensitive information like social security, credit card, or bank account numbers (Sensitive information type entity definitions)Document fingerprinting: recognizing an item because it’s a variation on a templateThe presence of exact strings exact data match

​Hello All,Just sharing some tips to assist with the process of data collection and the creation of trainable classifiers for the purpose of labelling/Data Loss prevention. -Regarding training Machine Learning to recognize a certain document type, It must have one or more recognizable aspects.  Possible usable recognizable aspects of the data/document type:Keyword or metadata values (keyword query language)Previously identified patterns of sensitive information like social security, credit card, or bank account numbers (Sensitive information type entity definitions)Document fingerprinting: recognizing an item because it’s a variation on a templateThe presence of exact strings exact data match  -In the below examples, we focus on Document Fingerprinting and Previously identifiable Sensitive information Type.    For e.g.Regarding positive samples, The below file samples display a pattern, CC info (dummy data), Include Keywords referring to CC info such CVV2/AMEX etc…. as well as SSN information.     -This can be regarded as a pattern for positive detection. The above data samples (about 150 samples of a similar pattern) are stored in a folder in a dedicated SharePoint Site(In the below screenshot, Same items are used as false samples for another classifier).   -Regarding Negative samples, It is the same concept, It can be also stored in a folder in a dedicated Sharepoint Site and have a unique pattern or fingerprint. for e.g. -The below samples represent Credential information (dummy), Need to be about 150 samples or so. The samples should strongly represent a uniform document/data type different from positive samples.       Similarly the data is stored in a dedicated folder in a  SharePoint Site:     Once the trainable classifier is created and fed this information, It will successfully identify data type to facilitate detection and minimize potential false positive.     Read More

​

1- Introduction

PTUs are reserved processing capacity, ensuring stable performance for uniform LLM workloads. The reserved capacity of PTUs makes KV caching more effective compared to Pay-As-You-Go (PayGo). This blog post delves into the role of Key-Value (KV) caching in enhancing PTU throughput, and practical strategies to create cache-friendly prompts that maximize efficiency.

2- What are Provisioned Throughput Units (PTUs)?

Provisioned Throughput Units (PTUs) in Azure represent a dedicated model processing capacity that can be reserved and deployed for handling prompts and generating completions. The key benefits of PTUs include:

Predictable Performance: Ensures stable maximum latency and throughput for uniform workloads.
Reserved Processing Capacity: Once deployed, the throughput is available irrespective of utilization.
Cost Savings: High throughput workloads may lead to cost savings compared to token-based consumption models.

3- KV Caching: Enhancing Efficiency in Language Models

Key-Value (KV) caching is a technique employed in generative transformer models, such as language models (LLMs), to optimize the inference process. Key aspects of KV caching include:

Reduction of Computational Cost: Minimizes the need to recompute key and value tensors for past tokens during each generation step.
Memory-Compute Trade-off: Tensors are stored (cached) in GPU memory, balancing memory usage and compute efficiency.

4- Crafting KV Cache-Friendly Prompts:

To optimize your prompts for KV caching, consider the following strategies:

Position Dynamic Elements Wisely: Place dynamic elements, such as grounding data, date & time, or chat history, toward the end of your prompt.
Maintain Order for Static Elements: Keep static elements like safety instructions, examples, and tool/function definitions at the beginning and in a consistent order.
Dedicate Your PTU Deployment: Dedicating your deployment to few use cases can further improve cache hit rates, as the requests will be more uniform.

5- A Case Study with GPT4-T-0409:

The following experiments focused on the impact of the cacheable/fixed percentage of the prompt on system performance, specifically average time-to-first-token and throughput. The results showed a clear trend: as the fixed/cacheable part of the prompt increased, the average latency decreased and the request capacity increased.

General Settings:

Model: GPT4-T-0409
Region: UK South
PTU: 100
Load test duration: 5 min

Experiment 1:

Input token size: 10245
Output token size: 192

Cacheable % of the prompt

1%

25%

50%

75%

Throughput (request/min)

7

9

12.5

20

Time to first token (sec)

2.4

2.0

1.77

1.3

      Analysis:

Throughput Improvement: As the cacheable percentage of the prompt increased from 1% to 75%, throughput saw a significant increase from 7 requests per minute to 20 requests per minute. This translates to nearly a threefold improvement, highlighting the efficiency gain from caching.
Latency Reduction: The time to the first token decreased from 2.4 seconds to 1.3 seconds as the cacheable percentage increased. This reduction in latency indicates faster initial response times, which is crucial for user experience.

Experiment 2:

Input token size: 5000
Output token size: 100

Cacheable % of the prompt

1%

25%

50%

75%

Throughput (request/min)

17

22

32

55

Time to first token (sec)

1.31

1.25

1.16

0.9

Analysis:

Throughput Improvement: When the cacheable percentage of the prompt increased from 1% to 75%, throughput saw an impressive rise from 17 requests per minute to 55 requests per minute. This more than threefold increase demonstrates the substantial impact of cache-friendly prompts on system performance.
Latency Reduction: The time to the first token improved from 1.31 seconds to 0.9 seconds with higher cacheable percentages. This faster response time is beneficial for applications requiring real-time or near-real-time interactions.

* The results may vary based on the model type, deployment region, and use case.

Summary of the results:

In both experiments, a longer cacheable part of the prompt resulted in significant boosts in throughput and reductions in latency. The improvements were more pronounced in Experiment 2, likely due to the smaller input token sizes.

Throughput: Across both experiments, a higher cacheable percentage of the prompt resulted in substantial increases in throughput. In Experiment 1, throughput increased by almost 186%, and in Experiment 2, it increased by approximately 224% from the lowest to the highest cacheable percentage.

Latency: The time to the first token decreased consistently as the cacheable percentage of the prompt increased. This reduction in latency enhances the user experience by providing quicker initial responses.

These results underscore the importance of optimizing prompts to be cache-friendly, thereby maximizing the performance of the system in terms of both throughput and latency. By leveraging caching strategies, systems can handle more requests per minute and provide faster responses, ultimately leading to a more efficient and scalable AI deployment.

6- Conclusion

Provisioned Throughput Units (PTUs) in Azure offer significant advantages in terms of performance, capacity, and cost savings. By leveraging KV caching and creating cache-friendly prompts, you can further enhance the efficiency of your AI workloads. Optimizing prompt structure not only maximizes the benefits of PTUs but also ensures more effective and resource-efficient model processing.

7- Acknowledgments

A special thanks to Michael Tremeer for his invaluable review and feedback on this blog post. Your insights have greatly enhanced the quality of this work.

8- References

Transformers KV Caching Explained | by João Lages | Medium

Techniques for KV Cache Optimization in Large Language Models (omrimallis.com)

​Microsoft Tech Community – Latest Blogs –Read More 

Co-authors: Priya Kedia, Michael Tremeer

Contributors: Ranjani Mani

Phi-3 Vision, a lightweight and state-of-the-art open multimodal model, is a significant advancement in Microsoft’s AI offerings. Developed with a focus on producing a high-quality, reasoning focused model, Phi-3 Vision utilizes synthetic data and curated publicly available web data to ensure its robustness and versatility. At only 4.2 billion parameters, it strikes an impressive balance between performance and efficiency, making it an attractive option for a wide range of applications.

As the first multimodal model in the Phi-3 family, Phi-3 Vision extends beyond the capabilities of its predecessors – Phi-3-mini, Phi-3-small, and Phi-3-medium – by seamlessly blending language and visual input. It boasts a context length of 128K tokens, allowing it to support complex and nuanced interactions. Designed with the intention to run on devices, Phi-3 Vision provides the benefits of offline operation, cost-effectiveness, and user privacy.

Phi-3 Vision has demonstrated versatility across various use cases, including Optical Character Recognition (OCR), Image Captioning, Table Parsing, and Reading Comprehension on Scanned Documents, among others. Its ability to provide high-quality reasoning with both visual and text input capabilities will drive innovation and lead to the development of new applications that are both transformative and sustainable. As an example, here is a quick demo showcasing how car footage can be analyzed to assess vehicle damages on an edge device, giving instant feedback to end user. When paired together with a larger LLM like GPT-4o, Phi-3 can form part of hybrid workflow that combines the efficiencies of Phi-3 for simpler tasks with the power of GPT-4o for more challenges tasks, unlocking the best of both worlds in a multi-step pipeline.

Market Trends

The landscape of artificial intelligence (AI) is in a state of rapid evolution, and within this space,

Microsoft’s Phi-3-Vision emerges as a noteworthy trendsetter. Phi-3-Vision, a member of Microsoft’s broader Phi-3 family, represents a significant leap in multimodal AI capabilities, blending language and vision processing.

The Rise of Multimodal AI Models

Multimodal AI models, such as the Phi-3-Vision, are increasingly gaining attention due to their ability to interpret and analyze both textual and visual data. This dual capability not only enhances user interaction with digital content but also opens up new avenues for data analysis and accessibility. As businesses and consumers alike demand more intuitive and capable AI solutions, the prominence of multimodal models is expected to grow.

Open Source as a Catalyst for Innovation

Phi-3-Vision’s open-source nature stands out as a key trend in the AI market. By allowing developers to access and build upon the model freely, Microsoft is fostering a community-driven ecosystem where innovation can thrive. This approach is likely to inspire other AI developers and companies to adopt and build upon the model, potentially leading to a surge in collaborative AI advancements.

Efficiency and Edge Computing

Another significant trend is the shift towards more efficient AI models that can operate on devices with limited computational power, such as smartphones and edge devices. Phi-3-Vision’s compact yet powerful architecture exemplifies this trend, which is driven by the need for cost-effective and less compute-intensive AI services. As a result, the market is witnessing a growing interest in AI models that are optimized for on-device, edge, and offline inference scenarios.

AI Accessibility and Democratization

The Phi-3 project’s goal to democratize AI through smaller, efficient models aligns with a broader market trend towards making AI more accessible to everyday users and developers. By making the model available on Azure AI Studio, Azure AI model catalog as well as on hugging face, Microsoft has simplified the adoption and integration of AI capabilities into various applications.

Future Integration in Various Industries

Phi-3-Vision’s adaptability and performance indicate a trend towards integrating advanced AI models into a wide array of industries. From document digitization to advanced automation solutions, Phi-3-Vision and similar models are set to transform various sectors by enhancing productivity and reducing operational costs.

Competitive Landscape

Despite its relatively compact size, Phi-3-Vision demonstrates impressive performance that is on par with much larger models, and it is one of the smallest LLMs with multimodal capabilities. This efficiency makes it particularly suitable for deployment on devices with limited computational resources, such as smartphones. In addition, the optimized versions of the model in ONNX format ensure accelerated inference on both CPU and GPU across different platforms, including server, desktop, and mobile environments.

Model Architecture and Capabilities

Phi-3 Vision is based on the Transformer model architecture, which has demonstrated remarkable success in various NLP tasks. It contains an image encoder, connector, projector, and Phi-3 Mini language model. The model’s ability to support up to 128K context length in tokens with just 4.2 billion parameters allows for extensive multimodal reasoning, making it adept at understanding and generating content from complex visual inputs like charts, graphs, and tables. Its integration into the development version (4.40.2) of the industry-standard transformers python library further simplifies its adoption in AI-driven applications.

Training Data and Quality

One of the factors that differentiates Phi-3 Vision is its training data. Unlike many other models that rely solely on human-generated data (such as from the web and published books etc.), the training datasets used to train the Phi-3 family of models are created using advanced synthetic data generation techniques, along with highly curated public web data. This approach aims to maximize the quality of the training data with a specific focus on helping the model to develop advanced reasoning skills and the ability to solve problems. This training dataset contributes to the model’s robustness and versatility, enabling it to perform well beyond expectations in various visual reasoning tasks. It has demonstrated superior performance in a range of multimodal benchmarks, outperforming competitors such as Claude 3 Haiku and coming close to the capabilities of more much larger models like OpenAI’s GPT-4V.

Performance Comparison of Phi-3 Vision

Target Use Cases and Applicability

In the broader AI industry, there is a strong trend of replacing larger models like GPT-4o with more efficient models like Phi-3 as AI builders seek to optimize their GenAI use-cases. A common pattern is to launch a use case with a powerful LLM like GPT-4o, and once the solution is in production, look to incorporate a more efficient SLM like Phi-3 for some of the less complicated and more narrow parts of the problem. This also means that the initial batch of production data that is generated by GPT-4o can be used to fine-tune the Phi-3 model, offering comparable accuracy of the large model at a fraction of the cost. This approach has been documented as a reliable and effective technique for reducing the costs of LLM-powered solutions while maintaining similar performance.

Given this trend, Phi-3 offers a potential to be leveraged for many use cases involving memory/compute constrained environments, latency bound scenarios, general image understanding, OCR, chart and table understanding etc.

Phi-3 Vision Use Case Spectrum

Document and Image Analysis for KYC

Use Case: Combining text extraction and image classification to streamline the Know Your Customer (KYC) process. This helps in verifying customer identity and ensuring compliance with legal and regulatory standards in sectors like banking and financial services. Example: Automating the verification of identity documents such as passports and driving licenses by extracting text and checking the validity of images to expedite the KYC process.

Enhanced Customer Support and Product Returns

Use Case: Using text and image analysis to enhance customer support operations, including the management of product returns. This approach helps in quickly identifying issues through customer descriptions and photos of returned items, thereby improving customer satisfaction and operational efficiency. Example: Automatically processing customer complaints that include photos of defective products, enabling rapid resolution through efficient handling of returns or exchanges.

Content Moderation for Social Media

Use Case: Integrating text and image analysis to identify and moderate inappropriate content on social media platforms. This helps in maintaining community standards and ensuring a safe environment for users. Example: Automatically detecting and removing posts with offensive language and harmful images, ensuring compliance with community guidelines and promoting a positive user experience.

Video Footage Analysis for Auto and Home Insurance

Use Case: Analyzing video footage for assessing damages and verifying claims in auto and home insurance sectors. This capability allows for accurate evaluation of incidents and helps in processing claims more efficiently. Example: Processing video footage of a car accident to identify the cause and extent of damage, aiding in quick and accurate claim settlements. Similarly, evaluating home damage videos for insurance claim assessments.

Visual Content Analysis for Educational Tools

Use Case: Utilizing text and image analysis to develop interactive and adaptive educational tools. This can enhance learning by providing customized content and feedback based on both text and visual inputs from students. Example: Creating adaptive learning platforms that analyze students’ written responses and hand-drawn diagrams to offer personalized feedback and additional resources.

With the trend towards decentralized computing, users of edge devices such as smartphones, tablets, and IoT devices require lightweight AI models that can operate with limited computing resources. Phi-3 Vision’s ability to run efficiently on smaller devices makes it attractive to this demographic. By leveraging ONNX Runtime Mobile and Web, Microsoft is working to enable Phi-3 Vision on a broad spectrum of devices, from smartphones to wearables. This has led to an interest in Phi-3 vision models from a wide demographic of customers.

Target Customer Demographics

Partnerships and Collaborations

Partnerships with industry players, as seen with DDI Labs’ integration of Phi-3 Vision, can lead to transformative applications in areas such as video analytics and automation. Its potential to improve operations, such as in dock automation, demonstrates the practical benefits of adopting such advanced AI tools that address real-world challenges.

Taking a Deep Dive into Code

Getting Started

With basics taken care of, what’s next?

Deploy a quantized version of the model at the edge

https://onnxruntime.ai/blogs/accelerating-phi-3

Finetune the model for your domain specific use case

Phi-3CookBook/md/04.Fine-tuning/FineTuning_Qlora.md at main · microsoft/Phi-3CookBook (github.com)

Considerations for the future for Phi-3 Vision Team

Ethical Considerations and Bias Mitigation

Despite safety post-training, the potential for unfair or biased outcomes remains a concern due to societal biases reflected in training data. Ongoing efforts to mitigate these risks are critical to maintaining the integrity and social acceptability of AI technologies like Phi-3 Vision.

Computational and Energy Efficiency

As AI models grow in complexity and capability, ensuring computational and energy efficiency becomes increasingly challenging. Striking a balance between performance and resource consumption is essential for sustainable AI development, especially for models intended for widespread use across various devices.

Security and Privacy

With the proliferation of AI in personal and professional domains, security and privacy concerns must be addressed. Protecting user data and preventing unauthorized access or misuse of AI technologies are paramount for maintaining user trust and complying with regulatory requirements.

Final Thoughts

In conclusion, the Phi-3 family, spearheaded by Phi-3-vision, exemplifies the progress and potential of AI. While there are challenges to be addressed, the opportunities these models present are vast and ripe for exploration. As AI continues to evolve, models like Phi-3 Vision will undoubtedly be instrumental in shaping innovative solutions that could redefine the way we interact with technology and process information in our digital world.

References

Microsoft Official and Tech Community

https://azure.microsoft.com/en-us/blog/new-models-added-to-the-phi-3-family-available-on-microsoft-azure/
https://mspoweruser.com/microsoft-announces-phi-3-vision-a-new-multimodal-slm-for-on-device-ai-scenarios/
https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/affordable-innovation-unveiling-the-pricing-of-phi-3-slms-on/ba-p/4156495
https://techcommunity.microsoft.com/t5/educator-developer-blog/using-phi-3-amp-c-with-onnx-for-textand-vision-samples/ba-p/4161020
https://techcommunity.microsoft.com/t5/microsoft-developer-community/getting-started-generative-ai-with-phi-3-mini-running-phi-3-mini/ba-p/4147246
GitHub – microsoft/Phi-3CookBook: Samples for getting a quick understanding and exploration of Phi-3 models

Technical and AI-focused Publications

https://huggingface.co/blog/Emma-N/enjoy-the-power-of-phi-3-with-onnx-runtime
https://huggingface.co/microsoft/Phi-3-medium-128k-instruct
https://huggingface.co/microsoft/Phi3 Vision
https://huggingface.co/microsoft/Phi3 Vision-onnx-cpu
https://huggingface.co/microsoft/Phi3 Vision-onnx-cuda
https://venturebeat.com/ai/microsoft-phi-3-generally-available-phi-3-vision-preview/
https://www.analyticsvidhya.com/blog/2024/05/microsoft-phi3/
https://towardsdatascience.com/6-real-world-uses-of-microsofts-newest-phi-3-vision-language-model-8ebbf
https://www.digit.in/news/general/microsoft-announced-phi-3-vision-an-ai-model-for-phones-that-can-analyse-pictures.html

​Microsoft Tech Community – Latest Blogs –Read More