Good Morning

I’m using this beautiful Gantt formatting from GitHub but I’m struggling to modify it. – To be clear, I’m trying to create a view  similar to an Excel sheet because people can’t cope with too much change at a time!

This is what it looks like.

I want to change two things and neither works. That’s a reflection of my JSON knowledge more than anything else.

I want to remove the inline edit function.And I don’t want to show this at all when another field has a value in it.

This is the code:

{
“$schema”: “https://developer.microsoft.com/json-schemas/sp/v2/column-formatting.schema.json”,
“elmType”: “div”,
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row”,
“width”: “100%”,
“align-items”: “center”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “column”,
“width”: “100%”,
“margin-bottom”: “3px”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“justify-content”: “space-between”,
“width”: “100%”,
“white-space”: “nowrap”
},
“attributes”: {
“class”: “ms-fontSize-xs”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row-reverse”
},
“children”: [
{
“elmType”: “div”,
“txtContent”: “=if(loopIndex(‘_startdate’)>1,”,’/’)+[$_startdate]”,
“forEach”: “_startdate in split(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’)),’-‘)”
}
]
},
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row-reverse”
},
“children”: [
{
“elmType”: “div”,
“txtContent”: “=if(loopIndex(‘_enddate’)>1,”,’/’)+[$_enddate]”,
“forEach”: “_enddate in split(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’)),’-‘)”
}
]
}
]
},
{
“elmType”: “div”,
“style”: {
“width”: “100%”,
“border”: “1px solid”,
“height”: “13px”,
“position”: “relative”,
“overflow”: “hidden”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“position”: “absolute”,
“height”: “100%”,
“width”: “=((Number(Date(substring(@currentField,indexOf(@currentField,’^’)+1,indexOf(@currentField,'(‘))))-Number(Date(substring(@currentField,0,indexOf(@currentField,’^’))))+86400000)/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”,
“left”: “=((Number(Date(substring(@currentField,0,indexOf(@currentField,’^’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’)))))/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”
},
“attributes”: {
“class”: “ms-bgColor-themePrimary”,
“title”: “=[$Start.displayValue]+’ ～ ‘+[$End.displayValue]”
}
},
{
“elmType”: “div”,
“style”: {
“position”: “absolute”,
“height”: “100%”,
“width”: “=(86400000/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”,
“left”: “=((Number(@now)-(60*60*1000*12)-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’)))))/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”,
“display”: “flex”,
“justify-content”: “center”,
“align-items”: “center”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“width”: “100%”,
“min-width”: “5px”,
“height”: “100%”
},
“attributes”: {
“class”: “ms-bgColor-sharedRed10”,
“title”: “Now”
}
}
]
}
]
}
]
},
{
“elmType”: “div”,
“style”: {
“padding”: “10px”,
“margin-left”: “5px”,
“cursor”: “pointer”,
“border-radius”: “50%”
},
“attributes”: {
“iconName”: “Edit”,
“class”: “ms-bgColor-themeLighter–hover”
},
“customCardProps”: {
“openOnEvent”: “click”,
“directionalHint”: “topCenter”,
“isBeakVisible”: true,
“formatter”: {
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “column”,
“padding”: “10px 15px”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row”,
“align-items”: “center”
},
“children”: [
{
“elmType”: “div”,
“inlineEditField”: “[$Start]”,
“style”: {
“border”: “1px solid”,
“padding”: “5px 10px”,
“border-radius”: “3px”,
“display”: “flex”,
“align-items”: “center”,
“margin”: “5px”
},
“children”: [
{
“elmType”: “span”,
“txtContent”: “[$Start.displayValue]”
},
{
“elmType”: “span”,
“style”: {
“margin-left”: “8px”
},
“attributes”: {
“iconName”: “Edit”
}
}
]
},
{
“elmType”: “div”,
“txtContent”: “～”,
“style”: {
“margin”: “5px”
}
},
{
“elmType”: “div”,
“inlineEditField”: “[$End]”,
“style”: {
“border”: “1px solid”,
“padding”: “5px 10px”,
“border-radius”: “3px”,
“display”: “flex”,
“align-items”: “center”,
“margin”: “5px”
},
“children”: [
{
“elmType”: “span”,
“txtContent”: “[$End.displayValue]”
},
{
“elmType”: “span”,
“style”: {
“margin-left”: “8px”
},
“attributes”: {
“iconName”: “Edit”
}
}
]
}
]
},
{
“elmType”: “div”,
“txtContent”: “If you changed the date manually instead of using the calendar, press [Enter] after changing the date.”,
“style”: {
“max-width”: “300px”,
“text-align”: “center”
},
“attributes”: {
“class”: “ms-fontSize-s”
}
}
]
}
}
}
]
}
]
}

I would appreciate any suggestions. Whatever I have taken out so far hasn’t changed a thing.

Thanks, 

Christine 

​Good Morning I’m using this beautiful Gantt formatting from GitHub but I’m struggling to modify it. – To be clear, I’m trying to create a view  similar to an Excel sheet because people can’t cope with too much change at a time! This is what it looks like. I want to change two things and neither works. That’s a reflection of my JSON knowledge more than anything else.I want to remove the inline edit function.And I don’t want to show this at all when another field has a value in it.This is the code: {
“$schema”: “https://developer.microsoft.com/json-schemas/sp/v2/column-formatting.schema.json”,
“elmType”: “div”,
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row”,
“width”: “100%”,
“align-items”: “center”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “column”,
“width”: “100%”,
“margin-bottom”: “3px”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“justify-content”: “space-between”,
“width”: “100%”,
“white-space”: “nowrap”
},
“attributes”: {
“class”: “ms-fontSize-xs”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row-reverse”
},
“children”: [
{
“elmType”: “div”,
“txtContent”: “=if(loopIndex(‘_startdate’)>1,”,’/’)+[$_startdate]”,
“forEach”: “_startdate in split(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’)),’-‘)”
}
]
},
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row-reverse”
},
“children”: [
{
“elmType”: “div”,
“txtContent”: “=if(loopIndex(‘_enddate’)>1,”,’/’)+[$_enddate]”,
“forEach”: “_enddate in split(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’)),’-‘)”
}
]
}
]
},
{
“elmType”: “div”,
“style”: {
“width”: “100%”,
“border”: “1px solid”,
“height”: “13px”,
“position”: “relative”,
“overflow”: “hidden”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“position”: “absolute”,
“height”: “100%”,
“width”: “=((Number(Date(substring(@currentField,indexOf(@currentField,’^’)+1,indexOf(@currentField,'(‘))))-Number(Date(substring(@currentField,0,indexOf(@currentField,’^’))))+86400000)/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”,
“left”: “=((Number(Date(substring(@currentField,0,indexOf(@currentField,’^’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’)))))/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”
},
“attributes”: {
“class”: “ms-bgColor-themePrimary”,
“title”: “=[$Start.displayValue]+’ ～ ‘+[$End.displayValue]”
}
},
{
“elmType”: “div”,
“style”: {
“position”: “absolute”,
“height”: “100%”,
“width”: “=(86400000/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”,
“left”: “=((Number(@now)-(60*60*1000*12)-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’)))))/(Number(Date(substring(@currentField,indexOf(@currentField,’_’)+1,indexOf(@currentField,’)’))))-Number(Date(substring(@currentField,indexOf(@currentField,'(‘)+1,indexOf(@currentField,’_’))))+86400000))*100+’%'”,
“display”: “flex”,
“justify-content”: “center”,
“align-items”: “center”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“width”: “100%”,
“min-width”: “5px”,
“height”: “100%”
},
“attributes”: {
“class”: “ms-bgColor-sharedRed10”,
“title”: “Now”
}
}
]
}
]
}
]
},
{
“elmType”: “div”,
“style”: {
“padding”: “10px”,
“margin-left”: “5px”,
“cursor”: “pointer”,
“border-radius”: “50%”
},
“attributes”: {
“iconName”: “Edit”,
“class”: “ms-bgColor-themeLighter–hover”
},
“customCardProps”: {
“openOnEvent”: “click”,
“directionalHint”: “topCenter”,
“isBeakVisible”: true,
“formatter”: {
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “column”,
“padding”: “10px 15px”
},
“children”: [
{
“elmType”: “div”,
“style”: {
“display”: “flex”,
“flex-direction”: “row”,
“align-items”: “center”
},
“children”: [
{
“elmType”: “div”,
“inlineEditField”: “[$Start]”,
“style”: {
“border”: “1px solid”,
“padding”: “5px 10px”,
“border-radius”: “3px”,
“display”: “flex”,
“align-items”: “center”,
“margin”: “5px”
},
“children”: [
{
“elmType”: “span”,
“txtContent”: “[$Start.displayValue]”
},
{
“elmType”: “span”,
“style”: {
“margin-left”: “8px”
},
“attributes”: {
“iconName”: “Edit”
}
}
]
},
{
“elmType”: “div”,
“txtContent”: “～”,
“style”: {
“margin”: “5px”
}
},
{
“elmType”: “div”,
“inlineEditField”: “[$End]”,
“style”: {
“border”: “1px solid”,
“padding”: “5px 10px”,
“border-radius”: “3px”,
“display”: “flex”,
“align-items”: “center”,
“margin”: “5px”
},
“children”: [
{
“elmType”: “span”,
“txtContent”: “[$End.displayValue]”
},
{
“elmType”: “span”,
“style”: {
“margin-left”: “8px”
},
“attributes”: {
“iconName”: “Edit”
}
}
]
}
]
},
{
“elmType”: “div”,
“txtContent”: “If you changed the date manually instead of using the calendar, press [Enter] after changing the date.”,
“style”: {
“max-width”: “300px”,
“text-align”: “center”
},
“attributes”: {
“class”: “ms-fontSize-s”
}
}
]
}
}
}
]
}
]
}  I would appreciate any suggestions. Whatever I have taken out so far hasn’t changed a thing. Thanks, Christine   Read More

​

A very common task for Microsoft Fabric, Azure Data Factory and Synapse Analytics Pipelines is to receive unstructured files, land them in an Azure Data Lake (ADLS Gen2) and load them into structured tables. This often leads to a very common issue with unstructured files when “SOMETHING HAS CHANGED” and the unstructured file does not meet the defined table format. If issues are not handled properly within the pipeline, the data workloads will fail and users will be asking “WHERE’S MY DATA???” You then need to communicate with the owner of the file, have them fix the issues, then rerun the pipeline after the issues have been fixed. Along with unhappy users, rerunning failed pipelines adds cost. Validating these files before they are processed allows your pipelines to continue ingesting files that do have the correct format. For pipelines that do fail, your code or process can pinpoint what caused the error, leading to faster resolution of the issue. In this blog, we’ll walk through a Microsoft Fabric Data Factory Pipeline that validates incoming CSV files for common errors before loading to a Microsoft Fabric Lakehouse delta table.

Overview

This source files in this process are in an Azure Data Lake storage account, which has a shortcut in the Fabric Lakehouse. A data pipeline calls a Spark notebook to check the file for any new or missing columns, any invalid data for the expected data type, or any duplicate key values. If the file has no errors, the pipeline loads the CSV data into a parquet file and then calls another Spark notebook to load the parquet file into a delta table in the Lakehouse. Otherwise if there are errors in the file, the pipeline sends a notification email.

Source files and metadata files

In this solution, files are landing in an ADLS Gen 2 container folder called scenario1-validatecsv which has a shortcut to it in the Fabric Lakehouse. The files folder contains the files to process; the metadata folder contains a file describing the format each CSV file type should conform to. 

This solution is to load to a table called customer, which has columns number, name, city and state. In the format definition file, customer_meta, there’s a row for each customer table column, providing the column name, the column data type, and whether or not it is a key value. This metadata file is later used in a Spark notebook to validate that the incoming file conforms to this format.

Orchestrator Pipeline

The orchestrator pipeline is very simple – since I am running my pipeline as a scheduled batch, it loops through the files folder and invokes another pipeline for each file.  Note the parametrization of the lakehouse path, the source folders, the destination folder and the file format. This allows the same process to be run for any lakehouse and for any file format/table to load.

For Each activity

When invoking the child pipeline from the For Each activity, it passes in the parameter values from the orchestrator pipeline plus the name of the current file being processed and the metadata file name, which is the file format name with ‘_meta’ appended to it.

Child pipeline – Validate and load pipeline

The Validate and load pipeline validates the current CSV file, and if the file conforms to the format, loads it into a parquet table then merges the parquet data into a delta table.

1 – Parameters

Parameters passed in from the orchestrator pipeline for the current CSV file to process

2 – Set variable activity – Set parquet file name

Removes .csv to define the parquet file name

3- Notebook activity – Validate CSV and load file to parquet 

Calls the notebook, passing in parameter and variable values

Below is the pyspark code for the notebook. It gets the column names and inferred data types from the CSV file as well as the column names, data types and key field names from the metadata file. It checks if the column names match, if the data types match, if there are keys defined for the file, and finally if there are any duplicate key values in the incoming file.  If there were duplicate key fields, it writes the duplicate key values to a file. If the names and data types match and there are no duplicate key values, it writes the file to parquet and passes back the key field names from the metadata file; otherwise, it returns the appropriate error message.

# Files/landingzone/files parameters
lakehousepath = ‘abfss://xxxxx@msit-onelake.dfs.fabric.microsoft.com/xxxxx’
filename = ‘customer_good.csv’
outputfilename = ‘customer_good’
metadatafilename = ‘customer_meta.csv’
filefolder = ‘scenario1-validatecsv/landingzone/files’
metadatafolder = ‘scenario1-validatecsv/landingzone/metadata’
outputfolder = ‘scenario1-validatecsv/bronze’
fileformat = ‘customer’

# Import pandas and pyarrow
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# Set path variables
inputfilepath = f'{lakehousepath}/Files/{filefolder}/’
metadatapath = f'{lakehousepath}/Files/{metadatafolder}/’
outputpath = f'{lakehousepath}/Files/{outputfolder}/’

# Read the text file and the metadata file
print(f'{inputfilepath}{filename}’)
data = pd.read_csv(f'{inputfilepath}{filename}’)
meta = pd.read_csv(f'{metadatapath}{metadatafilename}’)

# only get the column names for the file formattype that was input
meta = meta.loc[meta[‘formatname’] == fileformat]
print(data.dtypes)
print(list(meta[‘columname’]))

# get any key fields specified
keyfields = meta.loc[meta[‘iskeyfield’] == 1, ‘columname’].tolist()
print(keyfields)

# Check for errors in CSV
haserror = 0
# Check if the column names match
if list(data.columns) != list(meta[“columname”]):
# Issue an error
result = “Error: Column names do not match.”
haserror = 1
else:
# Check if the datatypes match
if list(data.dtypes) != list(meta[“datatype”]):
# Issue an error
result = “Error: Datatypes do not match.”
haserror = 1
else:
# If the file has key fields, check if there are any duplicate keys
# if there are duplicate keys, also write the duplicate key values to a file
if keyfields != ”:
checkdups = data.groupby(keyfields).size().reset_index(name=’count’)
print(checkdups)
if checkdups[‘count’].max() > 1:
dups = checkdups[checkdups[‘count’] > 1]
print(dups)
haserror = 1
(dups.to_csv(f'{lakehousepath}/Files/processed/error_duplicate_key_values/duplicaterecords_{filename}’,
mode=’w’,index=False))
result = ‘Error: Duplicate key values’

if haserror == 0:
# Write the data to parquet if no errors
df = spark.read.csv(f”{inputfilepath}{filename}”, header=True, inferSchema=True)
print(f’File is: {inputfilepath}{filename}’)
display(df)
df.write.mode(“overwrite”).format(“parquet”).save(f”{outputpath}{outputfilename}”)
result = f”Data written to parquet successfully. Key fields are:{keyfields} “

mssparkutils.notebook.exit(str(result))

4 – Copy data activity – Move File to processed folder

This Copy Data activity essentially moves the csv file from the ADLS Gen 2 files folder to a processed folder in the Fabric Lakehouse

Destination folder name is derived from the Notebook exit value, which returns success or the error message

5 – If condition activity: If File Validated

Check if the CSV file was successfully validated and loaded to parquet

5a – File validated successfully

If there were no errors in the file, call the spark notebook to merge the parquet file written from the previous pyspark notebook to the delta table. 

Parameters for the lakehousepath, the parquet file path and name, the table name, and the key fields passed in. As shown above, the key fields were derived from the previous pyspark notebook and are passed into the Create or Merge to Table notebook.

Below is the spark notebook code. If the delta table already exists and there are key fields, it builds a string expression to be used on the pyspark merge statement and then performs the merge on the delta table. If there are no key fields or the table does not exist, it writes or overwrites the delta table.

# create or merge to delta

# input parameters below
lakehousepath = ‘abfss://xxxe@yyyy.dfs.fabric.microsoft.com/xxx’
inputfolder = ‘scenario1-validatecsv/bronze’
filename = ‘customergood’
tablename = ‘customer’
keyfields = “[‘number’]”

# define paths
outputpath = f'{lakehousepath}/Tables/{tablename}’
inputpath = f'{lakehousepath}/Files/{inputfolder}/{filename}’

# import delta table and sql functions
from delta.tables import *
from pyspark.sql.functions import *

# get list of key values
keylist = eval(keyfields)
print(keylist)

# read input parquet file
df2 = spark.read.parquet(inputpath)
# display(df2)

# if there are keyfields define in the table, build the merge key expression
if keyfields != None:
mergekey = ”
keycount = 0
for key in keylist:
mergekey = mergekey + f’t.{key} = s.{key} AND ‘
mergeKeyExpr = mergekey.rstrip(‘ AND’)
print(mergeKeyExpr)

# if table exists and if table should be upserted as indicated by the merge key, do an upsert and return how many rows were inserted and updated;
# if it does not exist or is a full load, overwrite existing table return how many rows were inserted

if DeltaTable.isDeltaTable(spark,outputpath) and mergeKeyExpr is not None:
deltaTable = DeltaTable.forPath(spark,outputpath)
deltaTable.alias(“t”).merge(
df2.alias(“s”),
mergeKeyExpr
).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()
history = deltaTable.history(1).select(“operationMetrics”)
operationMetrics = history.collect()[0][“operationMetrics”]
numInserted = operationMetrics[“numTargetRowsInserted”]
numUpdated = operationMetrics[“numTargetRowsUpdated”]
else:
df2.write.format(“delta”).mode(“overwrite”).save(outputpath)
numInserted = df2.count()
numUpdated = 0
print(numInserted)

result = “numInserted=”+str(numInserted)+ “|numUpdated=”+str(numUpdated)
mssparkutils.notebook.exit(str(result))

5b – File validation failed

If there was an error in the CSV file, send an email notification

Summary

Building resilient and efficient data pipelines is critical no matter your ETL tool or data sources. Thinking ahead to what types of problems can, and inevitably will, occur and incorporating data validation into your pipelines will save you a lot of headaches when those pipelines are moved into production. The examples in this blog are just a few of the most common errors with CSV files. Get ahead of those data issues and resolve them without last minute fixes and disrupting other processes! You can easily enhance the methods in this blog by including other validations or validating other unstructured file types like Json. You can change the pipeline to run as soon as the unstructured file is loaded into ADLS rather than in batch. Using techniques like this to reduce hard errors gives your pipelines (and yourself!) more credibility!

​Microsoft Tech Community – Latest Blogs –Read More 

Organizations are increasingly adopting a hybrid workplace and Windows Autopilot provides flexibility to deliver devices to users anywhere with internet connectivity. With more and more adoption of Windows Autopilot, Microsoft Intune is enhancing this solution to support a greater variety of scenarios and use cases.

Today, Intune is releasing a new Autopilot profile experience, Windows Autopilot device preparation, which enables IT admins to deploy configurations efficiently and consistently and removes the complexity of troubleshooting for both commercial and government (Government Community Cloud (GCC) High, and U.S. Department of Defense (DoD)) organizations and agencies.

What is Windows Autopilot device preparation and why was it created?

While the existing Windows Autopilot experience supports multiple scenarios and device types, we’re extending this value across additional cloud instances and improving consistency and troubleshooting capabilities based on customer feedback. We’re introducing Autopilot device preparation in a way that won’t interrupt current deployments or experience and provides a more consistent and efficient experience.

Among some of the benefits of Autopilot device preparation are:

Availability in government clouds (GCC High and DoD) which will allow government customers to deploy at scale using Autopilot.
Providing more consistency in user experience during deployments by locking in IT admins intentions for onboarding.
Creating more error resiliency in the experience to allow users to recover without needing to call a help desk.
Sharing more insight into the Autopilot process with new reporting details.

A single Autopilot device preparation profile to configure deployment and OOBE settings

The Autopilot device preparation admin experience simplifies admin configuration by having a single profile to provision all policies in one location, including deployment settings and out-of-box (OOBE) settings. It also improves the consistency of the experience for users and gets them to the desktop faster by allowing you to select which apps (line-of-business (LOB), Win32, or Store apps) and PowerShell scripts must be delivered during OOBE.

A new Windows Autopilot device preparation section is available under Enrollment > Windows. Admins will configure a single Device preparation policy to configure deployment and user experience.

Grouping at enrollment time

An improved grouping experience places devices in a group at the time of enrollment. Simply assign all configurations to a device security group and include the group as part of the device preparation profile. The configuration will be saved and then delivered on the device as soon as the user authenticates during OOBE.

When creating the Device preparation profile, admins will also include a device security group. During provisioning, the device will be added to this group and receive the configuration applied to the group.

New user experience in OOBE

A simplified OOBE view shows the progress of the deployment in percentage % so that users know how far along in the process they are. When the device preparation configuration has been delivered to the device, the user will be informed that critical setup is complete, and they can continue to the desktop.

New Out-of-box-experience (OOBE) experience shows progress of the deployment in percentages.

The Autopilot device preparation deployment report

The new Autopilot device preparation deployment report captures the status of each deployment in near real-time and provides detailed information to help with troubleshooting. Here are some highlights of what to expect:

Easily track which devices went through Autopilot
Track status and deployment phase in near-real-time
Expand more details for each deployment:

Device details
Profile name and version
Deployment status details
Apps applied with status
Scripts applied with status

New Autopilot device preparation deployment report shows details such as apps and scripts status for configurations selected in the device preparation profile.

Coming soon: Corporate identifiers for Windows

While we don’t have a tenant association feature ready in this initial release, we understand the importance of only allowing known devices to enroll to your tenant. So, we’ll soon expand enrollment restrictions to include Windows corporate identifiers. Autopilot device preparation will support the new corporate identifier enrollment feature <link to doc>. This added functionality will allow you to pre-upload device identifiers and ensure only trusted devices go through Autopilot device preparation. Stay tuned to What’s new in Intune for the release!

New enrollment restriction to add corporate identifiers for Windows devices through a CSV upload.

Frequently Asked Questions

How is this new Autopilot profile different from the current Autopilot profile?

The new Autopilot profile is a re-architecture of the current Autopilot profile so while the experience to OEMs, IT admins and users may look the same, the underlying architecture is very different. The updated architecture in the new Autopilot profiles gives the admin new capabilities that improve the deployment experience.

New orchestration agent allows the experience to fail fast and provide more error details.
Targeting is more precise and avoids dynamic changes when dynamic grouping is used.
Reporting infrastructure provides more details on the deployment experience.

Who does the new Autopilot profile benefit?

The new profile will benefit government customers who can now use Windows Autopilot device preparation to streamline their deployments at scale. It’ll also benefit new customers onboarding Windows Autopilot by reducing the complexity of setting up the deployment.

Is the new profile available in all sovereign clouds?

The new profile is available for Government Community Cloud (GCC) High and U.S. Department of Defense (DoD). It’s expected to be available for Intune operated by 21Vianet in China later this year.

What about the other Autopilot scenarios like pre-provisioning and self-deploying mode?

These functionalities will be supported in the future but aren’t part of the initial release.

Why is there a limit on the number of apps I can select to be delivered during OOBE?

We limited the number of applications that can be applied during OOBE to increase stability and achieve a higher success rate. Looking at our telemetry, almost 90% of all Autopilot deployments are deployed with 10 or fewer apps. This limit is intended to improve the overall user experience so that users can become more productive quickly. We understand that there are outliers and companies that want to target more during setup, but for the user-driven approach, we want to leverage the desktop experience for non-essential applications.

What is the order of installation for the device preparation profile?

The process is described in detail in: Overview for Windows Autopilot device preparation user-driven Microsoft Entra join in Intune.

Can we now mix app types such as LOB and Win32 apps with the device preparation profile?

While we always recommend Win32 apps, in current Autopilot deployments, mixing apps may result in errors. With the device preparation profile, we’ve streamlined the providers so different app types should not impact each other.

What is the guidance on user- vs device-based targeting?

Only device-based configurations will be delivered during OOBE. Assign security policy to devices, ensure all selected apps in the device preparation profile are set to install in system context, and are targeted to the device security group specified in the profile.

How will users know when the setup is complete?

Many users aren’t sure when the provisioning process is complete. To help mitigate confusion and calls to the help desk, we’re adding a completion page in OOBE. Admins can configure the pageto require a user to manually select to continue or set the page to auto-continue. This message will let the user know that OOBE setup is complete but there may be additional installations happening that they can monitor in the Intune Company Portal.

Can the new profile be used by other MDMs?

Windows Autopilot device preparation will support 3rd party MDMs. In this initial release, configuration is only possible via Intune.

Will this be available on Windows 10 devices?

Currently, device preparation profiles are only available on:

Windows 11, version 23H2 with KB5035942 or later.
Windows 11, version 22H2 with KB5035942 or later.

How can I move my existing devices to the new device preparation profile?

If you’d like to have an existing device join your tenant through the device preparation profile, the device would first need to be de-registered from Autopilot, then retargeted to a security group within your device preparation profile.

Do I need to migrate my existing profiles from Autopilot-to-Autopilot device preparation?

There’s no need to migrate from existing Autopilot to the new Autopilot profile. We expect both environments to exist in parallel for a while as we work to improve the experience and add more functionality.

Does this mean we are no longer investing in Autopilot?

Not at all! We’re continuing to work on Autopilot in parallel with developing Autopilot device preparation. The first release of Autopilot device preparation won’t have all the scenarios of Autopilot, specifically pre-provisioning and self-deploying modes, so we’ll continue to invest in those areas. Additionally, where possible, we plan to add any high value features from Autopilot device preparation to Autopilot to improve the experience for all customers.

If you have any questions, leave a comment below or reach out to us on X @IntuneSuppTeam. Stay tuned to What’s new in Intune and What’s new in Autopilot as we continue developing this new deployment experience.

​Microsoft Tech Community – Latest Blogs –Read More