When to use AzCopy versus Azure PowerShell or Azure CLI
Overview
In this article you will learn the difference between API operations on Azure in the control plane and the data plane, how various tools such as AzCopy, Azure PowerShell, and Azure CLI leverage these APIs, and which tool fits best for your workload. All these tools are CLI based and work across platforms including Windows, Linux, and macOS.
Let’s start with a quick overview of the control plane and data plane, and how to perform operations such as creating a new storage account and uploading a new blob. After that, we’ll explore some of the tools available for interacting with your storage accounts and the data inside of them and the API surfaces those tools support.
API Operations
Azure API operations can be divided into two categories – control plane (also called the management plane) and data plane.
For in-depth details about Azure’s control plane and data plane, refer to the following link: Control plane and data plane operations – Azure Resource Manager.
Control plane
All requests for the control plane operation are sent to Azure Resource manager (ARM). Azure Resource Manager has a RESTful API surface with a URL that varies by the Azure environment. For public Azure regions the URL is: https://management.azure.com. You can find all the supported API calls for each Azure service in the Azure REST API reference documentation.
Azure Resource Manager sends the request to the resource provider, which completes the operation. In the case of Storage, it is called the Storage Resource Provider (SRP).
To create a new storage account, the corresponding HTTP PUT request must be transmitted.
Data plane
Request for data plane operations are sent to an endpoint that is specific to your storage account instance. You can find all supported API call for each Azure service in the Azure REST API reference documentation and for Storage at Azure Storage REST API Reference. Storage services have different data plane REST APIs for each proxy service including Blob, Data Lake Storage Gen2, Table, Queue, and File.
To upload a single blob in one operation, the PutBlob operation can be utilized:
PUT https://myaccount.blob.core.windows.net/mycontainer/myblob
The necessary request header is available at this link: Put Blob (REST API)
Tools
These tools are all designed to interact with the Azure APIs. The AzCopy command-line utility offers high-performance, scriptable data transfer for storage data plane operations. Azure CLI and Azure PowerShell offer more user-friendly options for executing control plane operations cross all Azure services. However, both can also be utilized for fundamental storage data plane operations.
AzCopy
Azure CLI
PowerShell
Control plane operations
No
Yes
Yes
Data plane operations
Yes
Yes
Yes
Single Files
Yes
Yes
Yes
Million Files
Yes (Multithreaded)
Not recommended
Not recommended
While Azure CLI and Azure PowerShell can be used to move multiple files, AzCopy is often better suited for it, especially when dealing with larger data sets, especially those that extend into the millions for copy operations.
AzCopy
This portable and very lightweight binary can be uses to copy files to and from Azure storage. It’s optimized for data plane operation at scale. For more on AzCopy see Get started with AzCopy.
AzCopy offers a wide range of authentication methods familiar to Azure users, such as device login and Managed Identity, both system and user-assigned. Furthermore, Service Principal authentication is supported, along with the capability to repurpose existing authentication tokens from Azure CLI or Azure PowerShell.
Once you’ve authenticated, or if you’ve supplied a SAS-Token to the source and/or destination, you can start utilizing the copy command. The following command should be used to copy data to or from Azure storage. It’s applicable to an individual file, a directory, or an entire container.
azcopy copy [source] [destination] [flags]
Refer to the documentation for detailed guidelines on uploading and downloading data, as well as using flags like include/exclude patterns, wildcards, and tagging data:
Upload files to Azure Blob storage by using AzCopy v10
Download blobs from Azure Blob Storage by using AzCopy v10
In addition, AzCopy integrates a job management subcommand to facilitate handling large-scale data transfers. The integrated sync command enables you to synchronize the source and destination based on the last modified timestamp. For more on how to synchronize data with AzCopy, see Synchronize with Azure Blob storage by using AzCopy v10.
AzCopy is not limited to uploading and downloading files to and from your local device. It also utilizes Azure storage server-to-server copy APIs, making it possible to transfer data between different storage accounts and import data directly from Amazon AWS S3 and Google Cloud Storage.
The diagrams below illustrate the process of migrating data between two Azure storage accounts by utilizing the server-to-server copy APIs.
A virtual machine client in Azure initiates the PutBlobFromURL server-to-server copy API call on the destination account using AzCopy.
The destination account understands the location of the original blob due to the details supplied in the “x-ms-copy-source” header.
The data is securely transferred from the source account to the destination account via the Microsoft backbone.
Examples of how to perform server-to-server copies with AzCopy can be found at:
Copy blobs between Azure storage accounts with AzCopy v10
Copy data from Amazon S3 to Azure Storage by using AzCopy
Copy from Google Cloud Storage to Azure Storage with AzCopy
Azure CLI
The Azure Command-Line Interface (CLI) is a cross-platform command-line tool to connect to Azure and execute administrative commands on Azure resources. It allows the execution of commands through a terminal using interactive command-line prompts or a script. For more on Azure CLI, see Get started with Azure Command-Line Interface (CLI).
Azure CLI supports several common authentication methods. The simplest starting point is the Azure Cloud Shell, but you can also use interactive login, Service Principal, and Managed Identities.
The Azure CLI is versatile and allows you to carry out a wide range of control plane tasks, ranging from creating a storage account to the more complex activities such as establishing network rules or configuring encryption scopes.
You can create a new storage account with a single command. For example:
az storage account create -n [accountName] -g [resourceGroupName] -l [region] –sku [storageSKU]
Azure CLI also is capable of performing simple data plane activities, which include uploading, downloading or copying either an individual file or a whole directory. Nonetheless, for scenarios involving a large number of files, it is strongly advised to utilize AzCopy.
To transfer an individual blob to a storage account container, the command below may be utilized:
az storage blob upload -f /path/to/file -c [containerName] -n [blobName]
Find the complete list of commands at: az storage.
Azure PowerShell
Azure PowerShell is a set of cmdlets for managing Azure resources directly from PowerShell. Azure PowerShell is designed to make it easy to learn and get started with, but provides powerful features for automation. For more on Azure PowerShell, see Get started with Azure PowerShell.
Azure PowerShell, like the previously mentioned tools, supports various authentication methods as well – both interactive and non-interactive.
Like Azure CLI, Azure PowerShell manages control plane operations. The choice of which set of tools you wish to use is entirely yours. Should you wish to incorporate Storage tasks into an existing PowerShell script, then the Az.Storage module could be a preferable option. For those already operating within a Linux shell environment, one might find the Azure CLI to be quickly recognizable and user-friendly.
You can create a new storage account quickly with Azure PowerShell. For example:
$StorageHT = @{
ResourceGroupName = $ResourceGroup
Name = ‘mystorageaccount’
SkuName = ‘Standard_LRS’
Location = $Location
}
$StorageAccount = New-AzStorageAccount @StorageHT
$Context = $StorageAccount.Context
For a basic blob upload to a storage account container, you can use a command such as the following:
$Blob1HT = @{
File = ‘D:ImagesImage001.jpg’
Container = $ContainerName
Blob = “Image001.jpg”
Context = $Context
StandardBlobTier = ‘Hot’
}
Set-AzStorageBlobContent @Blob1HT
Should you need to download or copy blobs or directories, a comprehensive set of commands can be found at Az.Storage Module.
Conclusion
There are many options when it comes to interacting with Azure storage accounts from the command line. Your considerations for which tool you use will depend on the activity you are performing and the API surface that is required.
If you’re interacting with your resources on the control plane, Azure CLI and Azure PowerShell are powerful tools that allow you to create, manage, and delete resources. If you are interacting with data in your storage accounts and performing operations on the data plane, AzCopy is purpose built for uploading, downloading, and copying your data to and from Azure Storage in the most performant way.
Microsoft Tech Community – Latest Blogs –Read More
