👁 Image
Azure.Storage.DataMovement.Blobs 12.3.0

Azure Storage Data Movement Blobs client library for .NET

Azure Storage is a Microsoft-managed service providing cloud storage that is highly available, secure, durable, scalable, and redundant.

The Azure Storage Data Movement Blobs library is optimized for uploading, downloading and copying blobs.

The Azure.Storage.DataMovement.Blobs library provides infrastructure shared by the other Azure Storage client libraries.

Source code | Package (NuGet) | API reference documentation | REST API documentation | Product documentation

Getting started

Install the package

Install the Azure Storage Data Movement Blobs client library for .NET with NuGet:

dotnet add package Azure.Storage.DataMovement.Blobs

Prerequisites

You need an Azure subscription and a Storage Account to use this package.

To create a new Storage Account, you can use the Azure Portal, Azure PowerShell, or the Azure CLI. Here's an example using the Azure CLI:

az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS

Authenticate the client

The Azure.Storage.DataMovement.Blobs library uses clients from the Azure.Storage.Blobs package to communicate with the Azure Blob Storage service. For more information see the Azure.Storage.Blobs authentication documentation.

Permissions

The authenticated blob storage resource needs the following permissions to perform a transfer:

1. Read
2. List (for container transfers)
3. Write
4. Add (specific to append blobs)
5. Delete (for cleanup of a failed transfer item)
6. Create

Key concepts

The Azure Storage Common client library contains shared infrastructure like [authentication credentials][auth_credentials] and RequestFailedException.

Thread safety

We guarantee that all client instance methods are thread-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across threads.

Additional concepts

Client options | Accessing the response | Long-running operations | Handling failures | Diagnostics | Mocking | Client lifetime

Examples

This section demonstrates usage of Data Movement for interacting with blob storage.

Using the TransferManager for Blob Transfers

The TransferManager is the primary class for managing data transfers between storage resources. See Setup the TransferManager sample.

Initializing Blob Storage StorageResource

Azure.Storage.DataMovement.Blobs exposes BlobsStorageResourceProvider to create StorageResource instances for each type of blob (block, page, append) as well as a blob container. The resource provider should be initialized with a credential to properly authenticate the storage resources. The following demonstrates this using an Azure.Core token credential.

BlobsStorageResourceProvider blobs = new(tokenCredential);

To create a blob StorageResource, use the methods FromBlob or FromContainer.

StorageResource container = await blobs.FromContainerAsync(
 new Uri("https://myaccount.blob.core.windows.net/container"));

// Block blobs are the default if no options are specified
StorageResource blockBlob = await blobs.FromBlobAsync(
 new Uri("https://myaccount.blob.core.windows.net/container/sample-blob-block"),
 new BlockBlobStorageResourceOptions());
StorageResource pageBlob = await blobs.FromBlobAsync(
 new Uri("https://myaccount.blob.core.windows.net/container/sample-blob-page"),
 new PageBlobStorageResourceOptions());
StorageResource appendBlob = await blobs.FromBlobAsync(
 new Uri("https://myaccount.blob.core.windows.net/container/sample-blob-append"),
 new AppendBlobStorageResourceOptions());

Storage resources can also be initialized with the appropriate client object from Azure.Storage.Blobs. Since these resources will use the credential already present in the client object, no credential is required in the provider when using FromClient(). However, a BlobsStorageResourceProvider must still have a credential if it is to be used in TransferManagerOptions for resuming a transfer.

StorageResource containerResource = BlobsStorageResourceProvider.FromClient(blobContainerClient);
StorageResource blockBlobResource = BlobsStorageResourceProvider.FromClient(blockBlobClient);
StorageResource pageBlobResource = BlobsStorageResourceProvider.FromClient(pageBlobClient);
StorageResource appendBlobResource = BlobsStorageResourceProvider.FromClient(appendBlobClient);

There are more options which can be used when creating a blob storage resource. Below are some examples.

BlobStorageResourceContainerOptions virtualDirectoryOptions = new()
{
 BlobPrefix = "blob/directory/prefix"
};

StorageResource virtualDirectoryResource = BlobsStorageResourceProvider.FromClient(
 blobContainerClient,
 virtualDirectoryOptions);

BlockBlobStorageResourceOptions resourceOptions = new()
{
 Metadata = new Dictionary<string, string>
 {
 { "key", "value" }
 }
};
StorageResource leasedBlockBlobResource = BlobsStorageResourceProvider.FromClient(
 blockBlobClient,
 resourceOptions);

Upload

An upload takes place between a local file StorageResource as source and blob StorageResource as destination.

Upload a block blob.

TransferOperation transferOperation = await transferManager.StartTransferAsync(
 sourceResource: LocalFilesStorageResourceProvider.FromFile(sourceLocalPath),
 destinationResource: await blobs.FromBlobAsync(destinationBlobUri));
await transferOperation.WaitForCompletionAsync();

Upload a directory as a specific blob type.

TransferOperation transferOperation = await transferManager.StartTransferAsync(
 sourceResource: LocalFilesStorageResourceProvider.FromDirectory(sourcePath),
 destinationResource: await blobs.FromContainerAsync(
 blobContainerUri,
 new BlobStorageResourceContainerOptions()
 {
 // Block blobs are the default if not specified
 BlobType = BlobType.Block,
 BlobPrefix = optionalDestinationPrefix,
 }));

Download

A download takes place between a blob StorageResource as source and local file StorageResource as destination.

Download a blob.

TransferOperation transferOperation = await transferManager.StartTransferAsync(
 sourceResource: await blobs.FromBlobAsync(sourceBlobUri),
 destinationResource: LocalFilesStorageResourceProvider.FromFile(downloadPath));
await transferOperation.WaitForCompletionAsync();

Download a container which may contain a mix of blob types.

TransferOperation transferOperation = await transferManager.StartTransferAsync(
 sourceResource: await blobs.FromContainerAsync(
 blobContainerUri,
 new BlobStorageResourceContainerOptions()
 {
 BlobPrefix = optionalSourcePrefix
 }),
 destinationResource: LocalFilesStorageResourceProvider.FromDirectory(downloadPath));
await transferOperation.WaitForCompletionAsync();

Blob Copy

A copy takes place between two blob StorageResource instances. Copying between to Azure blobs uses PUT from URL REST APIs, which do not transfer data through the machine running DataMovement.

Copy a single blob. Note the destination blob is an append blob, regardless of the first blob's type.

TransferOperation transferOperation = await transferManager.StartTransferAsync(
 sourceResource: await blobs.FromBlobAsync(sourceBlobUri),
 destinationResource: await blobs.FromBlobAsync(destinationBlobUri, new AppendBlobStorageResourceOptions()));
await transferOperation.WaitForCompletionAsync();

Copy a blob container.

TransferOperation transferOperation = await transferManager.StartTransferAsync(
sourceResource: await blobs.FromContainerAsync(
 sourceContainerUri,
 new BlobStorageResourceContainerOptions()
 {
 BlobPrefix = sourceDirectoryName
 }),
destinationResource: await blobs.FromContainerAsync(
 destinationContainerUri,
 new BlobStorageResourceContainerOptions()
 {
 // all source blobs will be copied as a single type of destination blob
 // defaults to block blobs if unspecified
 BlobType = BlobType.Block,
 BlobPrefix = downloadPath
 }));
await transferOperation.WaitForCompletionAsync();

Resume using ShareFilesStorageResourceProvider

To resume a transfer with Blob(s), valid credentials must be provided. See the sample below.

TokenCredential tokenCredential = new DefaultAzureCredential();
BlobsStorageResourceProvider blobs = new(tokenCredential);
TransferManager transferManager = new TransferManager(new TransferManagerOptions()
{
 ProvidersForResuming = new List<StorageResourceProvider>() { blobs }
});
// Get resumable transfers from transfer manager
await foreach (TransferProperties properties in transferManager.GetResumableTransfersAsync())
{
 // Resume the transfer
 if (properties.SourceUri.AbsoluteUri == "https://storageaccount.blob.core.windows.net/containername/blobpath")
 {
 await transferManager.ResumeTransferAsync(properties.TransferId);
 }
}

For more information regarding pause, resume, and/or checkpointing, see Pause and Resume Checkpointing.

Extensions on BlobContainerClient

For applications with preexisting code using Azure.Storage.Blobs, this package provides extension methods for BlobContainerClient to get some of the benefits of the TransferManager with minimal extra code.

Instantiate the BlobContainerClient

BlobServiceClient service = new BlobServiceClient(serviceUri, credential);

BlobContainerClient container = service.GetBlobContainerClient(containerName);

Upload a local directory to the root of the container

TransferOperation transfer = await container.UploadDirectoryAsync(WaitUntil.Started, localPath);

await transfer.WaitForCompletionAsync();

Upload a local directory to a virtual directory in the container by specifying a directory prefix

TransferOperation transfer = await container.UploadDirectoryAsync(WaitUntil.Started, localPath, blobDirectoryPrefix);

await transfer.WaitForCompletionAsync();

Upload a local directory to a virtual directory in the container specifying more advanced options

BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions
{
 BlobContainerOptions = new BlobStorageResourceContainerOptions
 {
 BlobPrefix = blobDirectoryPrefix
 },
 TransferOptions = new TransferOptions()
 {
 CreationMode = StorageResourceCreationMode.OverwriteIfExists,
 }
};

TransferOperation transfer = await container.UploadDirectoryAsync(WaitUntil.Started, localPath, options);

await transfer.WaitForCompletionAsync();

Download the entire container to a local directory

TransferOperation transfer = await container.DownloadToDirectoryAsync(WaitUntil.Started, localDirectoryPath);

await transfer.WaitForCompletionAsync();

Download a directory in the container by specifying a directory prefix

TransferOperation transfer = await container.DownloadToDirectoryAsync(WaitUntil.Started, localDirectoryPath2, blobDirectoryPrefix);

await transfer.WaitForCompletionAsync();

Download from the container specifying more advanced options

BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions
{
 BlobContainerOptions = new BlobStorageResourceContainerOptions
 {
 BlobPrefix = blobDirectoryPrefix
 },
 TransferOptions = new TransferOptions()
 {
 CreationMode = StorageResourceCreationMode.OverwriteIfExists,
 }
};

TransferOperation transfer = await container.DownloadToDirectoryAsync(WaitUntil.Started, localDirectoryPath2, options);

await transfer.WaitForCompletionAsync();

Troubleshooting

See Handling Failed Transfers and Enabling Logging to assist with any troubleshooting.

See Known Issues for detailed information.

Next steps

Get started with our Share Files Samples.

For more base Transfer Manager scenarios see DataMovement samples.

Contributing

See the Storage CONTRIBUTING.md for details on building, testing, and contributing to these libraries.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact with any additional questions or comments.