Storage Management for Amazon S3

Setup

Set up Storage Management for Amazon S3 with one of the following methods:

• CloudFormation: A guided in-product setup that configures the AWS integration, enables S3 Inventory on the buckets you select, and optionally enables S3 access logs. A CloudFormation stack applies the changes to your AWS account.
• Terraform: Use the official Datadog Storage Management Terraform module to configure inventory and access logs as code.
• Manual: Set up S3 inventory and the required permissions yourself in the AWS console, then register the inventory destination with Storage Management.

provider "aws" {
 region = "<AWS_REGION>"
}
provider "datadog" { # Configure with environment variables:
 # DD_API_KEY, DD_APP_KEY, DD_SITE
}
module "datadog_storage_management" {
 source = "DataDog/storage-management-datadog/aws"
 name = "<MODULE_NAME>"
 datadog_aws_integration_role_name = "<DATADOG_AWS_INTEGRATION_ROLE_NAME>"
 source_bucket_names = ["<SOURCE_BUCKET_1>", "<SOURCE_BUCKET_2>"]
 destination_bucket_name = "<DESTINATION_BUCKET_NAME>" # Prefix within the destination bucket where inventory files are written.
 # Defaults to "datadog-inventories/". The module keeps the inventory
 # configuration, bucket policy, and IAM permissions consistent with this
 # value. Datadog recommends to use the default value.
 # destination_prefix = "datadog-inventories/"
 # Bucket policy: "none", "create", or "merge" (default)
 destination_bucket_policy_management = "merge" # Optional: Enable S3 access logs for prefix-level request and latency metrics
 enable_access_logging = true
 datadog_forwarder_function_name = "<DATADOG_FORWARDER_FUNCTION_NAME>"
}

curl -X PUT "https://api.${DD_SITE}/api/v2/cloudinventoryservice/syncconfigs" \
 -H "Accept: application/json" \
 -H "Content-Type: application/json" \
 -H "DD-API-KEY: ${DD_API_KEY}" \
 -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
 -d '{
 "data": {
 "id": "aws",
 "type": "cloud_provider",
 "attributes": {
 "aws": {
 "aws_account_id": "<AWS_ACCOUNT_ID>",
 "destination_bucket_name": "<DESTINATION_BUCKET_NAME>",
 "destination_bucket_region": "<DESTINATION_BUCKET_REGION>",
 "destination_prefix": "<DESTINATION_PREFIX>"
 }
 }
 }
 }'

Validation

To verify your setup:

1. Wait for the first inventory report to generate (up to 24 hours for daily inventories).
2. Navigate to Infrastructure > Storage Management to see if the buckets you configured appear in the explorer list when Monitored buckets is selected.

Best practices

Follow these best practices to optimize Storage Management setup:

• Configure lifecycle policies for inventory destination buckets: S3 Inventory reports are generated daily and stored in your destination bucket. To prevent old inventory files from accumulating and incurring storage costs, add a lifecycle policy to automatically delete inventory reports older than three days.

• Configure life cycle policies for S3 access logs: If you have enabled S3 access logs for prefix-level request metrics, the raw log files accumulate in your destination bucket. After these logs are forwarded to Datadog, the raw files are no longer needed for Storage Management purposes. To automatically delete access log files after forwarding to Datadog, add a life cycle rule.

  Note: Before enabling automatic deletion, verify that there are no compliance or audit requirements in your organization that mandate retaining raw S3 access logs for a specific period.

• Create exclusion filters for S3 access logs: If S3 access logs are forwarded to Datadog only for Storage Management and don’t need to be indexed for search or analytics, add an exclusion filter to keep them out of indexed log volume.

Troubleshooting

If you don’t see data for buckets you set up for Storage Management, use the Storage Management Settings page to view all configured buckets, their inventory status, and any configuration errors. The page surfaces issues with actionable remediation steps. If you have any questions, contact Datadog.

Identify and act on cost savings with Bits Chat

FinOps and engineering teams can use Bits Chat and Storage Management to identify S3 cost savings opportunities, generate reports in Datadog Notebooks, and implement recommended changes. To use Bits Chat with Storage Management, enable the storage skill in the Bits Chat settings.

With the storage skill enabled for Bits Chat, you can:

• Find the biggest savings opportunities: Ask natural language questions to surface the highest-impact prefixes, storage classes, or buckets where lifecycle changes would reduce costs the most.
• Create reports through Notebooks: Generate a Datadog Notebook summarizing findings, estimated savings, and recommended actions for your team to review and share.
• Implement changes: Get step-by-step guidance with Bits Code to apply lifecycle policies, transition objects to cheaper storage tiers, or expire non-current versions in the prefixes with the highest savings potential.

Visualize granular S3 usage with inventory metrics

An out-of-the-box Storage Management S3 dashboard template is available to help you visualize the below metrics. You can clone and customize it to fit your needs.

*prefixN refers to prefix levels such as prefix0, prefix1, prefix2, and so on.

Note: Use the right metric for the question you’re answering:

• aws.s3.inventory.prefix_object_count and aws.s3.inventory.total_prefix_size (with the prefix tag) include everything inside a folder and all its subfolders. Use these when you want the total count or size for a specific folder (for example, “how much is in logs/2024/?”).
• aws.s3.inventory.prefix_object_count.levels and aws.s3.inventory.total_prefix_size.levels (with prefix1, prefix2, prefix3, and so on) count or size objects only at that exact depth. Use these when you want to build a treemap or compare folder sizes across levels (for example, “which top-level folders are the biggest?”).

Note: For the most accurate monitoring and visualization, include all object versions to see non-current object recommendations or metrics.

Act on optimizations with Storage Management Recommendations

Storage Management analyzes your inventory data and access logs to surface prefix-level recommendations for reducing S3 storage costs. These recommendations are available to all Storage Management customers. Potential savings are estimated using AWS list prices. If you have Cloud Cost Management enabled, recommendations also appear in Cloud Cost Recommendations, and you can track actual savings from optimizations.

Recommendations are run on a daily basis and are automatically refreshed in your account as soon as the recommendations are released.

Prerequisites

Seeing recommendations has the following prerequisites:

1. Configure S3 buckets for Storage Management by following the steps above on this page.
2. To see recommendations for moving infrequently accessed data to cheaper tiers by prefix, enable and forward S3 access logs to Datadog (Datadog Log Management fees apply).
3. To see recommendations for identifying non-current versions in prefixes, include “All versions” as part of the S3 Inventory configuration.

Available recommendations

• Transition unaccessed S3 data in prefix to Infrequent Access

• Expire old non-current version objects in S3 bucket prefix

• Consolidate small files in prefix to minimize per-object storage costs

Request a personalized demo