AWS

Overview

To set up Cloud Cost Management in Datadog, you need:

1. An AWS account with billing access
2. The AWS integration installed in Datadog
3. A Cost and Usage Report (follow the steps below to create one)

Setup

You can setup using the API, Terraform, or directly in Datadog by following the instructions below.

Configure the AWS integration

Navigate to Setup & Configuration, add an AWS account and follow the steps to configure the AWS integration.

Note: Datadog recommends configuring a Cost and Usage Report from an AWS management account for cost visibility into related member accounts.

If you send a Cost and Usage Report from an AWS member account, ensure that you have selected the following options in your management account’s preferences:

• Linked Account Access
• Linked Account Refunds and Credits
• Linked Account Discounts

These settings ensure complete cost accuracy by allowing periodic cost calculations against the AWS Cost Explorer.

{"Version": "2012-10-17","Statement": [{"Sid": "DDCloudCostReadBucket","Effect": "Allow","Action": ["s3:ListBucket"],"Resource": "arn:aws:s3:::BUCKETNAME"},{"Sid": "DDCloudCostGetBill","Effect": "Allow","Action": ["s3:GetObject"],"Resource": "arn:aws:s3:::BUCKETNAME/REPORT_PREFIX/REPORT_NAME/*"},{"Sid": "DDCloudCostCheckAccuracy","Effect": "Allow","Action": ["ce:Get*"],"Resource": "*"},{"Sid": "DDCloudCostListCURs","Action": ["cur:DescribeReportDefinitions"],"Effect": "Allow","Resource": "*"},{"Sid": "DDCloudCostListOrganizations","Action": ["organizations:Describe*","organizations:List*"],"Effect": "Allow","Resource": "*"}]}

Permissions for AWS Cost Optimization Hub recommendations

Cloud Cost Management generates some recommendations from data sourced from AWS Cost Optimization Hub. For Datadog to receive these recommendations, the Datadog AWS integration IAM role must include the following permissions:

• cost-optimization-hub:GetRecommendation
• cost-optimization-hub:ListRecommendations

These permissions are part of the default AWS integration IAM policy. If you configured the AWS integration recently with the CloudFormation or Terraform templates, you have these permissions already.

If you set up the AWS integration before these permissions were added to the default policy, update your IAM policy to include them. For the current policy, see the Amazon Web Services integration documentation.

Account filtering

Use Account Filtering to control which AWS member accounts to pull into Cloud Cost Management. Filtering out accounts does not incur additional Datadog costs.

Using Account Filtering requires an AWS management account. You can configure account filters after an account has been configured in Cloud Cost Management.

Note: Account filters are not supported for tag search.

Configure account filters for an existing account

Navigate to Cloud Cost > Settings, select Accounts, and then click Manage Account for the management account you want to filter.

Click Billing dataset to access the Account Filtering UI.

Getting historical data

If you configure a Cost and Usage Report that already has historical data available in S3, Datadog automatically ingests up to 15 months of historical cost data.

If your newly configured report does not have historical data, you can request a backfill from AWS:

To request a backfill of historical AWS cost data:

1. Open an AWS support case and request a backfill of your cost data.
2. Include the report name and the desired billing period in your request.
3. Wait for AWS to process the backfill request.

When the data is backfilled by AWS, Datadog automatically ingests the data within 24 hours.

AWS cannot backfill cost data that predates your AWS account or reflects a previous AWS Organizations structure.

For more information, see the AWS Cost and Usage Reports troubleshooting guide.

Cost types

Visualize your ingested data using out-of-the-box cost types. The cost types differ mainly in how they report on discount rates, savings plans and reservations.

On-demand

On-demand costs represent the cost of usage at the public, on-demand rate published by AWS. This excludes all savings plans, reservations, discounts, taxes, and fees.

Note: In most cases, on-demand costs are not a reliable source to estimate actual costs.

Amortized and unblended costs

Amortized cost metrics distribute commitment savings throughout the discount term. This is also called accrual basis. Reservations and savings plans are drawn down from a monthly commitment and applied directly to covered usage, at the time of usage. Any unused remainder appears as a fee.

In contrast, unblended cost metrics show all charges on the date that they are incurred. This is also called cost basis. Reservation and savings plan fees show up on the date they were charged, and are not applied directly to covered usage. After billing data for a month is finalized, unblended metrics match the AWS invoice exactly.

Net costs

Net costs apply private discounts directly to usage. The cost of usage for a specific resource represents the effective cost after all savings are realized.

In contrast, other metrics show private discounts as separate, negative-valued line items with no resource attribution tags. Rather than attributing the discounts directly to usage, those metrics subtract discounts from the total cost.

Net amortized costs provide the most accurate representation for cost allocation, with all savings applied directly to usage. Net cost metrics are available if your AWS account has privately negotiated enterprise discounts. If your account does not have enterprise discounts, then net amortized cost and amortized cost are equivalent.

Container allocation

Container allocation metrics contain all of the same costs as the AWS metrics, but with additional breakdowns and insights for container workloads. See container cost allocation for more details.

Example

The following scenario demonstrates how different cost types behave. Imagine you have:

• An EC2 instance running for one hour with the cost of $3 per compute-hour.
• A savings plan which prices this instance type at $2 per compute-hour.
• A negotiated EDP discount of 10% on top of all other discounts.

Here’s how the instance cost, savings plan hourly commitment, and discount appear in each cost type:

Cost metrics summary

In general:

• aws.cost.net.amortized.shared.resources.allocated provides the most complete cost allocation for specific workloads and teams.
• If you do not have container cost allocation, use aws.cost.net.amortized.
• If you do not have net amortized costs, use aws.cost.amortized.shared.resources.allocated or aws.cost.amortized.

How Datadog enriches your AWS cost data with tags

Datadog automatically enriches your AWS cost data with tags from multiple sources. For a comprehensive overview of how tags are applied to cost data, see Tags.

The following tag sources are available for AWS:

• Cost and Usage Report columns
• AWS Resource tags
• AWS Account tags
• AWS Integration tags
• Out-of-the-box tags
• Container workload tags
• Tag pipelines

Cost and Usage Report columns

All string-valued columns from the AWS Cost and Usage Report (CUR) are added as tags on cost metrics.

To ensure consistency, Datadog normalizes tag keys using underscores and lowercase. For example, the CUR column lineItem/ResourceId maps to the tag key line_item/resource_id. Tag values are generally unmodified, maintaining exact casing and most special characters.

Examples:

AWS resource tags

AWS resource tags are user-defined tags that appear in the AWS console when viewing a particular resource, such as an EC2 instance or S3 bucket.

When you enable the Datadog AWS integration, Datadog automatically collects resource tags for most AWS resources. These tags are applied to all costs found in the CUR for a given resource. Resource tags are retrieved regularly and are applied to cost data starting from the day they are created or modified. Historical tag values are not overwritten when tags change.

If the AWS integration is not enabled, you can enable resource tag enrichment by activating cost allocation tags in AWS billing. This allows you to select a subset of resource tag keys to include as columns in the AWS CUR. Datadog automatically includes those columns as tags when processing the CUR.

AWS organization and account tags

AWS Organizations support user-defined tags on organizational units and accounts. Datadog automatically fetches and applies these tags to cost data. Account tags are applied to all usage associated with those accounts. Organization tags are applied to all billing data for the matching payer account.

Requires the Datadog AWS Integration on the organization account.

AWS integration tags

AWS integration tags are tags set on the AWS Integration tile in the Datadog integrations page. They are applied to all costs found in the CUR for the associated AWS account.

Out-of-the-box tags

Datadog adds out-of-the-box tags to ingested cost data to help you further break down and allocate your costs. These tags are derived from your Cost and Usage Report (CUR) and make it easier to discover and understand cost data.

The following out-of-the-box tags are available for filtering and grouping data:

Cost and observability correlation

Viewing costs in context of observability data is important to understand how infrastructure changes impact costs, identify why costs change, and optimize infrastructure for both costs and performance. Datadog updates resource-identifying tags on cost data for top AWS products to simplify correlating observability and cost metrics.

For example, to view cost and utilization for each RDS database, you can create a table with aws.cost.amortized, aws.rds.cpuutilization, and aws.rds.freeable_memory (or any other RDS metric) and group by dbinstanceidentifier. To see Lambda usage and costs side by side, you can graph aws.lambda.concurrent_executions and aws.cost.amortized grouped by functionname.

The following out-of-the-box tags are available:

Container orchestrators

Container cost allocation adds tags from the workloads incurring cost. Examples include tags from Kubernetes pods and nodes and ECS tasks and containers.

Requires container cost allocation, and applies only to shared.resources.allocated metrics.

Tag pipelines

Finally, all of your tag pipeline rulesets are applied, providing complete cost allocation when infrastructure tagging is not possible. Tag pipelines are the final enrichment layer and add new tags to your cost data.

Billing conductor

AWS Billing Conductor is a custom billing service for AWS Marketplace Channel Partners and organizations that have chargeback requirements. Billing Conductor enables customers to create a second, pro forma version of their costs to share with their customers or account owners. Billing rates, credits and fees, and overhead costs can be customized at your discretion. You can also select which accounts to include in the CUR.

Important limitations:

• Pro forma Cost and Usage Reports do not include discounts and taxes, which makes it difficult to compare costs in Datadog to AWS Cost Explorer.
• Adding accounts to a billing group impacts how Reservations and Savings Plans are shared across AWS accounts.

To create a billing conductor CUR, follow the AWS Cost and Usage Reports user guide. Ensure the CUR meets Datadog’s requirements. After the billing conductor CUR is created, follow the Cloud Cost Management instructions above to set it up in Datadog.

Further reading

Request a personalized demo