This Terraform module creates dynamic Datadog dashboards from metrics discovered in Datadog, with flexible filtering and aggregation options. It uses an external script to query available metrics and automatically generates dashboard widgets.

Features

Dynamic Metric Discovery: Queries Datadog API for available metrics by prefix
Auto-Generated Widgets: Creates timeseries widgets automatically from metrics
Template Variables: Interactive filtering via dashboard variables
Dual Creation Modes: Metric discovery vs. pre-defined metric list
Metric Aliasing: Metadata expressions for cleaner metric names

Resources Created

datadog_dashboard: Dynamically generated dashboard with timeseries widgets

Requirements

Name	Version
terraform	>= 0.12
datadog	>= 2.0
bash	For list_metrics.sh script

Usage

Mode 1: Metric Discovery by Prefix

module "dynamic_dashboard" {
  source = "./terraform-datadog-dashboard"

  title              = "Application Metrics Dashboard"
  api_key            = var.datadog_api_key
  app_key            = var.datadog_app_key
  prefix             = "myapp.*"
  space_aggregation  = "avg"
  scope              = "environment:production,service:api"
  description        = "Auto-generated dashboard for application metrics"
}

Mode 2: Pre-Defined Metrics List

module "custom_dashboard" {
  source = "./terraform-datadog-dashboard"

  title              = "Custom Metrics Dashboard"
  api_key            = var.datadog_api_key
  app_key            = var.datadog_app_key
  prefix             = ""
  space_aggregation  = "sum"
  scope              = "env:prod"
  metrics_list       = "metric1,metric2,metric3"
  description        = "Dashboard with specific metrics"
}

Inputs

Name	Description	Type	Required	Default
`title`	Dashboard title	`string`	no	`"my dashboard"`
`api_key`	Datadog API key	`string`	yes	-
`app_key`	Datadog APP key	`string`	yes	-
`prefix`	Metric prefix pattern for discovery	`string`	yes	-
`space_aggregation`	Aggregation method (avg, max, min, sum)	`string`	yes	-
`scope`	Tag scope for filtering	`string`	yes	-
`description`	Dashboard description	`string`	no	`"generated by Terraform"`
`metrics_list`	Comma-separated list of metrics (alternative to prefix)	`string`	no	`""`
`template_variable_name`	Name of template variable	`string`	no	`"variable1"`
`template_variable_prefix`	Prefix of template variable	`string`	no	`""`
`template_variable_default`	Default value for template variable	`string`	no	`"*"`

Outputs

Output	Description
`url`	Dashboard URL (https://app.datadoghq.com/dashboard/{id})
`external_metrics`	List of metrics discovered from Datadog

External Script

The module uses list_metrics.sh to query the Datadog API:

#!/bin/bash
# list_metrics.sh
# Queries Datadog for metrics matching the prefix pattern

The script is called via the external data source:

data "external" "metrics" {
  program = ["bash", "${path.module}/list_metrics.sh"]
  
  query = {
    api_key = var.api_key
    app_key = var.app_key
    prefix  = var.prefix
  }
}

Dashboard Creation Logic

The module uses conditional resource creation:

dashboard_a: Created when using metric discovery (prefix-based)
dashboard_b: Created when using pre-defined metrics list
Only ONE dashboard is created based on which mode is used

count = var.metrics_list == "" ? 1 : 0  # dashboard_a
count = var.metrics_list != "" ? 1 : 0  # dashboard_b

Template Variables

The dashboard includes a template variable for interactive filtering:

template_variable {
  name    = var.template_variable_name
  prefix  = var.template_variable_prefix
  default = var.template_variable_default
}

Example: Filter by environment, host, service, etc.

Each discovered metric gets a timeseries widget:

Display Type: Line graph
Aggregation: Configured via space_aggregation variable
Scope: Filtered by scope variable
Alias: Metadata expression for cleaner names

request {
  q = "${var.space_aggregation}:${metric.value}{${var.scope}}"
  
  metadata {
    expression = "${var.space_aggregation}:${metric.value}{${var.scope}}"
    alias_name = metric.value
  }
}

Aggregation Methods

Supported aggregation methods:

avg: Average values
max: Maximum values
min: Minimum values
sum: Sum of values

Scope Filtering

Scope uses Datadog tag syntax:

environment:production                    # Single tag
environment:production,service:api        # Multiple tags (AND)
environment:production OR service:api     # OR logic
environment:prod*                         # Wildcard

Examples

CPU Metrics Dashboard

module "cpu_dashboard" {
  source = "./terraform-datadog-dashboard"

  title             = "CPU Utilization Dashboard"
  prefix            = "system.cpu.*"
  space_aggregation = "avg"
  scope             = "env:production"
  api_key           = var.datadog_api_key
  app_key           = var.datadog_app_key
}

Custom Application Dashboard

module "app_dashboard" {
  source = "./terraform-datadog-dashboard"

  title             = "Application Performance"
  prefix            = "myapp.performance.*"
  space_aggregation = "max"
  scope             = "service:myapp,env:prod"
  
  template_variable_name    = "host"
  template_variable_prefix  = "host"
  template_variable_default = "*"
  
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

Pre-Defined Metrics

module "specific_metrics" {
  source = "./terraform-datadog-dashboard"

  title             = "Key Metrics"
  prefix            = ""  # Not used in this mode
  space_aggregation = "avg"
  scope             = "env:prod"
  metrics_list      = "aws.ec2.cpuutilization,aws.rds.database_connections,custom.metric.value"
  api_key           = var.datadog_api_key
  app_key           = var.datadog_app_key
}

Layout

Layout Type: Ordered (widgets stacked vertically)
Widget Type: Timeseries
Auto-Generated: One widget per discovered metric

Notes

Requires external bash script (list_metrics.sh) in module directory
Script must be executable (chmod +x list_metrics.sh)
Metric discovery happens during Terraform plan/apply
Dashboard updates automatically when new metrics appear
Uses external data source which runs on every plan
Consider caching if metric list is large and changes infrequently

Troubleshooting

Script Execution Issues

Ensure the script is executable:

chmod +x terraform-datadog-dashboard/list_metrics.sh

No Metrics Discovered

Verify API/APP keys have correct permissions
Check that metrics matching prefix exist in Datadog
Confirm metric prefix syntax (e.g., "myapp." not "myapp")

Dashboard Not Created

Check if metrics_list vs prefix logic is correct
Verify exactly one creation mode is used (not both)

Best Practices

Prefix Naming: Use specific prefixes to limit widget count
Aggregation: Choose appropriate aggregation for metric type
Scope: Use specific scopes to reduce cardinality
Template Variables: Add variables for interactive filtering
Testing: Test metric discovery in dev before production

Limitations

One dashboard per module instance
Script runs on every plan (can be slow with many metrics)
Ordered layout only (no custom positioning)
All widgets are timeseries type
No support for mixed widget types

Future Enhancements

Potential improvements:

Widget type customization
Free layout support
Widget positioning control
Query value widgets
Toplist widgets
Note widgets for organization

License

Internal use only - Sanoma/WeBuildYourCloud

Authors

Created and maintained by the Platform Engineering team.

README.md

Terraform Datadog Dashboard Module

Overview