terraform-datadog-dashboard/README.md

Terraform Datadog Dashboard Module

Overview

This Terraform module creates dynamic Datadog dashboards from metrics discovered in Datadog, with flexible filtering and aggregation options. It uses an external script to query available metrics and automatically generates dashboard widgets.

Features

• Dynamic Metric Discovery: Queries Datadog API for available metrics by prefix
• Auto-Generated Widgets: Creates timeseries widgets automatically from metrics
• Template Variables: Interactive filtering via dashboard variables
• Dual Creation Modes: Metric discovery vs. pre-defined metric list
• Metric Aliasing: Metadata expressions for cleaner metric names

Resources Created

• datadog_dashboard: Dynamically generated dashboard with timeseries widgets

Requirements

Name	Version
terraform	>= 0.12
datadog	>= 2.0
bash	For list_metrics.sh script

Usage

Mode 1: Metric Discovery by Prefix

module "dynamic_dashboard" {
  source = "./terraform-datadog-dashboard"

  title              = "Application Metrics Dashboard"
  api_key            = var.datadog_api_key
  app_key            = var.datadog_app_key
  prefix             = "myapp.*"
  space_aggregation  = "avg"
  scope              = "environment:production,service:api"
  description        = "Auto-generated dashboard for application metrics"
}

Mode 2: Pre-Defined Metrics List

module "custom_dashboard" {
  source = "./terraform-datadog-dashboard"

  title              = "Custom Metrics Dashboard"
  api_key            = var.datadog_api_key
  app_key            = var.datadog_app_key
  prefix             = ""
  space_aggregation  = "sum"
  scope              = "env:prod"
  metrics_list       = "metric1,metric2,metric3"
  description        = "Dashboard with specific metrics"
}

Inputs

Name	Description	Type	Required	Default
title	Dashboard title	string	no	"my dashboard"
api_key	Datadog API key	string	yes	-
app_key	Datadog APP key	string	yes	-
prefix	Metric prefix pattern for discovery	string	yes	-
space_aggregation	Aggregation method (avg, max, min, sum)	string	yes	-
scope	Tag scope for filtering	string	yes	-
description	Dashboard description	string	no	"generated by Terraform"
metrics_list	Comma-separated list of metrics (alternative to prefix)	string	no	""
template_variable_name	Name of template variable	string	no	"variable1"
template_variable_prefix	Prefix of template variable	string	no	""
template_variable_default	Default value for template variable	string	no	"*"

Outputs

Output	Description
url	Dashboard URL (https://app.datadoghq.com/dashboard/{id})
external_metrics	List of metrics discovered from Datadog

External Script

The module uses list_metrics.sh to query the Datadog API:

#!/bin/bash
# list_metrics.sh
# Queries Datadog for metrics matching the prefix pattern

The script is called via the external data source:

data "external" "metrics" {
  program = ["bash", "${path.module}/list_metrics.sh"]
  
  query = {
    api_key = var.api_key
    app_key = var.app_key
    prefix  = var.prefix
  }
}

Dashboard Creation Logic

The module uses conditional resource creation:

• dashboard_a: Created when using metric discovery (prefix-based)
• dashboard_b: Created when using pre-defined metrics list
• Only ONE dashboard is created based on which mode is used

count = var.metrics_list == "" ? 1 : 0  # dashboard_a
count = var.metrics_list != "" ? 1 : 0  # dashboard_b

Template Variables

The dashboard includes a template variable for interactive filtering:

template_variable {
  name    = var.template_variable_name
  prefix  = var.template_variable_prefix
  default = var.template_variable_default
}

Example: Filter by environment, host, service, etc.

Widget Configuration

Each discovered metric gets a timeseries widget:

• Display Type: Line graph
• Aggregation: Configured via space_aggregation variable
• Scope: Filtered by scope variable
• Alias: Metadata expression for cleaner names

request {
  q = "${var.space_aggregation}:${metric.value}{${var.scope}}"
  
  metadata {
    expression = "${var.space_aggregation}:${metric.value}{${var.scope}}"
    alias_name = metric.value
  }
}

Aggregation Methods

Supported aggregation methods:

• avg: Average values
• max: Maximum values
• min: Minimum values
• sum: Sum of values

Scope Filtering

Scope uses Datadog tag syntax:

environment:production                    # Single tag
environment:production,service:api        # Multiple tags (AND)
environment:production OR service:api     # OR logic
environment:prod*                         # Wildcard

Examples

CPU Metrics Dashboard

module "cpu_dashboard" {
  source = "./terraform-datadog-dashboard"

  title             = "CPU Utilization Dashboard"
  prefix            = "system.cpu.*"
  space_aggregation = "avg"
  scope             = "env:production"
  api_key           = var.datadog_api_key
  app_key           = var.datadog_app_key
}

Custom Application Dashboard

module "app_dashboard" {
  source = "./terraform-datadog-dashboard"

  title             = "Application Performance"
  prefix            = "myapp.performance.*"
  space_aggregation = "max"
  scope             = "service:myapp,env:prod"
  
  template_variable_name    = "host"
  template_variable_prefix  = "host"
  template_variable_default = "*"
  
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

Pre-Defined Metrics

module "specific_metrics" {
  source = "./terraform-datadog-dashboard"

  title             = "Key Metrics"
  prefix            = ""  # Not used in this mode
  space_aggregation = "avg"
  scope             = "env:prod"
  metrics_list      = "aws.ec2.cpuutilization,aws.rds.database_connections,custom.metric.value"
  api_key           = var.datadog_api_key
  app_key           = var.datadog_app_key
}

Layout

• Layout Type: Ordered (widgets stacked vertically)
• Widget Type: Timeseries
• Auto-Generated: One widget per discovered metric

Notes

• Requires external bash script (list_metrics.sh) in module directory
• Script must be executable (chmod +x list_metrics.sh)
• Metric discovery happens during Terraform plan/apply
• Dashboard updates automatically when new metrics appear
• Uses external data source which runs on every plan
• Consider caching if metric list is large and changes infrequently

Troubleshooting

Script Execution Issues

Ensure the script is executable:

chmod +x terraform-datadog-dashboard/list_metrics.sh

No Metrics Discovered

• Verify API/APP keys have correct permissions
• Check that metrics matching prefix exist in Datadog
• Confirm metric prefix syntax (e.g., "myapp." not "myapp")

Dashboard Not Created

• Check if metrics_list vs prefix logic is correct
• Verify exactly one creation mode is used (not both)

Best Practices

1. Prefix Naming: Use specific prefixes to limit widget count
2. Aggregation: Choose appropriate aggregation for metric type
3. Scope: Use specific scopes to reduce cardinality
4. Template Variables: Add variables for interactive filtering
5. Testing: Test metric discovery in dev before production

Limitations

• One dashboard per module instance
• Script runs on every plan (can be slow with many metrics)
• Ordered layout only (no custom positioning)
• All widgets are timeseries type
• No support for mixed widget types

Future Enhancements

Potential improvements:

• Widget type customization
• Free layout support
• Widget positioning control
• Query value widgets
• Toplist widgets
• Note widgets for organization

License

Internal use only - Sanoma/WeBuildYourCloud

Authors

Created and maintained by the Platform Engineering team.