terraform-datadog-dashboard/README.md

# Terraform Datadog Dashboard Module

## Overview

This Terraform module creates dynamic Datadog dashboards from metrics discovered in Datadog, with flexible filtering and aggregation options. It uses an external script to query available metrics and automatically generates dashboard widgets.

## Features

- **Dynamic Metric Discovery**: Queries Datadog API for available metrics by prefix
- **Auto-Generated Widgets**: Creates timeseries widgets automatically from metrics
- **Template Variables**: Interactive filtering via dashboard variables
- **Dual Creation Modes**: Metric discovery vs. pre-defined metric list
- **Metric Aliasing**: Metadata expressions for cleaner metric names

## Resources Created

- `datadog_dashboard`: Dynamically generated dashboard with timeseries widgets

## Requirements

| Name | Version |
|------|---------|
| terraform | >= 0.12 |
| datadog | >= 2.0 |
| bash | For list_metrics.sh script |

## Usage

### Mode 1: Metric Discovery by Prefix

```hcl
module "dynamic_dashboard" {
  source = "./terraform-datadog-dashboard"

  title              = "Application Metrics Dashboard"
  api_key            = var.datadog_api_key
  app_key            = var.datadog_app_key
  prefix             = "myapp.*"
  space_aggregation  = "avg"
  scope              = "environment:production,service:api"
  description        = "Auto-generated dashboard for application metrics"
}
```

### Mode 2: Pre-Defined Metrics List

```hcl
module "custom_dashboard" {
  source = "./terraform-datadog-dashboard"

  title              = "Custom Metrics Dashboard"
  api_key            = var.datadog_api_key
  app_key            = var.datadog_app_key
  prefix             = ""
  space_aggregation  = "sum"
  scope              = "env:prod"
  metrics_list       = "metric1,metric2,metric3"
  description        = "Dashboard with specific metrics"
}
```

## Inputs

| Name | Description | Type | Required | Default |
|------|-------------|------|----------|---------|
| `title` | Dashboard title | `string` | no | `"my dashboard"` |
| `api_key` | Datadog API key | `string` | yes | - |
| `app_key` | Datadog APP key | `string` | yes | - |
| `prefix` | Metric prefix pattern for discovery | `string` | yes | - |
| `space_aggregation` | Aggregation method (avg, max, min, sum) | `string` | yes | - |
| `scope` | Tag scope for filtering | `string` | yes | - |
| `description` | Dashboard description | `string` | no | `"generated by Terraform"` |
| `metrics_list` | Comma-separated list of metrics (alternative to prefix) | `string` | no | `""` |
| `template_variable_name` | Name of template variable | `string` | no | `"variable1"` |
| `template_variable_prefix` | Prefix of template variable | `string` | no | `""` |
| `template_variable_default` | Default value for template variable | `string` | no | `"*"` |

## Outputs

| Output | Description |
|--------|-------------|
| `url` | Dashboard URL (https://app.datadoghq.com/dashboard/{id}) |
| `external_metrics` | List of metrics discovered from Datadog |

## External Script

The module uses `list_metrics.sh` to query the Datadog API:

```bash
#!/bin/bash
# list_metrics.sh
# Queries Datadog for metrics matching the prefix pattern
```

The script is called via the `external` data source:
```hcl
data "external" "metrics" {
  program = ["bash", "${path.module}/list_metrics.sh"]
  
  query = {
    api_key = var.api_key
    app_key = var.app_key
    prefix  = var.prefix
  }
}
```

## Dashboard Creation Logic

The module uses conditional resource creation:

- **dashboard_a**: Created when using metric discovery (prefix-based)
- **dashboard_b**: Created when using pre-defined metrics list
- Only ONE dashboard is created based on which mode is used

```hcl
count = var.metrics_list == "" ? 1 : 0  # dashboard_a
count = var.metrics_list != "" ? 1 : 0  # dashboard_b
```

## Template Variables

The dashboard includes a template variable for interactive filtering:

```hcl
template_variable {
  name    = var.template_variable_name
  prefix  = var.template_variable_prefix
  default = var.template_variable_default
}
```

Example: Filter by environment, host, service, etc.

## Widget Configuration

Each discovered metric gets a timeseries widget:
- **Display Type**: Line graph
- **Aggregation**: Configured via `space_aggregation` variable
- **Scope**: Filtered by `scope` variable
- **Alias**: Metadata expression for cleaner names

```hcl
request {
  q = "${var.space_aggregation}:${metric.value}{${var.scope}}"
  
  metadata {
    expression = "${var.space_aggregation}:${metric.value}{${var.scope}}"
    alias_name = metric.value
  }
}
```

## Aggregation Methods

Supported aggregation methods:
- `avg`: Average values
- `max`: Maximum values
- `min`: Minimum values
- `sum`: Sum of values

## Scope Filtering

Scope uses Datadog tag syntax:
```
environment:production                    # Single tag
environment:production,service:api        # Multiple tags (AND)
environment:production OR service:api     # OR logic
environment:prod*                         # Wildcard
```

## Examples

### CPU Metrics Dashboard

```hcl
module "cpu_dashboard" {
  source = "./terraform-datadog-dashboard"

  title             = "CPU Utilization Dashboard"
  prefix            = "system.cpu.*"
  space_aggregation = "avg"
  scope             = "env:production"
  api_key           = var.datadog_api_key
  app_key           = var.datadog_app_key
}
```

### Custom Application Dashboard

```hcl
module "app_dashboard" {
  source = "./terraform-datadog-dashboard"

  title             = "Application Performance"
  prefix            = "myapp.performance.*"
  space_aggregation = "max"
  scope             = "service:myapp,env:prod"
  
  template_variable_name    = "host"
  template_variable_prefix  = "host"
  template_variable_default = "*"
  
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}
```

### Pre-Defined Metrics

```hcl
module "specific_metrics" {
  source = "./terraform-datadog-dashboard"

  title             = "Key Metrics"
  prefix            = ""  # Not used in this mode
  space_aggregation = "avg"
  scope             = "env:prod"
  metrics_list      = "aws.ec2.cpuutilization,aws.rds.database_connections,custom.metric.value"
  api_key           = var.datadog_api_key
  app_key           = var.datadog_app_key
}
```

## Layout

- **Layout Type**: Ordered (widgets stacked vertically)
- **Widget Type**: Timeseries
- **Auto-Generated**: One widget per discovered metric

## Notes

- Requires external bash script (`list_metrics.sh`) in module directory
- Script must be executable (`chmod +x list_metrics.sh`)
- Metric discovery happens during Terraform plan/apply
- Dashboard updates automatically when new metrics appear
- Uses `external` data source which runs on every plan
- Consider caching if metric list is large and changes infrequently

## Troubleshooting

### Script Execution Issues

Ensure the script is executable:
```bash
chmod +x terraform-datadog-dashboard/list_metrics.sh
```

### No Metrics Discovered

- Verify API/APP keys have correct permissions
- Check that metrics matching prefix exist in Datadog
- Confirm metric prefix syntax (e.g., "myapp.*" not "myapp*")

### Dashboard Not Created

- Check if metrics_list vs prefix logic is correct
- Verify exactly one creation mode is used (not both)

## Best Practices

1. **Prefix Naming**: Use specific prefixes to limit widget count
2. **Aggregation**: Choose appropriate aggregation for metric type
3. **Scope**: Use specific scopes to reduce cardinality
4. **Template Variables**: Add variables for interactive filtering
5. **Testing**: Test metric discovery in dev before production

## Limitations

- One dashboard per module instance
- Script runs on every plan (can be slow with many metrics)
- Ordered layout only (no custom positioning)
- All widgets are timeseries type
- No support for mixed widget types

## Future Enhancements

Potential improvements:
- Widget type customization
- Free layout support
- Widget positioning control
- Query value widgets
- Toplist widgets
- Note widgets for organization

## License

Internal use only - Sanoma/WeBuildYourCloud

## Authors

Created and maintained by the Platform Engineering team.