Telegraf: Dynamic Tags (global tags)

Global tags can be specified in the [global_tags] table in key="value" format. All metrics that are gathered will be tagged with the tags specified.
With dynamic tags we want the use global_tags in a more flexible way. In short: The tags should be updated dynamically.

Explanation about the workflow

We want to add the ClusterName (Hyper-V Cluster) as a global tag, because we want to filter/group by the ClusterName.

So we have three important files:

• tags.global.conf: all global tags are defined in this Telegraf configuration file.
• inputs.exec.set-globaltags.conf: this runs only the PowerShell script without any output.
• tag.set-globaltags.ps1: PowerShell script that gets information about the ClusterName and put it into a RegKey as environment variables for the service "Telegraf".
  

Workflow

1. The Telegraf Agent runs the PowerShell script tag.set-globaltags.ps1 every 4h (because of the defined "intervall" in inputs.exec.set-globaltags.conf).
2. The PowerShell script tag.set-globaltags.ps1 runs a command to get the ClusterName, create/update the RegKey "Environment" with the ClusterName and restart the service "Telegraf".
3. Because of the restart of the Telegraf service, Telegraf reads the global_tags defined in the file "tags.global.conf".
4. In the tags.global.conf there is a tag defined named "Cluster". This tag has the value "$ClusterName" (this is an environment variable).
5. Telegraf searches the environment variable "ClusterName" in the environment variable of the service (this was created/updated by the PowerShell script before).
6. Telegraf find the the environment variable "ClusterName" and add a tag named Cluster with the ClusterNames as value to all metrics.

Telegraf global_tags configuration file "tags.global.conf"

 # Global tags can be specified here in key="value" format.
 [global_tags]
   # dc = "us-east-1" # will tag all metrics with dc=us-east-1
   # rack = "1a"
   ## Environment variables can be used as tags, and throughout the config file
   # user = "$USER"
   Cluster = "$ClusterName" 

Telegraf exec plugin file "inputs.exec.set-globaltags.conf"

 [[inputs.exec]]
 commands = ['powershell -NoProfile -File "C:\Program Files\Telegraf\scripts\tag.set-globaltags.ps1"']
 interval = "4h"
 timeout = "1m" 

PowerShell "tag.set-globaltags.ps1" script that gets the ClusterName

### region Variables
#Variable: ClusteName
$ClusterName = (Get-Cluster -ErrorAction SilentlyContinue).Name 
### endregion Variables
 
 
# Generate an array of the variables for the RegKey
$RegKeyValue =@(
"ClusterName=$ClusterName",
)
 
# Create or update the RegKey "Environment" with the Variables
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\telegraf" -Name "Environment" -Value $RegKeyValue -PropertyType "MultiString" -Force | Out-Null
 
# Restart the Telegrafg agent to apply the configuration changes
Restart-Service -Name Telegraf