Telegraf: Dynamic Tags (global tags)

This is not really a smart solution. This means if you can collect this tag informations different, write this in another measurement and join this later, then I would recommend it this way.
Global tags can be specified in the [global_tags] table in key="value" format. All metrics that are gathered will be tagged with the tags specified.
With dynamic tags we want the use global_tags in a more flexible way. In short: The tags should be updated dynamically.

Explanation about the workflow

We want to add the ClusterName (Hyper-V Cluster) as a global tag, because we want to have the opportunity to filter/group by the ClusterName.
Because Telegraf have no built-in feature to run commands to define global_tags, we write the the tag "ClusterName" to the Windows Telegraf service environment variables (Registry-Key).
It would also be possible to write the tag to the Windows system environment variables. But it is not recommended.

So we have three important files:

• tags.global.conf: all global tags are defined in this Telegraf configuration file.
• inputs.exec.set-globaltags.conf: this runs only the PowerShell script without any output.
• tag.set-globaltags.ps1: PowerShell script that gets information about the ClusterName and put it into a RegKey as environment variables for the service "Telegraf".
  

Workflow

1. The Telegraf Agent reads inputs.exec.set-globaltags.conf
2. and runs every 4h the PowerShell script tag.set-globaltags.ps1 because of the defined "interval".
3. The PowerShell script tag.set-globaltags.ps1 runs a command to get the ClusterName and create/update the RegKey "Environment" with the ClusterName.
4. The PowerShell script restart the service "Telegraf".
5. Because of the restart of the Telegraf service, Telegraf reads the global_tags defined in the file "tags.global.conf".
6. In the tags.global.conf there is a tag named "Cluster" defined. This tag has the value "$ClusterName" (this is an Windows service environment variable). Telegraf searches the environment variable "ClusterName" in the environment variable of the service.
7. Telegraf find the the environment variable "ClusterName" and add a tag named Cluster with the value of the ClusterName to all metrics that are send to the InfluxDB.

Telegraf global_tags configuration file "tags.global.conf"

 # Global tags can be specified here in key="value" format.
 [global_tags]
   # dc = "us-east-1" # will tag all metrics with dc=us-east-1
   # rack = "1a"
   ## Environment variables can be used as tags, and throughout the config file
   # user = "$USER"
   Cluster = "$ClusterName" 

Telegraf exec plugin file "inputs.exec.set-globaltags.conf"

 [[inputs.exec]]
 commands = ['powershell -NoProfile -File "C:\Program Files\Telegraf\scripts\tag.set-globaltags.ps1"']
 interval = "4h"
 timeout = "1m" 

PowerShell "tag.set-globaltags.ps1" script that gets the ClusterName

#Variable: ClusteName
$ClusterName = (Get-Cluster -ErrorAction SilentlyContinue).Name 
if ($null -eq $ClusterName) { 
    $ClusterName = "NotInCluster"
}
 
# Generate an array of the variables for the RegKey
$RegKeyValue =@(
"ClusterName=$ClusterName",
)
 
# Create or update the RegKey "Environment" with the Variables
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\telegraf" -Name "Environment" -Value $RegKeyValue -PropertyType "MultiString" -Force | Out-Null
 
# Restart the Telegrafg agent to apply the configuration changes
Restart-Service -Name Telegraf

Draw.io file: Datei:Drawio - Telegraf-Dynamic Tags.zip
More information's: https://www.influxdata.com/blog/using-telegraf-on-windows/