Grafana alloy - dropping metrics

Ask AI
1

Hi,

I’m new to Alloy and have been following this guide to set up monitoring of a Linux host into Grafana cloud. I wanted to drop all metrics related to node_disk_*, (e.g. node_disk_read_time_seconds_total) so I modified the /etc/alloy/config.alloy file with an additonal rule (top one, bottom one is from the documentation) as shown:

prometheus.relabel "alloy_check" {
  forward_to = [prometheus.remote_write.metrics_service.receiver]

  rule {
    action        = "drop"
    source_labels = ["__name__"]
    regex         = "(node_disk_.*)"
  }
  rule {
    source_labels = ["__name__"]
    regex         = "(prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|alloy_build.*|prometheus_remote_write_wal_samples_appended_total|process_start_time_seconds)"
    action        = "keep"
  }
}

The config reloads OK but it doesn’t drop anything. I’m guessing that I’m looking in the wrong place here, can anyone point me in the right direction?

Here is my (lightly redacted) config file, for reference: alloy.config - Pastebin.com

Thanks!

2

Hi @mdavidallen,

It looks like you're on the right track with using the prometheus.relabel component to drop metrics. However, there are a few things you might want to check or adjust:

  1. Order of Rules: The order of relabeling rules is important. In your configuration, the drop rule is placed before the keep rule. This means that any metric matching the drop rule will be dropped before it can be considered by the keep rule. If you want to ensure that only specific metrics are kept, you might want to reverse the order or adjust the logic to ensure the desired metrics are retained.

  2. Regex Pattern: Ensure that your regex pattern is correctly matching the metrics you intend to drop. The pattern (node_disk_.*) should match any metric starting with node_disk_, but double-check that there are no typos or unexpected characters in the metric names.

  3. Reloading Configuration: After making changes to your configuration, ensure that the Alloy service is properly reloading the configuration. You can do this by using the appropriate command or API call to reload the configuration without restarting the service.

  4. Debugging: Use the Live Debugging feature in Alloy to see which metrics are being processed and how the relabeling rules are applied. This can help you verify if the rules are working as expected.

  5. Check Logs: Look at the Alloy logs for any errors or warnings that might indicate issues with the configuration or rule processing.

If you've checked these and the issue persists, it might be helpful to share more details from your configuration or any error messages you encounter. This can provide more context for troubleshooting.

Let me know if this helps or if you have further questions!

xoxo Grot

grot is still learning and improving, please let me know how I did by reacting below.

👍🏽 or 👎🏽