Create grafana alert that scans log files for specific words

LGTM
1

Hello everyone,

I’ve spent hours googling and watching YouTube videos and unfortunately haven’t found a solution. I have to say that I am a total beginner.

I’m logging a txt file with Loki and would like to set an alert that reports when a certain word occurs in the file, for example Error. Can someone help me with how exactly I have to set this up? Many thanks in advance

2

Hello @mdavidallen,

Setting up an alert in Grafana to notify you when a specific word, like "Error," appears in your log files is a great way to keep track of issues. Here's a step-by-step guide to help you set this up:

  1. Ensure Loki is Configured: Make sure your Loki data source is properly configured in Grafana. You should be able to see your logs in the Grafana Explore section.

  2. Create a New Alert Rule:

  3. Navigate to Alerting > Alert rules in Grafana.

  4. Click on New alert rule.

  5. Define the Query:

  6. Select the Loki data source.
  7. Switch to Code mode in the query editor.

  8. Enter a query to filter logs containing the word "Error". For example: logql {filename="/path/to/your/logfile.txt"} |= "Error"

  9. Set the Alert Condition:

  10. In the Alert condition section, set the condition to trigger when the count of logs with "Error" is greater than 0.

  11. Use a function like count_over_time to count occurrences over a specific time window, e.g., [5m] for the last 5 minutes: logql count_over_time({filename="/path/to/your/logfile.txt"} |= "Error" [5m]) > 0

  12. Configure Notifications:

  13. Choose a contact point where you want to receive notifications, such as email or a webhook.

  14. Set up the notification channel under Contact points.

  15. Save and Test:

  16. Click Save rule and exit.
  17. Test the alert by generating a log entry with the word "Error" and see if the alert triggers.

Remember, if you have any issues, you can always check the Grafana logs for more information. And if you ever need to make a tissue dance, just put a little boogie in it! 🎶

xoxo Grot

kapa-discourse