Skip to content

Support Log Search Alerts #1168

Open

Description

@Ryan-Palmer

In the LogAnalytics docs, if you set a daily usage cap it is recommended to also set an alert if you hit that cap.

They provide an example log search alert configuration here.

The portal set up is more or less just this:
image

It would be great to deploy this alert with Farmer (plus set up alerts on any other custom queries we might need).

If I can work out what to do, I'm happy to have a go at adding it :)

Activity

  1. changed the title [-]`Query` property unavailable for custom log alerts[/-] [+]`Query` property for custom log alerts[/+] on Dec 5, 2024
  2. changed the title [-]`Query` property for custom log alerts[/-] [+]Support Log Search Alerts[/+] on Dec 9, 2024
  3. Ryan-Palmer commented on Dec 9, 2024

    @Ryan-Palmer
    ContributorAuthor

    Ah, looking at the ARM docs for log search alerts I just realised that the alert builder in Farmer is actually building a Microsoft.Insights/metricAlerts resource, but log search alerts use the Microsoft.Insights/scheduledQueryRules resource which I don't think exists in Farmer yet. Just picking through the docs / source etc now trying to work out what is going on.

    Metric alert (just 'alert' in Farmer) :

    {
          "type": "Microsoft.Insights/metricAlerts",
          "apiVersion": "2018-03-01",
          "name": "[parameters('alertName')]",
          "location": "global",
          "properties": {
            "description": "[parameters('alertDescription')]",
            "severity": "[parameters('alertSeverity')]",
            "enabled": "[parameters('isEnabled')]",
            "scopes": [
              "[parameters('resourceId')]"
            ],
            "evaluationFrequency": "[parameters('evaluationFrequency')]",
            "windowSize": "[parameters('windowSize')]",
            "criteria": {
              "odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria",
              "allOf": [
                {
                  "name": "1st criterion",
                  "metricName": "[parameters('metricName')]",
                  "dimensions": [],
                  "operator": "[parameters('operator')]",
                  "threshold": "[parameters('threshold')]",
                  "timeAggregation": "[parameters('timeAggregation')]",
                  "criterionType": "StaticThresholdCriterion"
                }
              ]
            },
            "actions": [
              {
                "actionGroupId": "[parameters('actionGroupId')]"
              }
            ]
          }
        }

    Log alert (scheduledQueryRule, not in Farmer afaik):

    {
          "type": "Microsoft.Insights/scheduledQueryRules",
          "apiVersion": "2021-08-01",
          "name": "[parameters('alertName')]",
          "location": "[parameters('location')]",
          "tags": {},
          "properties": {
            "description": "[parameters('alertDescription')]",
            "severity": "[parameters('alertSeverity')]",
            "enabled": "[parameters('isEnabled')]",
            "scopes": [
              "[parameters('resourceId')]"
            ],
            "evaluationFrequency": "[parameters('evaluationFrequency')]",
            "windowSize": "[parameters('windowSize')]",
            "criteria": {
              "allOf": [
                {
                  "query": "[parameters('query')]",
                  "metricMeasureColumn": "[parameters('metricMeasureColumn')]",
                  "resourceIdColumn": "[parameters('resourceIdColumn')]",
                  "dimensions": [],
                  "operator": "[parameters('operator')]",
                  "threshold": "[parameters('threshold')]",
                  "timeAggregation": "[parameters('timeAggregation')]",
                  "failingPeriods": {
                    "numberOfEvaluationPeriods": "[parameters('numberOfEvaluationPeriods')]",
                    "minFailingPeriodsToAlert": "[parameters('minFailingPeriodsToAlert')]"
                  }
                }
              ]
            },
            "muteActionsDuration": "[parameters('muteActionsDuration')]",
            "autoMitigate": "[parameters('autoMitigate')]",
            "checkWorkspaceAlertsStorageConfigured": "[parameters('checkWorkspaceAlertsStorageConfigured')]",
            "actions": {
              "actionGroups": [
                "[parameters('actionGroupId')]"
              ],
              "customProperties": {
                "key1": "value1",
                "key2": "value2"
              }
            }
          }
        }
  4. Ryan-Palmer commented on Dec 9, 2024

    @Ryan-Palmer
    ContributorAuthor

    Did the quick hack test, got it working with the JSON builder

    let scheduledQueryRule
        alertName
        location
        alertDescription
        alertSeverity
        isEnabled
        resourceId
        evaluationFrequency
        windowSize
        query
        metricMeasureColumn
        resourceIdColumn
        operator
        threshold
        timeAggregation
        numberOfEvaluationPeriods
        minFailingPeriodsToAlert
        muteActionsDuration
        autoMitigate
        checkWorkspaceAlertsStorageConfigured
        actionGroupId =
        $"""{{
            "type": "Microsoft.Insights/scheduledQueryRules",
            "apiVersion": "2021-08-01",
            "name": "{alertName}",
            "location": "{location}",
            "tags": {{}},
            "properties": {{
                "description": "{alertDescription}",
                "severity": "{alertSeverity}",
                "enabled": "{isEnabled}",
                "scopes": [
                    "{resourceId}"
                ],
                "evaluationFrequency": "{evaluationFrequency}",
                "windowSize": "{windowSize}",
                "criteria": {{
                    "allOf": [
                        {{
                            "query": "{query}",
                            "metricMeasureColumn": "{metricMeasureColumn |> Option.defaultValue ""}",
                            "resourceIdColumn": "{resourceIdColumn}",
                            "dimensions": [],
                            "operator": "{operator}",
                            "threshold": "{threshold}",
                            "timeAggregation": "{timeAggregation}",
                            "failingPeriods": {{
                                "numberOfEvaluationPeriods": "{numberOfEvaluationPeriods}",
                                "minFailingPeriodsToAlert": "{minFailingPeriodsToAlert}"
                            }}
                        }}
                    ]
                }},
                "muteActionsDuration": "{muteActionsDuration}",
                "autoMitigate": "{autoMitigate}",
                "checkWorkspaceAlertsStorageConfigured": "{checkWorkspaceAlertsStorageConfigured}",
                "actions": {{
                    "actionGroups": [
                        "{actionGroupId}"
                    ],
                    "customProperties": {{
                    }}
                }}
            }}
        }}"""
        |> Resource.ofJson
    
    let logDataAlert =
            scheduledQueryRule
                "Daily log data limit reached"
                "UKSouth"
                "Notify admins if log data limit reached"
                2
                "true"
                $"/subscriptions/{subId}/resourceGroups/{resGroupName}/providers/Microsoft.OperationalInsights/workspaces/{logAnalyticsName}"
                "PT5M"
                "PT5M"
                @"_LogOperation | where Category =~ 'Ingestion' | where Detail contains 'OverQuota'"
                None
                "_ResourceId"
                "GreaterThan"
                0
                "Count"
                1
                1
                "PT5M"
                "false"
                "false"
                alertAction.ActionGroupId
    
  5. Ryan-Palmer commented on Dec 17, 2024

    @Ryan-Palmer
    ContributorAuthor

    Here's a complete working sample.

    It has Serilog structured logging, OTel metric and Traces plus the search query alert for data limits.

    I will hopefully find time to make a proper Farmer PR rather than using the JSON, I just need to re-familiarise myself with the code, it's been a while!

    https://github.com/Ryan-Palmer/Safe-Serilog-Farmer-Demo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions