Skip to content

Kafka Monitoring Alert Configuration

Overview

This document outlines the configuration of alerts set up in Grafana for monitoring the health and performance of a Kafka cluster, Kafka Schema Registry and Connectors. The alerts are categorized into groups based on their monitoring focus.

Alert Groups

1. Group: Cluster Status

  • Name: Cluster Status
  • Folder: Cluster Metrics
  • Evaluation Interval: 6 hours

1.1 Alert Rule: Kafka Server Status

  • UID: cesb8dnkmp2pse
  • Title: Kafka Server Status
  • Condition: C
Alert Queries
  • Query A:
    • Expression: avg(confluent_kafka_server_connection_info{kafka_id="lkc-qj83r7"})
    • Purpose: Calculates the average number of connections to the Kafka broker identified by lkc-qj83r7.
  • Query B:
    • Purpose: Checks if the average connections are greater than 0.
  • Query C:
    • Purpose: Triggers an alert if the average connections are less than 1, indicating no active connections.
  • Query D:
    • Purpose: Monitors if the average connections exceed 80, indicating a potential overload.
Alert Properties
  • For Duration: 6 hours
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

2. Group: Cluster Load

  • Name: Cluster Load
  • Folder: Cluster Metrics
  • Evaluation Interval: 30 minutes

2.1 Alert Rule: kafka_server_cluster_load_percent_lkc-qj83r7

  • UID: bes2wa661xhj4e
  • Title: kafka_server_cluster_load_percent_lkc-qj83r7
Alert Queries
  • Query A:
    • Expression: confluent_kafka_server_cluster_load_percent{kafka_id="lkc-qj83r7"}
    • Purpose: Monitors the load on the Kafka broker identified by lkc-qj83r7.
  • Query B:
    • Purpose: Evaluates if the load exceeds 70% to trigger alerts.
Alert Properties
  • For Duration: 30 minutes
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

2.2 Alert Rule: kafka_server_cluster_load_percent_lkc-gwk8nr

  • UID: bes2wbqbb9m9sd
  • Title: kafka_server_cluster_load_percent_lkc-gwk8nr
Alert Queries
  • Query A:
    • Expression: confluent_kafka_server_cluster_load_percent{kafka_id="lkc-gwk8nr"}
    • Purpose: Monitors the load on the Kafka broker identified by lkc-gwk8nr.
  • Query B:
    • Purpose: Evaluates if the load exceeds 70% to trigger alerts.
Alert Properties
  • For Duration: 30 minutes
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

2.3 Alert Rule: kafka_server_cluster_load_percent_lkc-x7r3og

  • UID: bes2wdap19d6oe
  • Title: kafka_server_cluster_load_percent_lkc-x7r3og
Alert Queries
  • Query A:
    • Expression: confluent_kafka_server_cluster_load_percent{kafka_id="lkc-x7r3og"}
    • Purpose: Monitors the load on the Kafka broker identified by lkc-x7r3og.
  • Query B:
    • Purpose: Evaluates if the load exceeds 70% to trigger alerts.
Alert Properties
  • For Duration: 30 minutes
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

2.4 Alert Rule: kafka_server_cluster_load_percent_lkc-w1zpzj

  • UID: ces2weq7edq80b
  • Title: kafka_server_cluster_load_percent_lkc-w1zpzj
Alert Queries
  • Query A:
    • Expression: confluent_kafka_server_cluster_load_percent{kafka_id="lkc-w1zpzj"}
    • Purpose: Monitors the load on the Kafka broker identified by lkc-w1zpzj.
  • Query B:
    • Purpose: Evaluates if the load exceeds 70% to trigger alerts.
Alert Properties
  • For Duration: 30 minutes
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

3. Group: Connector Status

  • Name: Connector Status
  • Folder: Cluster Metrics
  • Evaluation Interval: 30 minutes

3.1 Alert Rule: Connector Status

  • UID: eesba05hggnpcf
  • Title: Connector Status
Alert Queries
  • Query A:
    • Expression: confluent_kafka_connect_connector_status{connector_id="lcc-ro82kp"}
    • Purpose: Monitors the operational status of the Kafka connector.
  • Query B:
    • Purpose: Checks if the average connections are greater than 0.
  • Query C:
    • Purpose: Triggers an alert if the connector status is not equal to 1, indicating the connector is down.
Alert Properties
  • For Duration: 30 minutes
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

4. Group: Consumer Lag

  • Name: Consumer Lag
  • Folder: Cluster Metrics
  • Evaluation Interval: 1 hour

4.1 Alert Rule: consumer_lag_offset_lkc-gwk8nr

  • UID: ces2x4ppjvtvkc
  • Title: consumer_lag_offset_lkc-gwk8nr
Alert Queries
  • Query A:
    • Expression: sum by(consumer_group_id) (confluent_kafka_server_consumer_lag_offsets{kafka_id="lkc-gwk8nr"})
    • Purpose: Calculates the sum of consumer lag offsets.
  • Query B:
    • Purpose: Triggers an alert if the consumer lag exceeds 100.
Alert Properties
  • For Duration: 1 hour
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

4.2 Alert Rule: consumer_lag_offset_lkc-qj83r7

  • UID: des2y8lhzs1z4f
  • Title: consumer_lag_offset_lkc-qj83r7
Alert Queries
  • Query A:
    • Expression: sum by(consumer_group_id) (confluent_kafka_server_consumer_lag_offsets{kafka_id="lkc-qj83r7"})
    • Purpose: Calculates the sum of consumer lag offsets.
  • Query B:
    • Purpose: Triggers an alert if the consumer lag exceeds 100.
Alert Properties
  • For Duration: 1 hour
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

4.3 Alert Rule: consumer_lag_offset_lkc-x7r3og

  • UID: ces2yamp8o5xcc
  • Title: consumer_lag_offset_lkc-x7r3og
Alert Queries
  • Query A:
    • Expression: sum by(consumer_group_id) (confluent_kafka_server_consumer_lag_offsets{kafka_id="lkc-x7r3og"})
    • Purpose: Calculates the sum of consumer lag offsets.
  • Query B:
    • Purpose: Triggers an alert if the consumer lag exceeds 100.
Alert Properties
  • For Duration: 1 hour
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

4.4 Alert Rule: consumer_lag_offset_lkc-w1zpzj

  • UID: fes2yc2pn24u8c
  • Title: consumer_lag_offset_lkc-w1zpzj
Alert Queries
  • Query A:
    • Expression: sum by(consumer_group_id) (confluent_kafka_server_consumer_lag_offsets{kafka_id="lkc-w1zpzj"})
    • Purpose: Calculates the sum of consumer lag offsets.
  • Query B:
    • Purpose: Triggers an alert if the consumer lag exceeds 100.
Alert Properties
  • For Duration: 1 hour
  • Notification Settings: Alerts are sent to "Kaas Ops Channel".

Notification Alert setup

All the generated notification alerts are sent to Microsoft Teams channel called KaaS Platform Teams channel. After sending the notification on the channel, the tasks are created in the planner. The process and notification setup is described in the wiki page: Microsoft-Teams-Alert-Workflow - Repos

KaaS-Confluent-Metrics-Alert-Rules

Conclusion

These alert configurations ensure comprehensive monitoring of Kafka's operational health, focusing on connection status, cluster load, connector status, and consumer lag. Alerts are sent to predefined channels, allowing for prompt responses to issues within the Kafka environment.