Expose kuberhealthy metrics on datadog

Kuberhealthy is an operator that helps in running synthetic checks on a kubernetes cluster.

Exposing these metrics on Prometheus/Grafana or Datadog will help in creating interesting dashboards that in turn could contribute in generating valuable KPIs

Datadog by default does not provide any integration with kuberhealthy to extract the metrics but how do we get them onto our datadog dashboards?

Photo by Mark Fletcher-Brown on Unsplash

It can be done by adding appropriate annotations to the kuberhealthy deployment configuration file

Add the below snippet to your spec metadata


annotations:
ad.datadoghq.com/kuberhealthy.init_configs: '[{}]'
ad.datadoghq.com/kuberhealthy.check_names: '["openmetrics"]'
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
ad.datadoghq.com/kuberhealthy.instances: |
[
{
"prometheus_url": "http://%%host%%:8080/metrics",
"type_overrides":
{
"kuberhealthy_check": "gauge",
"kuberhealthy_check_duration_seconds": "gauge",
"kuberhealthy_cluster_state": "gauge",
"kuberhealthy_running": "gauge"
},
"namespace": "kube-system",
"metrics": ["kuberhealthy_*"],
"leader_election": true,
"bearer_token_auth": true,
"bearer_token_path": "/var/run/secrets/kubernetes.io/serviceaccount/token",
"ssl_verify": false,
"prometheus_timeout": 15
}
]

You will now be able to scrape the kuberhealthy metrics on datadog using prometheus endpoints and create your own dashboards and KPIs

References:

https://docs.datadoghq.com/agent/kubernetes/prometheus/