Monitoring Kubernetes with Prometheus and Grafana

  • |
  • 12 November 2022
Post image

After experimenting with many different approaches and manually creating lots of prometheus jobs and grafana dashboards, I came across the kube-prometheus-stack helm chart from the prometheus-community GitHub repository (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack). That chart comes with all the components I was looking for, specially because it covers almost everything I had previously and a few more metrics. On top of that, it comes with preconfigured alertmanager rules for all of the components, which is very handy.

It is possible to select only the components you require by overriding the values.yaml file from the chart and tailor it to better suit your needs. In my case, I disabled the Grafana component since I already had one deployed on the environment and it was being used by other team members.

I’m usually not a big fan of helm charts, because altough they make it easy to deploy many components and have a lifecycle management of a helm release, if you need to change some parameter not covered by the template you will have to fork the chart and modify it locally. Another situation is when you integrate with ArgoCD. You can create an application object and call a helm chart on it, but the behavior is a bit different for drift detection and remediation when ArgoCD is dealing with the kubernetes object themselves directly, and in some cases if you update the chart, ArgoCD will end up deleting and recreating everything, or you might to manually trigger a redeployment because ArgoCD won’t detect a change on the chart automatically.

So usually what I do is: first try and adjust everything until I have a decent architecture for whatever I’m deploying, and then proceed to create my own set of manifests without the use of helm.

As is mentioned before, I created an override-values.yaml file with some customizations. I enabled the creation of ingress objects and set a pvc so I could have persistence. I also configured the etcd parameters since I was targeting a Kubernetes cluster with etcd deployed as a service directly on the hosts, and not as pods. that is the default Kubespray behavior, which was my scenario. And finally I disabled Grafana.

  1. Generate a new file called override-values.yaml:
alertmanager:
  ingress:
    enabled: true
    ingressClassName: haproxy
    hosts:
       - alertmanager.somedomain.br
    tls:
    - hosts:
      - alertmanager.somedomain.br
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: ssd-hc
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

prometheus:
  ingress:
    enabled: true
    ingressClassName: haproxy
    hosts:
       - prometheus.somedomain.br
    tls:
    - hosts:
      - prometheus.somedomain.br
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: ssd-hc
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi
    secrets: ["etcd-client-cert"]
    additionalScrapeConfigs:

kubeEtcd:
  enabled: true
  endpoints:
    - 10.1.2.11
    - 10.1.2.12
    - 10.1.2.13
  service:
    enabled: true
    port: 2379
    targetPort: 2379
  serviceMonitor:
    scheme: https
    insecureSkipVerify: false
    caFile: /etc/prometheus/secrets/etcd-client-cert/etcd-ca.pem
    certFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client-prometheus.crt
    keyFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client-prometheus.key

grafana:
  enabled: false

I still have to push the docker images to our private registry and change that on the file (todo).

An extra step is required so Prometheus can access the etcd metrics, since it has SSL enabled. In order to do that, I got the etcd ca certificate and key from a master node (they are located at /etc/ssl/etcd/ssl) and copied them to my working directory. Then i used cfssl to easily generate the client certificate to prometheus. I won’t cover installing cfssl, but you can find instructions at https://github.com/cloudflare/cfssl.

  1. Generate the json templates:
cfssl print-defaults csr > prometheus.json
cfssl print-defaults config > ca-config.json
  1. Customize ca-config.json:
{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
  1. Customize prometheus.json:
{
    "CN": "prometheus.somedomain.br",
    "hosts": [""],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "BR",
            "L": "PN",
            "ST": "Bacheland"
        }
    ]
}
  1. Generate and rename the certificate:
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=client prometheus.json | cfssljson -bare prometheus
mv etcd-client-prometheus.pem etcd-client-prometheus.crt
  1. Create the secret:
kubectl -n monitoring create secret generic etcd-client-cert --from-file=./etcd-ca.pem --from-file=./etcd-client-prometheus.crt --from-file=./etcd-client-prometheus.key

In order to collect kube-proxy metrics, it is necessary to change its ConfigMap and modify the metricsBindAddress parameter to make it listen on all interfaces.

  1. Edit kube-proxy ConfigMap and restart the pods:
kubectl -n kube-system edit cm kube-proxy
  # metricsBindAddress: 127.0.0.1:10249
  metricsBindAddress: 0.0.0.0:10249

kubectl -n kube-system delete pod kube-proxy-<id> # delete all pods
  1. Run helm install:
helm install --create-namespace -n monitoring kube-prometheus-stack prometheus-community/kube-prometheus-stack -f override-values.yaml
  1. Check the deployment:
$ kubectl -n monitoring get all
NAME                                                          READY   STATUS    RESTARTS        AGE
pod/alertmanager-kube-prometheus-stack-alertmanager-0         2/2     Running   1 (2d17h ago)   2d17h
pod/kube-prometheus-stack-kube-state-metrics-c7bbf884-428wq   1/1     Running   0               2d17h
pod/kube-prometheus-stack-operator-57f5866866-4l9kt           1/1     Running   0               2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-49wmv      1/1     Running   0               2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-9cvx4      1/1     Running   0               2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-llr9z      1/1     Running   0               2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-qdl74      1/1     Running   0               2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-t5n4x      1/1     Running   0               2d17h
pod/prometheus-kube-prometheus-stack-prometheus-0             2/2     Running   0               2d16h

NAME                                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated                            ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   2d17h
service/kube-prometheus-stack-alertmanager               ClusterIP   192.168.7.85     <none>        9093/TCP                     2d17h
service/kube-prometheus-stack-kube-state-metrics         ClusterIP   192.168.37.114   <none>        8080/TCP                     2d17h
service/kube-prometheus-stack-operator                   ClusterIP   192.168.49.194   <none>        443/TCP                      2d17h
service/kube-prometheus-stack-prometheus                 ClusterIP   192.168.10.77    <none>        9090/TCP                     2d17h
service/kube-prometheus-stack-prometheus-node-exporter   ClusterIP   192.168.39.150   <none>        9100/TCP                     2d17h
service/prometheus-operated                              ClusterIP   None             <none>        9090/TCP                     2d17h

NAME                                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/kube-prometheus-stack-prometheus-node-exporter   5         5         5       5            5           <none>          2d17h

NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-prometheus-stack-kube-state-metrics   1/1     1            1           2d17h
deployment.apps/kube-prometheus-stack-operator             1/1     1            1           2d17h

NAME                                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/kube-prometheus-stack-kube-state-metrics-c7bbf884   1         1         1       2d17h
replicaset.apps/kube-prometheus-stack-operator-57f5866866           1         1         1       2d17h

NAME                                                               READY   AGE
statefulset.apps/alertmanager-kube-prometheus-stack-alertmanager   1/1     2d17h
statefulset.apps/prometheus-kube-prometheus-stack-prometheus       1/1     2d17h

At this point we can access Prometheus and check if our targets were created correctly and are reachable:

If any of the targets is down, check the message and eventually verify the Prometheus pod logs. Any unreachable target will appear on the logs:

$ kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0
ts=2022-11-11T18:43:46.783Z caller=main.go:543 level=info msg="Starting Prometheus Server" mode=server version="(version=2.39.1, branch=HEAD, revision=dcd6af9e0d56165c6f5c64ebbc1fae798d24933a)"
ts=2022-11-11T18:43:46.783Z caller=main.go:548 level=info build_context="(go=go1.19.2, user=root@273d60c69592, date=20221007-15:57:09)"
ts=2022-11-11T18:43:46.783Z caller=main.go:549 level=info host_details="(Linux 5.10.142-flatcar #1 SMP Tue Oct 11 18:43:33 -00 2022 x86_64 prometheus-kube-prometheus-stack-prometheus-0 (none))"
ts=2022-11-11T18:43:46.783Z caller=main.go:550 level=info fd_limits="(soft=1073741816, hard=1073741816)"
ts=2022-11-11T18:43:46.783Z caller=main.go:551 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-11-11T18:43:46.785Z caller=web.go:559 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-11-11T18:43:46.786Z caller=main.go:980 level=info msg="Starting TSDB ..."
ts=2022-11-11T18:43:46.789Z caller=tls_config.go:231 level=info component=web msg="TLS is disabled." http2=false
ts=2022-11-11T18:43:46.790Z caller=head.go:551 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-11-11T18:43:46.790Z caller=head.go:595 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.654µs
ts=2022-11-11T18:43:46.790Z caller=head.go:601 level=info component=tsdb msg="Replaying WAL, this may take a while"

Next, we have to import the Grafana dashboards. If you installed grafana together, your deployment already has everything configured. In my case, i needed to “extract” the dashboard json files from the helm templates and manually import them. I uploaded the dashboards on my GitHub if you want to make use of them: https://github.com/biofalopes/grafana_dashboards.

And now you can check your metrics and alerts and start enabling notifications, thresholds and set up your monitoring process.

You May Also Like