Using Rook to leverage Ceph storage on a Kubernetes cluster
I recently got 10 bare-metal servers to play with, they used to be part of our first Ceph cluster and got replaced with more powerful hardware. So i …
After experimenting with many different approaches and manually creating lots of prometheus jobs and grafana dashboards, I came across the kube-prometheus-stack helm chart from the prometheus-community GitHub repository (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack). That chart comes with all the components I was looking for, specially because it covers almost everything I had previously and a few more metrics. On top of that, it comes with preconfigured alertmanager rules for all of the components, which is very handy.
It is possible to select only the components you require by overriding the values.yaml file from the chart and tailor it to better suit your needs. In my case, I disabled the Grafana component since I already had one deployed on the environment and it was being used by other team members.
I’m usually not a big fan of helm charts, because altough they make it easy to deploy many components and have a lifecycle management of a helm release, if you need to change some parameter not covered by the template you will have to fork the chart and modify it locally. Another situation is when you integrate with ArgoCD. You can create an application object and call a helm chart on it, but the behavior is a bit different for drift detection and remediation when ArgoCD is dealing with the kubernetes object themselves directly, and in some cases if you update the chart, ArgoCD will end up deleting and recreating everything, or you might to manually trigger a redeployment because ArgoCD won’t detect a change on the chart automatically.
So usually what I do is: first try and adjust everything until I have a decent architecture for whatever I’m deploying, and then proceed to create my own set of manifests without the use of helm.
As is mentioned before, I created an override-values.yaml file with some customizations. I enabled the creation of ingress objects and set a pvc so I could have persistence. I also configured the etcd parameters since I was targeting a Kubernetes cluster with etcd deployed as a service directly on the hosts, and not as pods. that is the default Kubespray behavior, which was my scenario. And finally I disabled Grafana.
alertmanager:
ingress:
enabled: true
ingressClassName: haproxy
hosts:
- alertmanager.somedomain.br
tls:
- hosts:
- alertmanager.somedomain.br
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
storageClassName: ssd-hc
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
prometheus:
ingress:
enabled: true
ingressClassName: haproxy
hosts:
- prometheus.somedomain.br
tls:
- hosts:
- prometheus.somedomain.br
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: ssd-hc
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
secrets: ["etcd-client-cert"]
additionalScrapeConfigs:
kubeEtcd:
enabled: true
endpoints:
- 10.1.2.11
- 10.1.2.12
- 10.1.2.13
service:
enabled: true
port: 2379
targetPort: 2379
serviceMonitor:
scheme: https
insecureSkipVerify: false
caFile: /etc/prometheus/secrets/etcd-client-cert/etcd-ca.pem
certFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client-prometheus.crt
keyFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client-prometheus.key
grafana:
enabled: false
I still have to push the docker images to our private registry and change that on the file (todo).
An extra step is required so Prometheus can access the etcd metrics, since it has SSL enabled. In order to do that, I got the etcd ca certificate and key from a master node (they are located at /etc/ssl/etcd/ssl
) and copied them to my working directory. Then i used cfssl
to easily generate the client certificate to prometheus. I won’t cover installing cfssl, but you can find instructions at https://github.com/cloudflare/cfssl.
cfssl print-defaults csr > prometheus.json
cfssl print-defaults config > ca-config.json
{
"signing": {
"default": {
"expiry": "43800h"
},
"profiles": {
"server": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
{
"CN": "prometheus.somedomain.br",
"hosts": [""],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "BR",
"L": "PN",
"ST": "Bacheland"
}
]
}
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=client prometheus.json | cfssljson -bare prometheus
mv etcd-client-prometheus.pem etcd-client-prometheus.crt
kubectl -n monitoring create secret generic etcd-client-cert --from-file=./etcd-ca.pem --from-file=./etcd-client-prometheus.crt --from-file=./etcd-client-prometheus.key
In order to collect kube-proxy metrics, it is necessary to change its ConfigMap and modify the metricsBindAddress parameter to make it listen on all interfaces.
kubectl -n kube-system edit cm kube-proxy
# metricsBindAddress: 127.0.0.1:10249
metricsBindAddress: 0.0.0.0:10249
kubectl -n kube-system delete pod kube-proxy-<id> # delete all pods
helm install --create-namespace -n monitoring kube-prometheus-stack prometheus-community/kube-prometheus-stack -f override-values.yaml
$ kubectl -n monitoring get all
NAME READY STATUS RESTARTS AGE
pod/alertmanager-kube-prometheus-stack-alertmanager-0 2/2 Running 1 (2d17h ago) 2d17h
pod/kube-prometheus-stack-kube-state-metrics-c7bbf884-428wq 1/1 Running 0 2d17h
pod/kube-prometheus-stack-operator-57f5866866-4l9kt 1/1 Running 0 2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-49wmv 1/1 Running 0 2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-9cvx4 1/1 Running 0 2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-llr9z 1/1 Running 0 2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-qdl74 1/1 Running 0 2d17h
pod/kube-prometheus-stack-prometheus-node-exporter-t5n4x 1/1 Running 0 2d17h
pod/prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 2d16h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 2d17h
service/kube-prometheus-stack-alertmanager ClusterIP 192.168.7.85 <none> 9093/TCP 2d17h
service/kube-prometheus-stack-kube-state-metrics ClusterIP 192.168.37.114 <none> 8080/TCP 2d17h
service/kube-prometheus-stack-operator ClusterIP 192.168.49.194 <none> 443/TCP 2d17h
service/kube-prometheus-stack-prometheus ClusterIP 192.168.10.77 <none> 9090/TCP 2d17h
service/kube-prometheus-stack-prometheus-node-exporter ClusterIP 192.168.39.150 <none> 9100/TCP 2d17h
service/prometheus-operated ClusterIP None <none> 9090/TCP 2d17h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-prometheus-stack-prometheus-node-exporter 5 5 5 5 5 <none> 2d17h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kube-prometheus-stack-kube-state-metrics 1/1 1 1 2d17h
deployment.apps/kube-prometheus-stack-operator 1/1 1 1 2d17h
NAME DESIRED CURRENT READY AGE
replicaset.apps/kube-prometheus-stack-kube-state-metrics-c7bbf884 1 1 1 2d17h
replicaset.apps/kube-prometheus-stack-operator-57f5866866 1 1 1 2d17h
NAME READY AGE
statefulset.apps/alertmanager-kube-prometheus-stack-alertmanager 1/1 2d17h
statefulset.apps/prometheus-kube-prometheus-stack-prometheus 1/1 2d17h
At this point we can access Prometheus and check if our targets were created correctly and are reachable:
If any of the targets is down, check the message and eventually verify the Prometheus pod logs. Any unreachable target will appear on the logs:
$ kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0
ts=2022-11-11T18:43:46.783Z caller=main.go:543 level=info msg="Starting Prometheus Server" mode=server version="(version=2.39.1, branch=HEAD, revision=dcd6af9e0d56165c6f5c64ebbc1fae798d24933a)"
ts=2022-11-11T18:43:46.783Z caller=main.go:548 level=info build_context="(go=go1.19.2, user=root@273d60c69592, date=20221007-15:57:09)"
ts=2022-11-11T18:43:46.783Z caller=main.go:549 level=info host_details="(Linux 5.10.142-flatcar #1 SMP Tue Oct 11 18:43:33 -00 2022 x86_64 prometheus-kube-prometheus-stack-prometheus-0 (none))"
ts=2022-11-11T18:43:46.783Z caller=main.go:550 level=info fd_limits="(soft=1073741816, hard=1073741816)"
ts=2022-11-11T18:43:46.783Z caller=main.go:551 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-11-11T18:43:46.785Z caller=web.go:559 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-11-11T18:43:46.786Z caller=main.go:980 level=info msg="Starting TSDB ..."
ts=2022-11-11T18:43:46.789Z caller=tls_config.go:231 level=info component=web msg="TLS is disabled." http2=false
ts=2022-11-11T18:43:46.790Z caller=head.go:551 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-11-11T18:43:46.790Z caller=head.go:595 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.654µs
ts=2022-11-11T18:43:46.790Z caller=head.go:601 level=info component=tsdb msg="Replaying WAL, this may take a while"
Next, we have to import the Grafana dashboards. If you installed grafana together, your deployment already has everything configured. In my case, i needed to “extract” the dashboard json files from the helm templates and manually import them. I uploaded the dashboards on my GitHub if you want to make use of them: https://github.com/biofalopes/grafana_dashboards.
And now you can check your metrics and alerts and start enabling notifications, thresholds and set up your monitoring process.
I recently got 10 bare-metal servers to play with, they used to be part of our first Ceph cluster and got replaced with more powerful hardware. So i …
So I needed to practice using decorators in Python. I tried to come up with something within my current knowledge and since I was studying sorting …