--- title: "Setting up Prometheus and Grafana on Kubernetes using Helm" description: "Setting up Prometheus and Grafana on Kubernetes using Helm" canonical_url: "https://www.bigbinary.com/blog/prometheus-and-grafana-integration" markdown_url: "https://www.bigbinary.com/blog/prometheus-and-grafana-integration.md" --- # Setting up Prometheus and Grafana on Kubernetes using Helm Setting up Prometheus and Grafana on Kubernetes using Helm - Author: Vishal Yadav - Published: January 25, 2024 - Categories: NeetoCI, Kubernetes In this blog, we will learn how to set up Prometheus and Grafana on Kubernetes using Helm. [Prometheus](https://prometheus.io/) along with [Grafana](https://grafana.com/) is a highly scalable open-source monitoring framework for [container orchestration platform](https://devopscube.com/docker-container-clustering-tools/). Prometheus probes the application and collects various data. It stores all this data in its time series database. Grafana is a visualization tool. It uses the data from the database to show the data that is meaningful to the user. Both Prometheus and Grafana are gaining popularity in the [observability](https://devopscube.com/what-is-observability/) space as it helps with metrics and alerts. Learning to integrate them using Helm will allow us to monitor our Kubernetes cluster and troubleshoot problems easily. Furthermore, we can deep dive into our cluster's well-being and efficiency, focusing on resource usage and performance metrics within our Kubernetes environment. We will also learn how to create a simple [dashboard](https://grafana.com/grafana/dashboards/) on Grafana. ## **Why using Prometheus and Grafana for monitoring is good** Using Prometheus and Grafana for monitoring has many benefits: - **Scalability:** Both tools are highly scalable and can handle the monitoring needs of small to large Kubernetes clusters. - **Flexibility:** They allow us to create custom dashboards tailored to our specific monitoring requirements. - **Real-time Monitoring:** Prometheus provides real-time monitoring, helping us to quickly detect and respond to issues. - **Alerting:** Prometheus enables us to set up alerts based on specific metrics, so we can be notified when issues arise. - **Data Visualization:** Grafana offers powerful data visualization capabilities, making it easier to understand complex data. - **Open Source:** Both Prometheus and Grafana are open-source, reducing monitoring costs. - **Community Support:** We can benefit from active communities, ensuring continuous development and support. - **Integration:** They seamlessly integrate with other Kubernetes components and applications, simplifying setup. - **Historical Data:** Grafana allows us to explore historical data, aiding in long-term analysis and trend identification. - **Extensible:** Both tools are extensible, allowing us to integrate additional data sources and plugins. - **Efficient Resource Usage:** Prometheus efficiently utilizes resources, ensuring minimal impact on our cluster's performance. Two common ways to use Prometheus and Grafana on Kubernetes: 1. **Manual Kubernetes deployment**: In this method, we need to write  [Kubernetes Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)  and  [Services](https://kubernetes.io/docs/concepts/services-networking/service/) for both Prometheus and Grafana. In the YAML file, we need to put all the settings for Prometheus and Grafana on Kubernetes. Then we send these files to our Kubernetes cluster. But we can end up with many YAML files, which can be hard. If we make a mistake in any YAML file, Prometheus and Grafana won't work on Kubernetes. 2. **Using Helm**: This is an easy way to send any application container to Kubernetes. [Helm](https://helm.sh/) is the official package manager for Kubernetes. With Helm, we can make installing, sending, and managing Kubernetes makes applications easier. A [Helm Chart](https://helm.sh/) has all the YAML files: - Deployments. - Services. - Secrets. - ConfigMaps manifests. We use these files to send the application container to Kubernetes. Instead of making individual YAML files for each application container, Helm lets us download Helm charts that already have YAML files. ## Setting up Prometheus and Grafana using Helm chart We will use [ArtifactHub](https://artifacthub.io/), which offers public and private repositories for Helm Charts. We will use these Helm Charts to arrange the pods and services in our Kubernetes cluster. To get Prometheus and Grafana working on Kubernetes with Helm, we will start by installing Helm. #### Installing Helm on Linux ```bash sudo apt-get install helm ``` #### Installing Helm on Windows ```bash choco install Kubernetes-helm ``` #### Installing Helm on macOS ```bash brew install helm ``` We can check out the official  [Helm documentation](https://helm.sh/docs/intro/install/)  if we run into any issues while installing Helm. The image below represents the successful Helm installation on macOS. ![Screenshot 2023-11-01 at 1.09.49 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-01_at_1.09.49_PM.png) For this blog, we’re going to install Helm [chart](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack) and by default, this chart also installs additional, dependent charts (including Grafana): - [prometheus-community/kube-state-metrics](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-state-metrics) - [prometheus-community/prometheus-node-exporter](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter) - [grafana/grafana](https://github.com/grafana/helm-charts/tree/main/charts/grafana) To get this Helm chart, let's run this command: ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update ``` ![Screenshot 2023-11-01 at 1.15.43 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-01_at_1.15.43_PM.png) We have downloaded the latest version of Prometheus & Grafana. To install the Prometheus Helm Chart on a Kubernetes Cluster, let's run the following command: ```bash helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack ``` ![Screenshot 2023-11-01 at 1.21.46 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-01_at_1.21.46_PM.png) We have successfully installed Prometheus & Grafana on the Kubernetes Cluster. We can access the Prometheus & Grafana servers via ports 9090 & 80, respectively. Now, let's run the following command to view all the resources created by the Helm Chart in our Kubernetes cluster: ```bash kubectl get all ``` ![Screenshot 2023-11-02 at 11.19.39 AM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.19.39_AM.png) The Helm chart created the following resources: - **Pods**: It hosts the deployed Prometheus Kubernetes application inside the cluster. - **Replica Sets**: A collection of instances of the same application inside the Kubernetes cluster. It enhances application reliability. - **Deployments**: It is the blueprint for creating the application pods. - **Services**: It exposes the pods running inside the Kubernetes cluster. We use it to access the deployed Kubernetes application. - **Stateful Sets**: They manage the deployment of the stateful application components and ensure stable and predictable network identities for these components. - **Daemon Sets**: They ensure that all (or a specific set of) nodes run a copy of a pod, which is useful for tasks such as logging, monitoring, and other node-specific operations. Run this command to view all the Kubernetes Services for Prometheus & Grafana: ```bash kubectl get service ``` ![Screenshot 2023-11-02 at 11.37.39 AM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.37.39_AM.png) Listed services for Prometheus and Grafana are: - alertmanager-operated - kube-prometheus-stack-alertmanager - kube-prometheus-stack-grafana - kube-prometheus-stack-kube-state-metrics - kube-prometheus-stack-operator - kube-prometheus-stack-prometheus - kube-prometheus-stack-prometheus-node-exporter - prometheus-operated `kube-prometheus-stack-grafana` and `kube-prometheus-stack-prometheus` are the `ClusterIP` type services, which means we can only access them within the Kubernetes cluster. To expose the Prometheus and Grafana to be accessed outside the Kubernetes cluster , we can either use the NodeIP or LoadBalance service. ## Exposing Prometheus and Grafana using NodePort services Let's run the following command to expose the `Prometheus` Kubernetes service: ```bash kubectl expose service kube-prometheus-stack-prometheus --type=NodePort --target-port=9090 --name=prometheus-node-port-service kubectl expose service kube-prometheus-stack-grafana --type=NodePort --target-port=3000 --name=grafana-node-port-service ``` That command will create new services of `NodePort` type & make the Prometheus and Grafana is accessible outside the Kubernetes Cluster on ports `9090` and `80`. ![Screenshot 2023-11-02 at 11.59.15 AM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.59.15_AM.png) As we can see, the `grafana-node-port-service` and `prometheus-node-port-service` are successfully created and are being exposed on node ports `32489` & `30905` Now, we can run this command and get the external IP of any node to access the Prometheus and Grafana: ```jsx kubectl get nodes -o wide ``` ![Screenshot 2023-11-02 at 11.57.17 AM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.57.17_AM.png) We can use the External-IP and the node ports to access the Prometheus and Grafana dashboards outside the cluster environment. Prometheus Dashboard ![Prometheus Dashboard](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.04.36_PM.png) Grafana Dashboard ![Grafana Dashboard](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.04.53_PM.png) Run this command, to get the password for the **admin** user of the Grafana dashboard: ``` kubectl get secret --namespace default kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` ## Grafana Dashboard Upon login to the Grafana dashboard, use `admin` as the username and our generated password. We will see "Welcome to Grafana" homepage as shown below. ![Screenshot 2023-11-02 at 12.12.04 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.12.04_PM.png) Since we used the Kube Prometheus Stack helm chart, the data source for Prometheus and Alert Manager is added by default. ![Screenshot 2023-11-02 at 12.18.14 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.18.14_PM.png) We can add more data sources by clicking on the **Add new data source** button on the top right side. By default, this Helm chart adds multiple dashboards to monitor the health of the Kubernetes cluster and its resources. ![Screenshot 2023-11-02 at 12.22.37 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.22.37_PM.png) Additionally, we also have the option of creating our dashboards from scratch as well as importing multiple Grafana dashboards provided by the [Grafana library](https://grafana.com/grafana/dashboards/). To import a Grafana Dashboard, let's follow these steps: - From this [Grafana library](https://grafana.com/grafana/dashboards/), we can add any dashboard ![Screenshot 2023-11-02 at 12.28.44 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.28.44_PM.png) - Select Dashboard and copy the Dashboard ID ![Screenshot 2023-11-02 at 12.34.54 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.34.54_PM.png) - Under **Dashboards** page we can get the **Import** option ![Screenshot 2023-11-02 at 12.32.55 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.32.55_PM.png) - Under "Import Dashboard" page, we need to paste the Dashboard IP that we copied earlier & click on the **Load** button. ![Screenshot 2023-11-02 at 12.35.39 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.35.39_PM.png) - After clicking on the **Load** button, it will auto-load the dashboard from the library after which we can import the dashboard by clicking on the **Import** button. ![Screenshot 2023-11-02 at 12.37.50 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.37.50_PM.png) - Once the import is complete, we’ll be redirected to the new imported dashboard which’ll also be visible under the Dashboards page. ![Screenshot 2023-11-02 at 12.39.42 PM.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.39.42_PM.png) We can use this Node Exporter dashboard to monitor & observe the health of our nodes present in our Kubernetes Cluster. ## Conclusion In this blog, we learned how to integrate Prometheus and Grafana using the helm chart. We also learned how to import dashboards into Grafana from the [Grafana library](https://grafana.com/grafana/dashboards/). In the next blog, we will explore how to integrate [Grafana Loki](https://grafana.com/oss/loki/) with Grafana and collect and store event-related metrics using the [Kubernetes Event Exporter](https://github.com/resmoio/kubernetes-event-exporter). ## Links - [Human page](https://www.bigbinary.com/blog/prometheus-and-grafana-integration)