--- title: "Grafana Loki and Kubernetes Event exporter" description: "In this blog, we'll explore how to integrate the Kubernetes Event exporter & Grafana Loki into your Kubernetes Cluster using a helm chart." canonical_url: "https://www.bigbinary.com/blog/k8s-event-exporter-and-grafana-loki-integration" markdown_url: "https://www.bigbinary.com/blog/k8s-event-exporter-and-grafana-loki-integration.md" --- # Grafana Loki and Kubernetes Event exporter In this blog, we'll explore how to integrate the Kubernetes Event exporter & Grafana Loki into your Kubernetes Cluster using a helm chart. - Author: Vishal Yadav - Published: May 7, 2024 - Categories: NeetoCI, Prometheus, Grafana, Kubernetes In the previous [blog](https://www.bigbinary.com/blog/prometheus-and-grafana-integration), we discussed integrating [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) in the Kubernetes Cluster. In this blog, we'll explore how to integrate the [Kubernetes Event exporter](https://github.com/resmoio/kubernetes-event-exporter) & [Grafana Loki](https://grafana.com/oss/loki/) into your Kubernetes Cluster using a helm chart. Additionally, you’ll also learn how to add Grafana Loki as a data source to your Grafana Dashboard. This will help you visualize the Kubernetes events. Furthermore, we'll delve into the specifics of setting up the Event exporter and Grafana Loki, ensuring you understand each step of the process. From downloading and configuring the necessary helm charts to understanding the Grafana Loki dashboard, we'll cover it all. By the end of this blog, you'll be able to fully utilize Grafana Loki and Kubernetes Event Exporter, gaining insights from your Kubernetes events. ## How Kubernetes event exporter can help us in monitoring health Objects in Kubernetes, such as Pod, Deployment, Ingress, Service publish events to indicate status updates or problems. Most of the time, these events are overlooked and their 1-hour lifespan might cause missing important updates. They are also not searchable and cannot be aggregated. For instance, they can alert you to changes in the state of pods, errors in scheduling, and resource constraints. Therefore, exporting these events and visualizing them can be crucial for maintaining the health of your cluster. Kubernetes event exporter allows exporting the often missed Kubernetes events to various outputs so that they can be used for observability or alerting purposes. We can have multiple receivers to export the events from the Kubernetes cluster. - [Opsgenie](https://www.opsgenie.com/) - [Webhooks/HTTP](https://github.com/resmoio/kubernetes-event-exporter#webhookshttp) - [Elasticsearch](https://www.elastic.co/) - [OpenSearch](https://opensearch.org/) - [Slack](https://github.com/resmoio/kubernetes-event-exporter#slack) - [Kinesis](https://github.com/resmoio/kubernetes-event-exporter#kinesis) - [Firehose](https://github.com/resmoio/kubernetes-event-exporter#firehose) - [SNS](https://github.com/resmoio/kubernetes-event-exporter#sns) - [SQS](https://github.com/resmoio/kubernetes-event-exporter#sqs) - [File](https://github.com/resmoio/kubernetes-event-exporter#file) - [Stdout](https://github.com/resmoio/kubernetes-event-exporter#stdout) - [Kafka](https://github.com/resmoio/kubernetes-event-exporter#kafka) - [OpsCenter](https://docs.aws.amazon.com/systems-manager/latest/userguide/OpsCenter.html) - [Customize Payload](https://github.com/resmoio/kubernetes-event-exporter#customizing-payload) - [Pubsub](https://github.com/resmoio/kubernetes-event-exporter#pubsub) - [Teams](https://github.com/resmoio/kubernetes-event-exporter#teams) - [Syslog](https://github.com/resmoio/kubernetes-event-exporter#syslog) - [Bigquery](https://github.com/resmoio/kubernetes-event-exporter#bigquery) - [Pipe](https://github.com/resmoio/kubernetes-event-exporter#pipe) - [Event Bridge](https://github.com/resmoio/kubernetes-event-exporter#aws-eventbridge) - [Grafana Loki](https://github.com/resmoio/kubernetes-event-exporter#loki) ## Setting up Grafana Loki & Kubernetes event exporter using Helm chart We will once again use [ArtifactHub](https://artifacthub.io/), which provides a helm chart for installing Grafana Loki onto a Kubernetes Cluster. If you need instructions on how to install Helm on your system, you can refer to this blog. In this blog post, we will install a Helm [chart](https://artifacthub.io/packages/helm/grafana/loki) that sets up Loki in scalable mode, with separate read-and-write components that can be independently scaled. Alternatively, we can install Loki in monolithic mode, where the Helm Chart installation runs the Grafana Loki _single binary_ within a Kubernetes cluster. You can learn more about this [here](https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/#install-the-monolithic-helm-chart). ### 1. Create S3 buckets - grafana-loki-chunks-bucket - grafana-loki-admin-bucket - grafana-loki-ruler-bucket ![loki-s3-buckets.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-s3-buckets.png) ### 2. Create a policy for Grafana Loki Create a new policy under IAM on Amazon AWS using the below snippet. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "LokiStorage", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::grafana-loki-chunks-bucket", "arn:aws:s3:::grafana-loki-chunks-bucket/*", "arn:aws:s3:::grafana-loki-admin-bucket", "arn:aws:s3:::grafana-loki-admin-bucket/*", "arn:aws:s3:::grafana-loki-ruler-bucket", "arn:aws:s3:::grafana-loki-ruler-bucket/*" ] } ] } ``` Output: ![grafana-loki-policy.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/grafana-loki-policy.png) ### 3. Create a Role with the above permission Create a role with a custom trust policy & use the below snippet ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::account_id:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/open_id" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.us-east-1.amazonaws.com/id/open_id:aud": "sts.amazonaws.com", "oidc.eks.us-east-1.amazonaws.com/id/open_id:sub": "system:serviceaccount:default:grafana-loki-access-s3-role-sa" } } } ] } ``` Note: Please update the account_id and open_id in the above given snippet. **grafana-loki-access-s3-role-sa** is the service account name that we will mention in the Loki values. ### 4. Add Grafana using the helm chart To get this Helm chart, run this command: ```jsx helm repo add grafana https://grafana.github.io/helm-charts helm repo update ``` Output: ![chart-add-output.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/chart-add-output.png) We have downloaded the latest version of the Grafana. ### 5. Install grafana/loki stack using the helm chart Create a **loki-values yaml** file with the below snippet ``` loki: readinessProbe: {} auth_enabled: false storage: bucketNames: chunks: grafana-loki-chunks-bucket ruler: grafana-loki-rules-bucket admin: grafana-loki-admin-bucket type: s3 s3: endpoint: null region: us-east-1 secretAccessKey: null accessKeyId: null s3ForcePathStyle: false insecure: false monitoring: lokiCanary: enabled: false selfMonitoring: enabled: false test: enabled: false serviceAccount: create: true name: grafana-loki-access-s3-role-sa imagePullSecrets: [] annotations: eks.amazonaws.com/role-arn: arn:aws:iam::account_id:role/loki-role automountServiceAccountToken: true ``` To install Loki using the Helm Chart on Kubernetes Cluster, run this `helm install` command: ```jsx helm install my-loki grafana/loki --values loki-values.yaml ``` Output: ![chart-installation-output.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/chart-installation-output.png) We have successfully installed Loki on the Kubernetes Cluster. Run the following command to view all the resources created by the Loki Helm Chart in your Kubernetes cluster: ```jsx kubectl get all -l app.kubernetes.io/name=loki ``` Output: ![all-resources-output.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/all-resources-output.png) Helm chart created the following components: - **Loki read and write:** Loki is installed in scalable mode by default, which includes a read-and-write component. These components can be independently scaled out. - **Gateway:** Inspired by Grafana’s [Tanka setup](https://github.com/grafana/loki/blob/main/production/ksonnet/loki), the chart installs a gateway component by default. This NGINX component exposes Loki’s API and automatically proxies requests to the appropriate Loki components (read or write, or a single instance in the case of filesystem storage). The gateway must be enabled to provide an Ingress since the Ingress only exposes the gateway. If enabled, Grafana and log shipping agents, such as Promtail, should be configured to use the gateway. If NetworkPolicies are enabled, they become more restrictive when the gateway is active. - **Caching:** In-memory caching is enabled by default. If this type of caching is unsuitable for your deployment, consider setting up memcache. Run this command to view all the Kubernetes Services for Prometheus & Grafana: ```jsx kubectl get service -l app.kubernetes.io/name=loki ``` Output: ![all-services-output.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/all-services-output.png) Listed services for Loki are: - loki-backend - loki-backend-headless - loki-gateway - loki-memberlist - loki-read - loki-read-headless - loki-write - loki-write-headless - query-scheduler-discovery The `loki-gateway` service will be used to add Loki as a Datasource into Grafana. ### 6. Adding Loki data source in Grafana On the main page of Grafana, click on "**Home**". Under "**Connections**", you will find the "**Data sources**" option. ![/blog/images/images_used_in_blog/event-exporter-and-grafana-loki-integration/grafana-dashboard.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/grafana-dashboard.png) On the Data Sources page, click on the "Add new data source" button. ![/blog/images/images_used_in_blog/event-exporter-and-grafana-loki-integration/data-sources-page.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/data-sources-page.png) In the search bar, type "Loki" and search for it. ![/blog/images/images_used_in_blog/event-exporter-and-grafana-loki-integration/add-data-source.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/add-data-source.png) Clicking on "Loki" will redirect you to the dedicated page for the Loki data source. ![/blog/images/images_used_in_blog/event-exporter-and-grafana-loki-integration/loki-data-source.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-data-source.png) To read the metrics from Loki, we will use the `loki-gateway` service. Add the URL of the service as `http://loki-gateway`. ![/blog/images/images_used_in_blog/event-exporter-and-grafana-loki-integration/loki-form.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-form.png) After clicking on the "Save & test" button, you will receive a toastr message shown in the image below. This message is received because no clients have been created for Loki yet. ![loki-addon-output.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-addon-output.png) ### 7. Install Kubernetes event exporter using the helm chart Create an **event-exporter-values yaml** file with the below snippet ```jsx config: leaderElection: {} logLevel: debug logFormat: pretty metricsNamePrefix: event_exporter_ receivers: - name: "dump" file: path: "/dev/stdout" layout: {} - name: "loki" loki: url: "http://loki-gateway/loki/api/v1/push" streamLabels: source: kubernetes-event-exporter container: kubernetes-event-exporter route: routes: - match: - receiver: "dump" - receiver: "loki" ``` With the use of the above snippet, run this command to install the Kubernetes event exporter in your Kubernetes Cluster. ```jsx helm repo add bitnami [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami) helm install event-exporter bitnami/kubernetes-event-exporter --values event-exporter.yaml ``` Output: ![event-exporter-installation-output.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-installation-output.png) To view all the resources created by the above helm chart, run this command: ```jsx kubectl get all -l app.kubernetes.io/name=kubernetes-event-exporter ``` Output: ![event-exporter-all-resources.png](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-all-resources.png) To view the logs of the event exporter POD, run this command: ```jsx kubectl logs -f pod/kubernetes-event-exporter-586455bbdd-sqlqc ``` Note: Replace **kubernetes-event-exporter-586455bbdd-sqlqc** with your pod name. Output: ![event-exporter-logs](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-logs.png) As you can see in the above image, the event exporter is working and running fine. Event logs are being sent to both the receivers that we’d configure in the values YAML file. Once the POD is created & running, we can go back to the Loki data source under **Connections** > **Data Sources** page. Again click on the “Save & test” button & this time you’ll receive a success toastr message. Output: ![loki-data-source-added-output](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-data-source-added-output.png) ### 8. Kubernetes event exporter dashboard We will import this [dashboard](https://grafana.com/grafana/dashboards/17882-kubernetes-event-exporter/) into Grafana to monitor and track the events received from the Kubernetes cluster. You can go through this blog if you want to learn how to import an existing dashboard into Grafana. After successfully importing the dashboard, you can view all the events from the cluster, as shown in the image below. Additionally, you can filter the events based on any value within any interval. Kubernetes Event Exporter ![Kubernetes Event Exporter](https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-dashboard.png) ## Conclusion In this blog post, we discussed the process of setting up Grafana Loki and Kubernetes Event exporter. We covered various steps, such as creating a policy for Grafana Loki, creating a role with the necessary permissions, and adding Grafana using the Helm chart, installing the Loki stack, and adding Loki as a data source in Grafana, installing Kubernetes event exporter using the Helm chart, and finally, setting up the Kubernetes event exporter dashboard in Grafana. By following the steps outlined in this blog post, you can effectively monitor and track events from your Kubernetes cluster using Grafana Loki and Kubernetes Event exporter. This setup provides valuable insights and helps in troubleshooting and analyzing events in your cluster. If you have any questions or feedback, please feel free to reach out. Happy monitoring! ## Links - [Human page](https://www.bigbinary.com/blog/k8s-event-exporter-and-grafana-loki-integration)