logs

Monitor Kubernetes Logs with Grafana Alloy and Loki

Note this scenario works using the K8s Monitoring Helm chart. This abstracts the need to configure Alloy and deploys best practices for monitoring Kubernetes clusters. The chart supports; metrics, logs, profiling, and tracing. For this scenario, we will use the K8s Monitoring Helm chart to monitor Kubernetes logs.

This scenario demonstrates how to setup the Kubernetes monitoring helm and Loki. This scenario will install three Helm charts: Loki, Grafana, and k8s-monitoring-helm. Loki will be used to store the logs, Grafana will be used to visualize the logs, and Alloy (k8s-monitoring-helm) will be used to collect three different log sources:

• Pod Logs
• Kubernetes Events

Prerequisites

Clone the repository:

git clone https://github.com/grafana/alloy-scenarios.git

Change to the directory:

cd alloy-scenarios/k8s-logs

Next you will need a Kubernetes cluster (In this example, we will configure a local Kubernetes cluster using Kind)

An example kind cluster configuration is provided in the kind.yml file. To create a kind cluster using this configuration, run the following command:

kind create cluster --config kind.yml

Lastly you will need to make sure you install Helm on your local machine. You can install Helm by following the instructions here. You will also need to install the Grafana Helm repository:

helm repo add grafana https://grafana.github.io/helm-charts

Create the meta and prod namespaces

The first step is to create the meta and prod namespaces. To create the namespaces, run the following commands:

kubectl create namespace meta && \
kubectl create namespace prod

Install the Loki Helm Chart

The first step is to install the Loki Helm chart. This will install Loki in the meta namespace. The loki-values.yml file contains the configuration for the Loki Helm chart. To install Loki, run the following command:

helm install --values loki-values.yml loki grafana/loki -n meta

This installs Loki in monolithic mode. For more information on Loki modes, see the Loki documentation.

Install the Grafana Helm Chart

The next step is to install the Grafana Helm chart. This will install Grafana in the meta namespace. The grafana-values.yml file contains the configuration for the Grafana Helm chart. To install Grafana, run the following command:

helm install --values grafana-values.yml grafana grafana/grafana --namespace meta

Note that within the grafana-values.yml file, the grafana.ini configuration is set to use the Loki data source. This is done by setting the datasources.datasources.yaml field to the Loki data source configuration.

Install the K8s Monitoring Helm Chart

The final step is to install the K8s monitoring Helm chart. This will install Alloy in the meta namespace. The k8s-monitoring-values.yml file contains the configuration for the K8s monitoring Helm chart. To install the K8s monitoring Helm chart, run the following command:

helm install --values ./k8s-monitoring-values.yml k8s grafana/k8s-monitoring -n meta --create-namespace

Within the k8s-monitoring-values.yml file we declare the Alloy configuration. This configuration specifies the log sources that Alloy will collect logs from. In this scenario, we are collecting logs from two different sources: Pod Logs and Kubernetes Events.

Accessing the Grafana UI

To access the Grafana UI, you will need to port-forward the Grafana pod to your local machine. First, get the name of the Grafana pod:

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")

Next, port-forward the Grafana pod to your local machine:

kubectl --namespace meta port-forward $POD_NAME 3000

Open your browser and go to http://localhost:3000. You can log in with the default username admin and password adminadminadmin.

Accessing the Alloy UI

To access the Alloy UI, you will need to port-forward the Alloy pod to your local machine. First, get the name of the Alloy pod:

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=alloy-logs,app.kubernetes.io/instance=k8s" -o jsonpath="{.items[0].metadata.name}")

Next, port-forward the Alloy pod to your local machine:

kubectl --namespace meta port-forward $POD_NAME 12345

View the logs using Explore Logs in Grafana

Explore Logs is a new feature in Grafana which provides a queryless way to explore logs. To access Explore Logs. To access Explore logs open a browser and go to http://localhost:3000/a/grafana-lokiexplore-app.

Adding a demo prod app

The k8s monitoring app is configured to collect logs from two namespaces: meta and prod. To add a demo prod app, run the following command:

helm install tempo grafana/tempo-distributed -n prod

This will install the Tempo distributed tracing system in the prod namespace.