After using Mend Renovate for some time now I had to SSH into the control plane to view how FluxCD redeploys my applications. This takes a while every time, and with multiple pull requests it gets tiring quickly.

My current Prometheus & Grafana stack is more suited for monitoring performance metrics that provide excellent measurements of system performance. What I needed however, was a way to monitor logs, as they provide in depth timestamped records of events, making them essential for troubleshooting. This is where the EFK stack (Elasticsearch, Fluent Bit and Kibana) comes in.

This will allow me to monitor two valuable insights:

• Flux events: The Flux controllers produces Kubernetes events during the reconciliation operation to provide information about the object being reconciled.
• Kubernetes events: These events are generated when cluster resources such as, pods, deployments or nodes change their state. Examples are: SuccessfulCreate, Scheduled, Pulled, Started.

Monitoring these two insights allows me to follow a paper trail of events triggered by Mend Renovate.

1. Merge Pull requests generated by Renovate
2. Flux storing the artifact for the commit
3. The new pods are scheduled and started
4. Once the new pods are ready, the old pods are terminated
5. This continues until all pods are replaced with new ones

How to configure Fluent Bit

Fluent Bit is deployed as a DaemonSet, which is a pod that runs on every node of my K3s cluster. It has a lot of moving parts to consider first.

• 1. ServiceAccount
• 2. ClusterRole
• 3. ClusterRoleBinding
• 4. Fluent Bit configuration
• 5. DaemonSet
• 6. Deployment and Verification
• 7. Wrapping up

1. ServiceAccount

First up is to create a ServiceAccount and later on granting the correct permissions defined in the ClusterRole and ClusterRoleBinding

My ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit

I left out the namespace because FluxCD will take care of that with targetNamespace.

2. ClusterRole

Now we need to grant the get, list and watch permissions for the following resources:

• namespaces
• pods
• nodes
• events
• nodes/proxy

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - pods
    - nodes
    - events
    - nodes/proxy
  verbs:
    - get
    - list
    - watch

These permissions allow Fluent Bit to use the Kubelet to send requests for Pod information instead of using the kube-apiserver to avoid unnecessary bottlenecks. It also allows Fluent Bit to retrieve Kubernetes events as logs process them through the pipeline.

Troubleshooting issues with Kubernetes events is extremely valuable because it gives you visibility in the cluster.

3. ClusterRoleBinding

To give Fluent Bit permission to enrich logs with Kubernetes metadata we have to grant it cluster-wide read-only access:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit

4. Fluent Bit configuration

I struggled a couple of hours with the Fluent Bit configuration and finally settled on the following configmap. The main configuration file supports four sections:

• Service
• Input
• Filter
• Output

fluent-bit.conf

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
        Parsers_File  /fluent-bit/etc/parsers.conf

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Tag               kube.<namespace_name>.<pod_name>.<container_name>.<container_id>
        Tag_Regex         (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64})\.log$
        Parser            cri
        multiline.parser  cri
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On

    [INPUT]
        Name              kubernetes_events
        Tag               k8s_events
        Kube_URL          https://kubernetes.default.svc

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.
        Regex_Parser        custom-tag
        Merge_Log           On
        Merge_Log_Key       log_processed
        Buffer_Size         0
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Use_Kubelet         true
        Kubelet_Port        10250

    [OUTPUT]
        Name            es
        Match           *
        Host            elasticsearch
        Port            9200
        Index           fluentbit-%Y.%m.%d
        Replace_Dots    On
        Retry_Limit     False
        Suppress_Type_Name On

  parsers.conf: |
    [PARSER]
        Name    cri
        Format  regex
        Regex   ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[FP]) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep   On

    [PARSER]
        Name    custom-tag
        Format  regex
        Regex   ^(?<namespace_name>[^_]+)\.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?<container_name>.+)\.(?<container_id>[a-z0-9]{64})

1. Multiple input sources:

• Tail Input: This reads the standard application logs from all containers on the node, written to /var/log/containers/*.log.

• Kubernetes Events Input: Pulling events from the API server, giving insights about image pulls, scheduling, failures etc.

For more information about Kubernetes events you can look into the fluentbit docs: Kubernetes events

2. Enhanced Filtering

The custom-tag embeds namespace, pod, container and container ID directly in the tag. With the [FILTER], Kubernetes doesn’t need to parse the filename every time. For more information look into: Tail Input and Custom tags for enhanced filtering

3. Elasticsearch Output

The Elasticsearch Output creates a fluentbit- index that lets us transport these records to Elasticsearch. You can check the Fluent Bit: Elasticsearch docs for details.

5. Daemonset

The Fluent Bit docs provide an excellent DaemonSet configuration example.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit-v2
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:4.0.10
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        resources:
          requests:
            memory: "250Mi"
            cpu: "150m"
          limits:
            memory: "250Mi"
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config

This DaemonSet is almost identical to the example. I only changed the resources requests and limits. Another thing to keep in mind is to set is to set hostNetwork to true and dnsPolicy to ClusterFirstWithHostNet so the Fluent Bit DaemonSet can call Kubelet locally. Otherwise it can't resolve DNS for kubelet.

6. Deployment and Verification

We use FluxCD to manage our Kubernetes manifests, and these Fluent Bit resources are deployed automatically to the monitoring namespace.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: fluentbit-kustomization
  namespace: default
spec:
  interval: 5m
  targetNamespace: monitoring
  sourceRef:
    kind: GitRepository
    name: fluentbit-repo
  path: "./apps/overlays/rudolf/fluentbit"
  prune: true
  wait: true
  timeout: 1m

Once Flux has reconciled these resources, it's a good idea to check if the pods are running on each node:

kubectl get pods -n monitoring

NAME                                  READY   STATUS    RESTARTS       AGE
fluent-bit-v2-f7xfw                   1/1     Running   0              21h
fluent-bit-v2-lkl9w                   1/1     Running   0              21h
fluent-bit-v2-rk594                   1/1     Running   0              21h
fluent-bit-v2-tklhm                   1/1     Running   0              21h       

Next, look inside the pod logs for errors:

kubectl logs fluent-bit-v2-f7xfw -n monitoring

To know if Fluent Bit is using the kubelet, you can review Fluent Bit logs. There should be a log like this:

[2025/09/23 10:46:54] [ info] [filter:kubernetes:kubernetes.0] connectivity OK

This is what my fluent-bit logs looks like after merging a pull request for Fluent Bit v4.1.0:

Fluent Bit v4.1.0
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___   __
|  ___| |                | |   | ___ (_) |           /   | /  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| | `| |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| |  | |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |__| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/


[2025/09/25 09:59:50.698848279] [ info] [fluent bit] version=4.1.0, commit=a8bd9e4cf6, pid=1
[2025/09/25 09:59:50.698913312] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/09/25 09:59:50.698917675] [ info] [simd    ] SSE2
[2025/09/25 09:59:50.698920196] [ info] [cmetrics] version=1.0.5
[2025/09/25 09:59:50.698922786] [ info] [ctraces ] version=0.6.6
[2025/09/25 09:59:50.698974641] [ info] [input:tail:tail.0] initializing
[2025/09/25 09:59:50.698977980] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/09/25 09:59:50.699120101] [ info] [input:tail:tail.0] multiline core started
[2025/09/25 09:59:50.699652737] [ info] [input:kubernetes_events:kubernetes_events.1] initializing
[2025/09/25 09:59:50.699657185] [ info] [input:kubernetes_events:kubernetes_events.1] storage_strategy='memory' (memory only)
[2025/09/25 09:59:50.699906249] [ info] [input:kubernetes_events:kubernetes_events.1] API server: https://kubernetes.default.svc:443
[2025/09/25 09:59:50.702434193] [ info] [input:kubernetes_events:kubernetes_events.1] thread instance initialized
[2025/09/25 09:59:50.702523151] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2025/09/25 09:59:50.703456018] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2025/09/25 09:59:50.703463984] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2025/09/25 09:59:50.704342198] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with Kubelet...
[2025/09/25 09:59:50.714275092] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2025/09/25 09:59:50.723436389] [ info] [output:es:es.0] worker #0 started
[2025/09/25 09:59:50.723534608] [ info] [output:es:es.0] worker #1 started
[2025/09/25 09:59:50.724027952] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/09/25 09:59:50.724035501] [ info] [sp] stream processor started
[2025/09/25 09:59:50.724098316] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2

If you do come across any warnings or errors you can explore the Troubleshooting section.

7. Wrapping up

In this first part we successfully deployed Fluent Bit with the Kubernetes filter. Next up we will take a look at configuring Elasticsearch with basic authentication so that we can run it in our local cluster with username and password authentication in part two.

Sources

Kubernetes Events
Kubernetes Filter