# Logging from Docker Containers to Elasticsearch with Fluent Bit

This guide explains how to setup the lightweight log processor and forwarder Fluent Bit (opens new window) as docker logging driver to catch all stdout produced by your containers, process the logs, and forward them to Elasticsearch.

Twelve-Factor (opens new window) says the following about logs

A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.

Rather than configuring your applications to send logs to your log storage (Elasticsearch/Graylog/...) or even write log files itself, your applications should simply log to stdout.

In a dockerized environment with multiple services, where each container is isolated and logs on it's own, we need an interface to collect those logs. Docker Logging Driver to the rescue. Each docker daemon has a logging driver, which each container uses. That way, each log entry will flow through the logging driver, enabling us to process and forward it in a central place.

# Fluent Bit vs Fluentd

A popular library to solve this is Fluentd (opens new window). An open source log collector, processor and aggregator written in Ruby, first published in 2011.

Fluent Bit is an open source log collector created by the same folks, written in C, first published in 2015.

	Fluentd	Fluent Bit
Scope	Containers / Servers	Containers / Servers
Language	C & Ruby	C
Memory	~40MB	~450KB
Performance	High Performance	High Performance
Dependencies	Built as a Ruby Gem, it requires a certain number of gems.	Zero dependencies, unless some special plugin requires them.
Plugins	More than 650 plugins available	Around 35 plugins available
License	Apache License v2.0 (opens new window)	Apache License v2.0 (opens new window)

Taken from the official documentation (opens new window).

Regarding ecosystem, Fluentd is a lot more mature and adpoted. However, Fluent Bit takes about 1/10 of the resources and does offer plugins for standard tooling.

For simple cases that involve standard tooling (like Elasticsearch) and not focus on aggregation and rather processing and forwarding, I'd recommend using Fluent Bit.

# Getting started with Fluent Bit

Fluent Bit offers official production-ready docker images. For a detailed list check the official docs (opens new window).

Fluent Bit can be configured by file or command line. We will go for configuration by file.

There are different sections of configuration:

Service (opens new window) - defines the global behavior of the Fluent Bit engine
Input (opens new window) - defines the source from where Fluent Bit can collect data
Parser (opens new window) - take unstructured log entries and give them structure
Filter (opens new window) - allows to alter the incoming data generated by the input plugins
Output (opens new window) - defines where Fluent Bit should flush the information it gathers from the input

Let's take a look at an example configuration.

[SERVICE]
	log_level debug

# The stdin plugin allows to retrieve valid JSON text messages over the standard input interface (stdin)
[INPUT]
	Name stdin

# The Record Modifier Filter plugin allows to append fields or to exclude specific fields.
[FILTER]
	Name record_modifier
	Match *
	Record hostname ${HOSTNAME}

# The stdout output plugin allows to print to the standard output the data received through the input plugin.
[OUTPUT]
	Name stdout

Building a custom Fluent Bit Docker Image is fairly simple

FROM fluent/fluent-bit:1.2
ADD fluent-bit.conf /fluent-bit/etc/

Let's test our simple configuration.

Simple Config

# Fluent Bit as Docker Driver

For Fluent Bit to receive every log produced by a container to process and forward, we need to setup Fluent Bit as Docker Logging Driver.

We need to use the forward input plugin for Fluent Bit. Forward is the protocol used by Fluent Bit and Fluentd to route messages between peers. This plugin implements the input service to listen for Forward messages. To check if everything is running just fine, we will keep the stdout plugin for now.

[SERVICE]
    log_level debug

[INPUT]
    Name forward
    Listen 0.0.0.0
    port 24224

[OUTPUT]
    Name stdout
    Match **

Let's spin everything up with docker-compose:

version: "3.5"
services:
  fluentbit:
    build: .
    ports:
      - "24224:24224"
      - "24224:24224/udp"
  ubuntu:
    image: ubuntu
    command: [/bin/echo, "Kevcodez"]
    depends_on:
      - fluentbit
    logging:
      driver: fluentd
      options:
        tag: docker-ubuntu

In case you are wondering if fluentd as logging driver was a typo - it's not. Fluentd and Fluent Bit both use fluentd Docker Logging Driver. The forward protocol (opens new window) is used.

Simple Config

To use an alternative logging driver, we can simply pass a --log-driver argument when starting the container. This can be configured globally as well. Refer to the Docker Docs (opens new window).

Our Fluent Bit container should log something like this

[0] docker.{.ID}}: [1565471735.000000000, {"container_id"=>"50f42a398149729c3c24b621f6da2ac943a19b565c99b665e37ec5b8c8c9a3df", "container_name"=>"/zealous_proskuriakova", "source"=>"stdout", "log"=>"Kevcodez"}][2019/08/10 21:15:39] [debug][task] created task=0x7f2c838430c0 id=0 OK [2019/08/10 21:15:39][debug] [task] destroy task=0x7f2c838430c0 (task_id=0)

# Send logs to Elasticsearch

To forward the logs to Elasticsearch, we simply have to change the output plugin. Fluent Bit comes with an Elasticsearch Output Plugin (opens new window) built-in.

[OUTPUT]
    Name es
    Match **
    Host 127.0.0.1
    Port 9243
    # When Logstash_Format is enabled, the Index name is composed using a prefix and the date
    Logstash_Format True
    # HTTP_User <user>
    # HTTP_Passwd <pw>
    # Alternative time key, useful if your log entries contain an @timestamp field that is used by Elasticsearch
    # Time_Key es_time
    # If your Elasticsearch is using TLS, configure this
    # tls On
    # tls.verify Off

Let's spin up Elasticsearch, Fluent Bit and our sample ubuntu application that produces a log.

version: "3.5"
services:
  elasticsearch:
    image: elasticsearch:7.3.0
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
  fluentbit:
    build: .
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    depends_on:
      - elasticsearch
  ubuntu:
    image: ubuntu
    command: [/bin/echo, "Kevcodez"]
    depends_on:
      - fluentbit
    logging:
      driver: fluentd
      options:
        tag: docker-ubuntu

docker-compose up

Let's check if our logs arrived in Elasticsearch

curl localhost:9200/_cat/indices

yellow open logstash-2019.08.10 RevxDUH3Qpm1JTyObFhbqA 1 1 1 0 6.2kb 6.2kb

curl localhost:9200/logstash-2019.08.10/_search?pretty=true&q={'matchAll':{''}}

{
  "took": 43,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "logstash-2019.08.10",
        "_type": "flb_type",
        "_id": "ID-DfWwBIbjYepeiDcpc",
        "_score": 1.0,
        "_source": {
          "@timestamp": "2019-08-10T21:50:25.000Z",
          "container_id": "dd2cc9f525b8a59c239d8728d10e044c8ab3640e08b082e497020b3a1241124a",
          "container_name": "/config-driver-elasticsearch_ubuntu_1",
          "source": "stdout",
          "log": "Kevcodez"
        }
      }
    ]
  }
}

Since we configured the logging level in Fluent Bit to debug, we should also see the forwarding in action:

fluentbit_1 | [2019/08/10 21:50:36][debug] [out_es] HTTP Status=200 URI=/_bulk fluentbit_1 | [2019/08/10 21:50:36][debug] [out_es Elasticsearch response fluentbit_1 | {"took":424,"errors":false,"items":[{"index":{"_index":"logstash-2019.08.10","_type":"flb_type","_id":"ID-DfWwBIbjYepeiDcpc","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}}]} fluentbit_1 | [2019/08/10 21:50:36][debug] [task] destroy task=0x7f9aff4430c0 (task_id=0)

That's it. The docker application simply uses stdout, the docker logging driver forwards the logs to Fluent Bit. Fluent Bit forwards them to Elasticsearch.

Sources from the docker-compose files and configs can be found here (opens new window).

If you like this post, feel free to follow me or hit me up on Twitter (opens new window).