Configuring Kibana and ElasticSearch for Log Analysis with Fluentd on Docker Swarm

Table of Contents

Using Kibana and ElasticSearch for Log Analysis with Fluentd on Docker Swarm

Using Kibana and ElasticSearch for Log Analysis with Fluentd on Docker Swarm

Introduction

In my previous post, I talked about how to configure fluentd for logging for multiple Docker containers. The post explained how to create a single file for each micro service irrespective of its multiple instances it could have.
However, Log files have limitations it is not easy to extract analysis or find any trends.

Elastic Search and Splunk have become very popular in recent years as they give you allow you to events in real-time, visualise trends and search through logs.

Elastic Search and Kibana

Elastic Search is an open source search engine based on Apache Lucene.It is an extremely fast search engine and is commonly used for log analytics, full-text search and much more.
Along with Kibana, which is a visualisation tool, Elasticsearch can be used for real-time analytics. With Kibana you can create intuitive charts and reports, filters, aggregations and trends based on data.

Changing the fluent.conf

Since this post is continuation of previous post, I will show you how to modify the fluent.conf for elastic search changes

All we need to do is that we need to add another “store” block like below

<store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix logstash
    logstash_dateformat %Y%m%d
    include_tag_key true
    tag_key @log_name
    flush_interval 1s
  </store>

<store>

@type elasticsearch

host elasticsearch

port 9200

logstash_format true

logstash_prefix logstash

logstash_dateformat %Y%m%d

include_tag_key true

tag_key @log_name

flush_interval 1s

</store>

In the above config, we are telling that elastic search is running on port 9200 and the host is elasticsearch (which is docker container name). Also we have defined the general Date format and flush_interval has been set to 1s which tells fluentd to send records to elasticsearch after every 1sec.

This is how the complete configuration will look like

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match tutum>
  @type copy
   <store>
    @type file
    path /fluentd/log/tutum.*.log
    time_slice_format %Y%m%d
    time_slice_wait 10m
    time_format %Y%m%dT%H%M%S%z
    compress gzip
    utc
    format json
  </store>
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix logstash
    logstash_dateformat %Y%m%d
    include_tag_key true
    tag_key @log_name
    flush_interval 1s
  </store>
</match>
<match visualizer>
  @type copy
   <store>
    @type file
    path /fluentd/log/visualizer.*.log
    time_slice_format %Y%m%d
    time_slice_wait 10m
    time_format %Y%m%dT%H%M%S%z
    compress gzip
    utc
    format json
  </store>
    <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix logstash
    logstash_dateformat %Y%m%d
    include_tag_key true
    tag_key @log_name
    flush_interval 1s
  </store>
</match>

@type forward

port 24224

bind 0.0.0.0

</source>

@type copy

<store>

@type file

path /fluentd/log/tutum.*.log

time_slice_format %Y%m%d

time_slice_wait 10m

time_format %Y%m%dT%H%M%S%z

compress gzip

utc

format json

</store>

<store>

@type elasticsearch

host elasticsearch

port 9200

logstash_format true

logstash_prefix logstash

logstash_dateformat %Y%m%d

include_tag_key true

tag_key @log_name

flush_interval 1s

</store>

</match>

@type copy

<store>

@type file

path /fluentd/log/visualizer.*.log

time_slice_format %Y%m%d

time_slice_wait 10m

time_format %Y%m%dT%H%M%S%z

compress gzip

utc

format json

</store>

<store>

@type elasticsearch

host elasticsearch

port 9200

logstash_format true

logstash_prefix logstash

logstash_dateformat %Y%m%d

include_tag_key true

tag_key @log_name

flush_interval 1s

</store>

</match>

Create Dockerfile with our custom configuration

So the next step is to create a custom image of fluentd which has the above configuration file.
Save above file as fluent.conf in a folder named conf and then create a file called DockerFile at the same level as conf folder

# fluentd/Dockerfile
FROM fluent/fluentd:v0.12-debian
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-rdoc", "--no-ri", "--version", "1.9.2"]
RUN rm /fluentd/etc/fluent.conf
COPY ./conf/fluent.conf /fluentd/etc

# fluentd/Dockerfile

FROM fluent/fluentd:v0.12-debian

RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-rdoc", "--no-ri", "--version", "1.9.2"]

RUN rm /fluentd/etc/fluent.conf

COPY ./conf/fluent.conf /fluentd/etc

In this Docker file as you can see we are replacing the fluent.conf in the base image with the version of ours and also installing elastic search plugin

Now let us create a Docker image by
run "docker build -t ##YourREPOname##/myfluentd:latest ."

and then push it to the docker hub repository
"docker push ##YourREPOname##/myfluentd"

Elastic Search and Kibana Docker Image

This is how the YAML configuration for ElasticSearch looks like

  elasticsearch:
    image: elasticsearch
    ports:
      - "9200:9200"
    networks:
      - net
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    logging:
        driver: "json-file"
        options:
          max-size: 10M
          max-file: 1  
    deploy:
      restart_policy:
        condition: on-failure
        delay: 20s
        max_attempts: 3
        window: 120s
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
      update_config:
        delay: 2s
      resources:
        limits:
          memory: 1000M
    volumes:
      - ./esdata:/usr/share/elasticsearch/data

elasticsearch:

image: elasticsearch

ports:

- "9200:9200"

networks:

- net

environment:

- bootstrap.memory_lock=true

- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

logging:

driver: "json-file"

options:

max-size: 10M

max-file: 1

deploy:

restart_policy:

condition: on-failure

delay: 20s

max_attempts: 3

window: 120s

mode: replicated

replicas: 1

placement:

constraints: [node.role == manager]

update_config:

delay: 2s

resources:

limits:

memory: 1000M

volumes:

- ./esdata:/usr/share/elasticsearch/data

Things to note here for this elastic search configuration file

The name of the container is set to elasticsearch, this is same to whta has been mentioned in fluent.conf
Port 9200 has been exposed to access elastic search locally too
The environment variables are important here as they restrict the max heap space this container can have and disable any Elasticsearch memory from being swapped out.
Using Logging Driver as JSON File as I want to restrict the log file size
Storing the elastic search data on a directory called “esdata” so data can persist between container restarts

This is how the YAML configuration for Kibana looks like

  kibana:
    image: kibana
    ports:
      - "5601:5601"
    networks:
      - net
    logging:
        driver: "json-file"
        options:
           max-size: 10M
           max-file: 1        
    deploy:
      restart_policy:
        condition: on-failure
        delay: 20s
        max_attempts: 3
        window: 120s
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
      update_config:
        delay: 2s

kibana:

image: kibana

ports:

- "5601:5601"

networks:

- net

logging:

driver: "json-file"

options:

max-size: 10M

max-file: 1

deploy:

restart_policy:

condition: on-failure

delay: 20s

max_attempts: 3

window: 120s

mode: replicated

replicas: 1

placement:

constraints: [node.role == manager]

update_config:

delay: 2s

Things to note here for this kibana configuration file

The port 5601 has been exposed locally to access Kibana dashboard

Complete Config file

So now we need a complete docker-compose file which will have whoami service with multiple instances, docker visualiser service along with elastic, kibana and fluentd services.
This is how the complete files looks like

version: "3"
 
services:
       
  whoami:
    image: tutum/hello-world
    networks:
      - net
    ports:
      - "80:80"
    logging:
      driver: "fluentd"# Logging Driver
      options:
        tag: tutum    # TAG 
    deploy:
      restart_policy:
           condition: on-failure
           delay: 20s
           max_attempts: 3
           window: 120s
      mode: replicated
      replicas: 4
      placement:
        constraints: [node.role == worker]
      update_config:
        delay: 2s
 
  vizualizer:
      image: dockersamples/visualizer
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock
      ports:
        - "8080:8080"
      networks:
        - net
      logging:
        driver: "fluentd"
        options:
         tag: visualizer   #TAG 
      deploy:
          restart_policy:
             condition: on-failure
             delay: 20s
             max_attempts: 3
             window: 120s
          mode: replicated # one container per manager node
          replicas: 1
          update_config:
            delay: 2s
          placement:
             constraints: [node.role == manager]
 
        
  fluentd:
    image: abhishekgaloda/myfluentd
    volumes:
      - ./Logs:/fluentd/log
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    networks:
      - net
    deploy:
      restart_policy:
           condition: on-failure
           delay: 20s
           max_attempts: 3
           window: 120s
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
      update_config:
        delay: 2s
 
  elasticsearch:
    image: elasticsearch
    ports:
      - "9200:9200"
    networks:
      - net
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    logging:
        driver: "json-file"
        options:
          max-size: 10M
          max-file: 1  
    deploy:
      restart_policy:
        condition: on-failure
        delay: 20s
        max_attempts: 3
        window: 120s
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
      update_config:
        delay: 2s
      resources:
        limits:
          memory: 1000M
    volumes:
      - ./esdata:/usr/share/elasticsearch/data    
      
  kibana:
    image: kibana
    ports:
      - "5601:5601"
    networks:
      - net
    logging:
        driver: "json-file"
        options:
           max-size: 10M
           max-file: 1        
    deploy:
      restart_policy:
        condition: on-failure
        delay: 20s
        max_attempts: 3
        window: 120s
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
      update_config:
        delay: 2s  
 
networks:
  net:

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

version: "3"

services:

whoami:

image: tutum/hello-world

networks:

- net

ports:

- "80:80"

logging:

driver: "fluentd"# Logging Driver

options:

tag: tutum # TAG

deploy:

restart_policy:

condition: on-failure

delay: 20s

max_attempts: 3

window: 120s

mode: replicated

replicas: 4

placement:

constraints: [node.role == worker]

update_config:

delay: 2s

vizualizer:

image: dockersamples/visualizer

volumes:

- /var/run/docker.sock:/var/run/docker.sock

ports:

- "8080:8080"

networks:

- net

logging:

driver: "fluentd"

options:

tag: visualizer #TAG

deploy:

restart_policy:

condition: on-failure

delay: 20s

max_attempts: 3

window: 120s

mode: replicated # one container per manager node

replicas: 1

update_config:

delay: 2s

placement:

constraints: [node.role == manager]

fluentd:

image: abhishekgaloda/myfluentd

volumes:

- ./Logs:/fluentd/log

ports:

- "24224:24224"

- "24224:24224/udp"

networks:

- net

deploy:

restart_policy:

condition: on-failure

delay: 20s

max_attempts: 3

window: 120s

mode: replicated

replicas: 1

placement:

constraints: [node.role == manager]

update_config:

delay: 2s

elasticsearch:

image: elasticsearch

ports:

- "9200:9200"

networks:

- net

environment:

- bootstrap.memory_lock=true

- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

logging:

driver: "json-file"

options:

max-size: 10M

max-file: 1

deploy:

restart_policy:

condition: on-failure

delay: 20s

max_attempts: 3

window: 120s

mode: replicated

replicas: 1

placement:

constraints: [node.role == manager]

update_config:

delay: 2s

resources:

limits:

memory: 1000M

volumes:

- ./esdata:/usr/share/elasticsearch/data

kibana:

image: kibana

ports:

- "5601:5601"

networks:

- net

logging:

driver: "json-file"

options:

max-size: 10M

max-file: 1

deploy:

restart_policy:

condition: on-failure

delay: 20s

max_attempts: 3

window: 120s

mode: replicated

replicas: 1

placement:

constraints: [node.role == manager]

update_config:

delay: 2s

networks:

net:

You can run above services on docker swarm by using below command, make sure you save the file by the name docker-swarm.yml

docker stack deploy -c docker-swarm.yml test

docker stack deploy -c docker-swarm.yml test

Accessing kibana and elasticsearch

Once you make sure that all services are up and running using

docker service ls

docker service ls

you can access Kibana on
http://##domain#Or#IP##:5601/
Once You see something similar like below click on Create Button and then Discover on top left, you should see some bars indicating logs

Now if you click on http://##domain#Or#IP##/hello the whomai container will generate some logs which should appear on Kibana provided the right time has been chosen and auto-refresh has been enabled. See the screenshot below.