Table of Contents
Using Kibana and ElasticSearch for Log Analysis with Fluentd on Docker Swarm
Introduction
However, Log files have limitations it is not easy to extract analysis or find any trends.
Elastic Search and Splunk have become very popular in recent years as they give you allow you to events in real-time, visualise trends and search through logs.
Elastic Search and Kibana
Elastic Search is an open source search engine based on Apache Lucene.It is an extremely fast search engine and is commonly used for log analytics, full-text search and much more.
Along with Kibana, which is a visualisation tool, Elasticsearch can be used for real-time analytics. With Kibana you can create intuitive charts and reports, filters, aggregations and trends based on data.
Changing the fluent.conf
Since this post is continuation of previous post, I will show you how to modify the fluent.conf for elastic search changes
All we need to do is that we need to add another “store” block like below
1 2 3 4 5 6 7 8 9 10 11 12 13 | <store> @type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix logstash logstash_dateformat %Y%m%d include_tag_key true tag_key @log_name flush_interval 1s </store> |
In the above config, we are telling that elastic search is running on port 9200 and the host is elasticsearch (which is docker container name). Also we have defined the general Date format and flush_interval has been set to 1s which tells fluentd to send records to elasticsearch after every 1sec.
This is how the complete configuration will look like
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | <source> @type forward port 24224 bind 0.0.0.0 </source> <match tutum> @type copy <store> @type file path /fluentd/log/tutum.*.log time_slice_format %Y%m%d time_slice_wait 10m time_format %Y%m%dT%H%M%S%z compress gzip utc format json </store> <store> @type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix logstash logstash_dateformat %Y%m%d include_tag_key true tag_key @log_name flush_interval 1s </store> </match> <match visualizer> @type copy <store> @type file path /fluentd/log/visualizer.*.log time_slice_format %Y%m%d time_slice_wait 10m time_format %Y%m%dT%H%M%S%z compress gzip utc format json </store> <store> @type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix logstash logstash_dateformat %Y%m%d include_tag_key true tag_key @log_name flush_interval 1s </store> </match> |
Create Dockerfile with our custom configuration
So the next step is to create a custom image of fluentd which has the above configuration file.
Save above file as fluent.conf in a folder named conf and then create a file called DockerFile at the same level as conf folder
1 2 3 4 5 6 7 | # fluentd/Dockerfile FROM fluent/fluentd:v0.12-debian RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-rdoc", "--no-ri", "--version", "1.9.2"] RUN rm /fluentd/etc/fluent.conf COPY ./conf/fluent.conf /fluentd/etc |
In this Docker file as you can see we are replacing the fluent.conf in the base image with the version of ours and also installing elastic search plugin
Now let us create a Docker image by
run "docker build -t ##YourREPOname##/myfluentd:latest ."
and then push it to the docker hub repository
"docker push ##YourREPOname##/myfluentd"
Elastic Search and Kibana Docker Image
This is how the YAML configuration for ElasticSearch looks like
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | elasticsearch: image: elasticsearch ports: - "9200:9200" networks: - net environment: - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" logging: driver: "json-file" options: max-size: 10M max-file: 1 deploy: restart_policy: condition: on-failure delay: 20s max_attempts: 3 window: 120s mode: replicated replicas: 1 placement: constraints: [node.role == manager] update_config: delay: 2s resources: limits: memory: 1000M volumes: - ./esdata:/usr/share/elasticsearch/data |
Things to note here for this elastic search configuration file
- The name of the container is set to elasticsearch, this is same to whta has been mentioned in fluent.conf
- Port 9200 has been exposed to access elastic search locally too
- The environment variables are important here as they restrict the max heap space this container can have and disable any Elasticsearch memory from being swapped out.
- Using Logging Driver as JSON File as I want to restrict the log file size
- Storing the elastic search data on a directory called “esdata” so data can persist between container restarts
This is how the YAML configuration for Kibana looks like
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | kibana: image: kibana ports: - "5601:5601" networks: - net logging: driver: "json-file" options: max-size: 10M max-file: 1 deploy: restart_policy: condition: on-failure delay: 20s max_attempts: 3 window: 120s mode: replicated replicas: 1 placement: constraints: [node.role == manager] update_config: delay: 2s |
Things to note here for this kibana configuration file
- The port 5601 has been exposed locally to access Kibana dashboard
Complete Config file
So now we need a complete docker-compose file which will have whoami service with multiple instances, docker visualiser service along with elastic, kibana and fluentd services.
This is how the complete files looks like
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | version: "3" services: whoami: image: tutum/hello-world networks: - net ports: - "80:80" logging: driver: "fluentd"# Logging Driver options: tag: tutum # TAG deploy: restart_policy: condition: on-failure delay: 20s max_attempts: 3 window: 120s mode: replicated replicas: 4 placement: constraints: [node.role == worker] update_config: delay: 2s vizualizer: image: dockersamples/visualizer volumes: - /var/run/docker.sock:/var/run/docker.sock ports: - "8080:8080" networks: - net logging: driver: "fluentd" options: tag: visualizer #TAG deploy: restart_policy: condition: on-failure delay: 20s max_attempts: 3 window: 120s mode: replicated # one container per manager node replicas: 1 update_config: delay: 2s placement: constraints: [node.role == manager] fluentd: image: abhishekgaloda/myfluentd volumes: - ./Logs:/fluentd/log ports: - "24224:24224" - "24224:24224/udp" networks: - net deploy: restart_policy: condition: on-failure delay: 20s max_attempts: 3 window: 120s mode: replicated replicas: 1 placement: constraints: [node.role == manager] update_config: delay: 2s elasticsearch: image: elasticsearch ports: - "9200:9200" networks: - net environment: - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" logging: driver: "json-file" options: max-size: 10M max-file: 1 deploy: restart_policy: condition: on-failure delay: 20s max_attempts: 3 window: 120s mode: replicated replicas: 1 placement: constraints: [node.role == manager] update_config: delay: 2s resources: limits: memory: 1000M volumes: - ./esdata:/usr/share/elasticsearch/data kibana: image: kibana ports: - "5601:5601" networks: - net logging: driver: "json-file" options: max-size: 10M max-file: 1 deploy: restart_policy: condition: on-failure delay: 20s max_attempts: 3 window: 120s mode: replicated replicas: 1 placement: constraints: [node.role == manager] update_config: delay: 2s networks: net: |
You can run above services on docker swarm by using below command, make sure you save the file by the name docker-swarm.yml
1 2 3 | docker stack deploy -c docker-swarm.yml test |
Accessing kibana and elasticsearch
Once you make sure that all services are up and running using
1 2 3 | docker service ls |
you can access Kibana on
http://##domain#Or#IP##:5601/
Once You see something similar like below click on Create Button and then Discover on top left, you should see some bars indicating logs
Now if you click on http://##domain#Or#IP##/hello the whomai container will generate some logs which should appear on Kibana provided the right time has been chosen and auto-refresh has been enabled. See the screenshot below.
Kibana offers a lot of ways to create the visualization like charts, graphs etc for log analysis which can be explored further.
Drop in suggestions or comments for any feedback.
Follow this Video for demonstration