Install ELK Stack & Setup Spring Boot Logging with Filebeat

Posted on by


In this guide, I will show you how to install ELK stack (Elasticsearch, Logstash, Kibana) and send logs from a Spring Boot app using Logback and Filebeat.

blek



Prepare Your System

sudo apt update && sudo apt upgrade -y
sudo apt install apt-transport-https wget curl gnupg -y
Enter fullscreen mode

Exit fullscreen mode

Add the Elastic repository:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
Enter fullscreen mode

Exit fullscreen mode



Install ELK Stack



Elasticsearch

sudo apt install elasticsearch -y
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
Enter fullscreen mode

Exit fullscreen mode

Check with:

curl -k https://localhost:9200
Enter fullscreen mode

Exit fullscreen mode



Kibana

Install and start Kibana, Kibana shows dashboards and visualizations.

sudo apt install kibana -y
sudo systemctl enable kibana
sudo systemctl start kibana
Enter fullscreen mode

Exit fullscreen mode

Edit the config to allow access from outside:

sudo nano /etc/kibana/kibana.yml
# change: server.host: "0.0.0.0"
Enter fullscreen mode

Exit fullscreen mode

Open port and access:

sudo ufw allow 5601
Enter fullscreen mode

Exit fullscreen mode

Then visit http://YOUR_SERVER_IP:5601

Config Elasctic & Kibana
Open Kibana in your browser http://YOUR_SERVER_IP:5601

Find the Elastic token :

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
Enter fullscreen mode

Exit fullscreen mode

Find Kibana Verification Code :

sudo /usr/share/kibana/bin/kibana-verification-code
Enter fullscreen mode

Exit fullscreen mode

Reset the password where username is elastic

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
Enter fullscreen mode

Exit fullscreen mode



Logstash

Install and start Logstash. Logstash processes logs and sends them to Elasticsearch.

sudo apt install logstash -y
sudo systemctl enable logstash
sudo systemctl start logstash
Enter fullscreen mode

Exit fullscreen mode



How to send Spring Boot Logs to Logstash

How it works:

  • Spring Boot writes logs to file using Logback
  • Filebeat reads the log files
  • Filebeat sends logs to Logstash (or directly to Elasticsearch)
  • Logstash (optional) parses logs
  • Elasticsearch stores logs
  • Kibana displays logs



Configure Spring Boot Logback

In your Spring Boot project, create logback-spring.xml in src/main/resources/, here the Example simple config that makes your app write logs to a file that Filebeat can read.

<configuration scan="true">

    <property name="LOG_DIR" value="/var/log/taxes-backend"/>
    <property name="APP_NAME" value="taxes-backend"/>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%dHH:mm:ss %-5level [%thread] %logger36 - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>$LOG_DIR/$APP_NAME-app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>$LOG_DIR/$APP_NAME.%dyyyy-MM-dd.gz</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%dyyyy-MM-dd HH:mm:ss %-5level [%thread] %logger36 - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
    </root>

</configuration>
Enter fullscreen mode

Exit fullscreen mode

Make sure the log directory is available on both your local machine and the VM server where your Spring Boot JAR is deployed.

sudo mkdir -p /var/log/myapp
sudo chmod 777 /var/log/myapp
Enter fullscreen mode

Exit fullscreen mode



Install and Configure Filebeat

Install Filebeat:

sudo apt install filebeat -y
Enter fullscreen mode

Exit fullscreen mode

Open filebeat config file:

sudo nano /etc/filebeat/filebeat.yml
Enter fullscreen mode

Exit fullscreen mode

Edit filebeat.yml, set input to read your Spring Boot log file:

# ============================== Filebeat inputs ===============================

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/taxes-backend/taxes-backend-app.log

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: $path.config/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

# =================================== Kibana ===================================

setup.kibana:

# ------------------------------ Logstash Output -------------------------------

output.logstash:
  hosts: ["localhost:5044"]

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
Enter fullscreen mode

Exit fullscreen mode

This localhost:5044 to tell Filebeat where to send logs (Logstash or Elasticsearch).

Start Filebeat and it will read your log files and forward them:

sudo systemctl enable filebeat
sudo systemctl restart filebeat
sudo systemctl status filebeat
Enter fullscreen mode

Exit fullscreen mode



Configure Logstash Pipeline

Create a Logstash config, e.g., springboot-logstash.conf:

sudo nano /etc/logstash/conf.d/springboot.conf
Enter fullscreen mode

Exit fullscreen mode

Edit the config file :

input 
  beats 
    port => 5044
  


filter 
  grok 
    match =>  "message" => "%TIMESTAMP_ISO8601:timestamp %LOGLEVEL:level \[%DATA:thread\] %DATA:logger - %GREEDYDATA:msg" 
  

  date 
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
    timezone => "UTC"
  

  mutate 
    remove_field => ["timestamp"]
  


output 
  elasticsearch 
    hosts => ["https://localhost:9200"]
    user => "elastic"
    password => "ENTER_YOUR_ELASTIC_PASSWORD"
    ssl_verification_mode => "none"
    index => "springboot-%+YYYY.MM.dd"
Enter fullscreen mode

Exit fullscreen mode

Test the Logstash config syntax:

sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/springboot.conf
Enter fullscreen mode

Exit fullscreen mode

Start Logstash and this receives logs from Filebeat and sends them to Elasticsearch.

sudo systemctl restart logstash
Enter fullscreen mode

Exit fullscreen mode

Run this test command and make sure an index like springboot-2025 should appear in Elasticsearch.

curl -k -u elastic:ENTER_YOUR_ELASTIC_PASSWORD "https://localhost:9200/_cat/indices?v"
Enter fullscreen mode

Exit fullscreen mode



View Logs in Kibana

  • Open Kibana: http://YOUR_SERVER_IP:5601
  • Go to Stack Management
  • Create an Index Pattern that matches your logs (e.g., springboot-logs-*)
  • Use Discover to see logs
  • Build Dashboards to visualize log levels, services, errors, etc.



Source link