Filebeat with Elasticsearch 8.x - Part 1: Install & Secure

Published on 2023-01-04

« Back to all documents Contact Us
Filebeat with Elasticsearch 8.x - Part 1: Install & Secure


Code on Github: Elasticsearch and Beats

We will setup Filebeat with Elasticsearch and Kibana. If you do not have Elasticsearch and Kibana set up yet, then follow these instructions.

This video assumes you are using Publicly Signed Certificates. If you are using Self Signed Certificates, go here TBD.


A Running instance of Elasticsearch and Kibana.

An instance of another Ubuntu 20.04 server running any kind of service.


Step 1 - Download Filebeat [01:10]

On the Ubuntu machine that will run filebeat, run these commands to download dependencies:

wget -qO - | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg; echo 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] stable main' | sudo tee /etc/apt/sources.list.d/elastic-8.x.list; apt-get install -y apt-transport-https; apt-get install -y filebeat;

Step 2 - Configure Filebeat [02:20]

Edit these fields for the /etc/filebeat.yml

filebeat.inputs: ...etc... - type: filestream ...etc... enabled: true ...etc... filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true # 10s reload for demonstration purposes reload.period: 10s setup.kibana: host: "https://<kibana-domain>:<kibana-port>" output.elasticsearch: hosts: ["<elasticsearch-domain>:<elasticsearch-port>"] protocol: "https" username: "elastic" password: "<your elastic password>"

Completed filebeat.yml can be found here.

IMPORTANT - we are using the elastic super user for the initial set up and configuration. We will downgrade the privileges later.

Then test your configuration with these commands:

/usr/share/filebeat/bin/filebeat test config -c /etc/filebeat/filebeat.yml /var/lib/filebeat --path.home /usr/share/filebeat /usr/share/filebeat/bin/filebeat test output -c /etc/filebeat/filebeat.yml /var/lib/filebeat --path.home /usr/share/filebeat

Confirm you get success messages.

Step 3 - Setup Filebeat [11:12]

Now run this command to set up filebeat datastreams and views in Elasticsearch and Kibana:

/usr/share/filebeat/bin/filebeat setup -c /etc/filebeat/filebeat.yml /var/lib/filebeat --path.home /usr/share/filebeat

Once the command finishes, go to Kibana Menu and visit Dashboard to see many pre-made dashboards.

Step 4 - Create a Publishing User [12:58]

Create Role

In Kibana, go to Stack Management > Roles > Create role. Then fill out these fields:

Role name: filebeat-publisher Cluster privileges: monitor read_ilm read_pipeline Indices: filebeat-* Privileges: create_doc

Create User

In Kibana, go to Stack Management > Users > Create user. Then fill out these fields:

Username: filebeat-publisher Full name: filebeat-publisher Email address: Password: anything Roles: filebeat-publisher editor

Create API Key for User

In Kibana, go to Dev Tools > Console. Then run this command:

POST /_security/api_key/grant { "grant_type": "password", "username": "filebeat-publisher", "password": "anything", "api_key": { "name": "filebeat-publisher" } }

This should produce a result like:

Filebeat user token Filebeat user token

Typing mistake: the name metric should show filebeat-publisher instead

Edit the /etc/filebeat/filebeat.yml by commenting out the elastic username and password and enabling the api_key like so:

output.elasticsearch: ...etc... api_key: "${ES_API_KEY}" #username: "elastic" #password: "" ...etc...

We will be using the filebeat keystore to load secrets for run time. Now run this command to set the ES_API_KEY keystore variable:

/usr/share/filebeat/bin/filebeat keystore add ES_API_KEY -c /etc/filebeat/filebeat.yml /var/lib/filebeat --path.home /usr/share/filebeat

Press enter and when prompted, paste in <id>:<api_key> where the <id> and the <api_key> are the values from the user token response you got previously.

Step 5 - Run Filebeat [19:00]

systemctl enable filebeat; systemctl start filebeat;

In a moment, you should start seeing results in Kibana in either Discover, Observability, Stack Management > Index Management > Datastream, Dashboard >Select a Filebeat dashboard.

Step 6 - Enable Other Log Modules [22:30]

At this point, there are not too many interesting logs to review. We can go back to the Filebeat server , go to the /etc/filebeat/modules.d directory, and rename the relevant *.yml.disabled to *.yml file to instruct Filebeat to ship data to Elasticsearch. For some of these services, you may need to add the enabled: true option to the *.yml file as well as type in appropriate connection and configuration details.

If you need any assistance, email us through our Contact Form.