Introduction
Code on Github: Elasticsearch and Beats
We will setup Packetbeat with Elasticsearch and Kibana. If you do not have Elasticsearch and Kibana set up yet, then follow these instructions.
This video assumes you are using Publicly Signed Certificates. If you are using Self Signed Certificates, go here TBD.
Requirements
A running instance of Elasticsearch and Kibana.
An instance of another Ubuntu 20.04 server running any kind of service.
Steps
Step 1 - Download Packetbeat [01:15]
On the Ubuntu machine that will run packetbeat, run these commands to download dependencies:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg;
sudo apt-get install -y apt-transport-https;
echo 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main' | sudo tee /etc/apt/sources.list.d/elastic-8.x.list;
sudo apt-get update && sudo apt-get install -y packetbeat;
Step 2 - Configure Packetbeat [02:21]
Edit these fields for the /etc/packetbeat/packetbeat.yml.
setup.dashboards.enabled: true
setup.kibana:
host: "https://<kibana-domain>:<kibana-port>"
output.elasticsearch:
hosts: ["<elasticsearch-domain>:<elasticsearch-port>"]
protocol: "https"
username: "elastic"
password: "your elastic password"
Completed packetbeat.yml can be found here
IMPORTANT - we are using the elastic super user for the initial set up and configuration. We will downgrade the privileges later.
Then test your configuration with these commands:
/usr/share/packetbeat/bin/packetbeat test config -c /etc/packetbeat/packetbeat.yml --path.data /var/lib/packetbeat --path.home /usr/share/packetbeat
/usr/share/packetbeat/bin/packetbeat test output -c /etc/packetbeat/packetbeat.yml --path.data /var/lib/packetbeat --path.home /usr/share/packetbeat
You should see something like this:
Verify packetbeat configuration
Step 3 - Setup Packetbeat [9:38]
Now run this command to set up packetbeat datastreams and views in Elasticsearch and Kibana:
/usr/share/packetbeat/bin/packetbeat setup -c /etc/packetbeat/packetbeat.yml --path.data /var/lib/packetbeat --path.home /usr/share/packetbeat
Once the command finishes, go to Kibana Menu and visit Dashboard to see many pre-made dashboards.
Step 4 - Create a Publishing User [11:05]
Create Role
In Kibana, go to Stack Management > Roles > Create role. Then fill out these fields:
Role name: packetbeat-publisher
Cluster privileges: monitor, read_ilm
Indices: packetbeat-*
Privileges: create_doc
Create User
In Kibana, go to Stack Management > Users > Create user. Then fill out these fields:
Username: packetbeat-publisher
Full name: packetbeat-publisher
Email address: anything@anything.com
Password: anything
Roles: packetbeat-publisher, editor
Create API Key for User
In Kibana, go to Dev Tools > Console. Then run this command:
POST /_security/api_key/grant
{
"grant_type": "password",
"username": "packetbeat-publisher",
"password": "anything",
"api_key": {
"name": "packetbeat-publisher"
}
}
This should produce a result like:
Packetbeat user token
Edit the /etc/packetbeat/packetbeat.yml by commenting out the elastic username and password and enabling the api_key like so:
output.elasticsearch:
...etc...
api_key: "${ES_API_KEY}"
#username: "elastic"
#password: ""
...etc...
We will be using the packetbeat keystore to load secrets for run time. Now run this command to set the ES_API_KEY keystore variable:
/usr/share/packetbeat/bin/packetbeat keystore add ES_API_KEY -c /etc/packetbeat/packetbeat.yml --path.data /var/lib/packetbeat --path.home /usr/share/packetbeat
Press enter and when prompted, paste in <id>:<api_key> where the <id> and the <api_key> are the values from the user token response you got previously.
Step 5 - Run Packetbeat [17:37]
systemctl enable packetbeat.service;
systemctl start packetbeat.service;
In a moment, you should start seeing results in Kibana in either Discover, Observability, Stack Management > Index Management > Datastream, Dashboard > Select a Packetbeat dashboard.
Step 6 - Setup GeoIP in elastic
In Kibana, go to Dev Tools > Console. Then run this command:
GET _ingest/geoip/stats
If the successful_downlads field shows 1 then you can totally skip all the next few processes, but if GeoIp fails to download in elastic, it would produce a result like:
Packetbeat GeoIP checks
Disable the GeoIP auto-download on elasticsearch
Go to etc/elasticsearch/ directory and open the elasticsearch.yml file with
vi elasticsearch.yml
Add the below line to the file
ingest.geoip.downloader.enabled: false
then restart elasticsearch
systemctl restart elasticsearch.service
confirm elasticsearch doesn't try to download GeoIP by running the below command again in Dev Tools > Consiole:
GET _ingest/geoip/stats
You should see something like this:
Allow Elasticsearch to download GeoIP
Go to etc/elasticsearch/ directory and open the elasticsearch.yml file with
vi elasticsearch.yml
And remove the below line that was previously added to the file.
ingest.geoip.downloader.enabled: false
Proceed to restart elasticsearch for changes to be reflected.
systemctl restart elasticsearch.service
Confirm it works by running the below command in the Dev Tools > Console:
GET _ingest/geoip/stats
You should get an output like this:
Packetbeat GeoIP Successful Download
Configure GeoIP
Define an ingest pipeline with the below command in the Dev Tools > Console:
PUT _ingest/pipeline/geoip-info
{
"description": "Add geoip info",
"processors": [
{
"geoip": {
"field": "client.ip",
"target_field": "client.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "client.ip",
"target_field": "client.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "source.ip",
"target_field": "source.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "source.ip",
"target_field": "source.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "destination.ip",
"target_field": "destination.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "destination.ip",
"target_field": "destination.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "server.ip",
"target_field": "server.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "server.ip",
"target_field": "server.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "host.ip",
"target_field": "host.geo",
"ignore_missing": true
}
},
{
"rename": {
"field": "server.as.asn",
"target_field": "server.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "server.as.organization_name",
"target_field": "server.as.organization.name",
"ignore_missing": true
}
},
{
"rename": {
"field": "client.as.asn",
"target_field": "client.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "client.as.organization_name",
"target_field": "client.as.organization.name",
"ignore_missing": true
}
},
{
"rename": {
"field": "source.as.asn",
"target_field": "source.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "source.as.organization_name",
"target_field": "source.as.organization.name",
"ignore_missing": true
}
},
{
"rename": {
"field": "destination.as.asn",
"target_field": "destination.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "destination.as.organization_name",
"target_field": "destination.as.organization.name",
"ignore_missing": true
}
}
]
}
Then run the below command to confirm that everything that was sent with the put request was correct:
GET _ingest/pipeline/geoip-info
Edit the /etc/packetbeat/packetbeat.yml by adding pipeline: geoip-info like so:
output.elasticsearch:
...etc...
api_key: "${ES_API_KEY}"
pipeline: geoip-info
#username: "elastic"
#password: ""
...etc...
And then restart packetbeat using the below command on your terminal:
systemctl restart packetbeat.service