Introduction
We will setup Metricbeat with Elasticsearch and Kibana 9.x. If you do not have Elasticsearch and Kibana set up yet, then follow these instructions.
Based on this documentation
Requirements
A Running instance of Elasticsearch and Kibana.
An instance of another Ubuntu 24.04 server running any kind of service.
Steps
Step 1 - Install Metricbeat
On the Ubuntu machine that will run metricbeat, run these commands to download dependencies:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
apt-get update;
apt-get install -y apt-transport-https;
apt-get install -y metricbeat;
Step 2 - Enable Modules
Go to the /etc/metricbeat/modules.d directory. Pick which types of logs you want to enable logging for by renaming the corresponding *.yml.disabled to *.yml. For some of these services, you may need to add the enabled: true option to the *.yml file as well as type in appropriate connection and configuration details.
Step 3 - Create Setup Role
In Kibana, go to Stack Management > Roles > Create role. Then fill out these fields:
Role name: metricbeat_setup
Cluster privileges: monitor manage_ilm
Indices: metricbeat-*
Privileges: manage
Step 4 - Create Metricbeat User
In Kibana, go to Stack Management > Users > Create user. Then fill out these fields:
Username: metricbeat_user
Full name: metricbeat_user
Email address: anything@anything.com
Password: anything
Roles: metricbeat_setup kibana_admin ingest_admin
Press save.
Step 5 - Configure Metricbeat
Edit these fields for the /etc/metricbeat.yml
setup.kibana:
host: "https://<kibana-domain>:<kibana-port>"
ssl.certificate_authorities: ["/path/to/http_ca.crt"]
output.elasticsearch:
hosts: ["<elasticsearch-domain-or-ip>:<elasticsearch-port>"]
protocol: "https"
username: "metricbeat_user"
password: "<your metricbeat_user password>"
ssl.certificate_authorities: ["/path/to/http_ca.crt"]
setup.ilm.check_exists: false
Completed metricbeat.yml can be found here.
IMPORTANT - we are using the setup user for the initial set up and configuration. We will change the privileges later.
Then test your configuration with these commands:
/usr/share/metricbeat/bin/metricbeat test config -c /etc/metricbeat/metricbeat.yml --path.data /var/lib/metricbeat --path.home /usr/share/metricbeat
/usr/share/metricbeat/bin/metricbeat test output -c /etc/metricbeat/metricbeat.yml --path.data /var/lib/metricbeat --path.home /usr/share/metricbeat
Confirm you get success messages.
Step 6 - Setup Metricbeat
Now run this command to set up metricbeat datastreams and views in Elasticsearch and Kibana:
/usr/share/metricbeat/bin/metricbeat setup -c /etc/metricbeat/metricbeat.yml --path.data /var/lib/metricbeat --path.home /usr/share/metricbeat
Once the command finishes, go to Kibana Menu and visit Dashboard to see many pre-made dashboards.
Step 7 - Create a Publishing Role
In Kibana, go to Stack Management > Roles > Create role. Then fill out these fields:
Role name: metricbeat_publisher
Cluster privileges: monitor read_ilm
Indices: metricbeat-*
Privileges: create_doc auto_configure
Step 8 - Change Metricbeat Role
Delete the previous roles for metricbeat_user and set just this:
Roles: metricbeat_publisher
Press save.
Create API Key for User
In Kibana, go to Dev Tools > Console. Then run this command:
POST /_security/api_key/grant
{
"grant_type": "password",
"username": "metricbeat_user",
"password": "anything",
"api_key": {
"name": "metricbeat_user"
}
}
This should produce a result like:
{
"id": "J3oInZgBRvUg0VanE8wj",
"name": "metricbeat_user",
"api_key": "6Wwht52HgB-M8reGwXUM6g",
"encoded": "SjNvSW5aZ0JSdlVnMFZhbkU4d2o6Nld3aHQ1MkhnQi1NOHJlR3dYVU02Zw=="
}
Edit the /etc/metricbeat/metricbeat.yml by commenting out the metricbeat_user username and password and enabling the api_key like so:
output.elasticsearch:
...etc...
api_key: "${ES_API_KEY}"
#username: "metricbeat_user"
#password: ""
...etc...
We will be using the metricbeat keystore to load secrets for run time. Now run this command to set the ES_API_KEY keystore variable:
/usr/share/metricbeat/bin/metricbeat keystore add ES_API_KEY -c /etc/metricbeat/metricbeat.yml --path.data /var/lib/metricbeat --path.home /usr/share/metricbeat
Press enter and when prompted, paste in <id>:<api_key> where the <id> and the <api_key> are the values from the user token response you got previously.
Step 9 - Run Metricbeat
systemctl enable metricbeat;
systemctl start metricbeat;
In a moment, you should start seeing results in Kibana in either Discover, Observability, Stack Management > Index Management > Datastream, Dashboard >Select a Metricbeat dashboard.
Final Note
Anytime you enable a new module in modules.d, you need to run systemctl restart metricbeat. systemctl restart metricbeat may trigger background actions similar to setup which means the user in your metricbeat.yml should have the setup privileges mentioned in Step 3. If you do not want to manually run systemctl restart metricbeat after changes in the modules.d directory, you can have metricbeat automatically load newly enabled modules by setting this to property to true: metricbeat.config.modules.reload.enabled: true. Again, make sure the user in metricbeat.yml has the setup privileges.