Introduction
Code on Github: Elasticsearch Cluster
We will setup an Elasticsearch cluster with 5 different nodes. We will integrate Kibana. We will use self-signed certificates for Elasticsearch Transport protocol. We will use publicly signed Let's Encyrpt certificates for Elasticsearch REST API.
We assume you already made publicly signed SSL certificates (eg. Let's Encrypt) in advance. If you do not have publicly signed SSL certificates yet, then follow these instructions to generate free Let's Encrypt SSL certificates.
Requirements
This video will use 6 different VM instances from a cloud service provider. For a little variety, some machines will use Ubuntu 20.04 while others will use Ubuntu 22.04.
We have mapped the domains node1.evermight.net, node2.evermight.net, node3.evermight.net, node4.evermight.net, node5.evermight.net and kibana.evermight.net to each of the 6 machines.
Steps
Step 1 - Update Ubuntu [03:35]
All Ubuntu installations are brand new. We update the distribution as well as install some tools we typically use on all machines.
apt-get update && apt dist-upgrade -y && apt-get install -y vim curl gnupg gpg
Step 2 - Install Elasticsearch [05:10]
Install Elasticsearch on node1 to node5 by running these commands on all of them.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg;
echo 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main' | sudo tee /etc/apt/sources.list.d/elastic-8.x.list;
apt-get update;
apt-get install -y apt-transport-https;
apt-get install -y elasticsearch;
Step 3 - Configure node1.evermight.net [07:10]
Go to node1.evermight.net.
Copy over SSL certificates:
mkdir -p /etc/elasticsearch/certs/node1.evermight.net/
Upload your certificates to the directory.
Edit elasticsearch.yml
Go to the /etc/elasticsearch/elasticsearch.yml file. Edit the following fields:
cluster.name: es-demo
node.name: node1
network.host: node1.evermight.net
http.port: 9200
cluster.initial_master_nodes: ["node1"]
Completed elasticsearch.yml available on github
Note - We do NOT use the publicly signed SSL for http yet on node1.evermight.net. We need to keep the self-signed certificates in order to use command line tools for generating enrollment tokens for our nodes to use to join the cluster. We will swap out the self-signed certificates for the public certificates in the final step. If we need to use the command line tools again, then we need to swap back to the self-signed certificates.
Change ownership
chown -R elasticsearch:elasticsearch /etc/elasticsearch
Step 4 - Start node1.evermight.net [13:45]
Start elasticsearch with these commands:
systemctl enable elasticsearch;
systemctl daemon-reload;
systemctl start elasticsearch;
Reset elastic password
Reset password for elastic user with this command:
/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic
Type in a password when prompted.
Confirm elasticsearch is working with this command:
curl -k -u elastic:<password> https://node1.evermight.net:9200/_cluster/health?pretty
And you should see something like this:
Other useful commands
The following commands can also be useful for status checks and investigation:
curl -k -u elastic:<password> https://node1.evermight.net:9200/_cat/nodes?pretty
curl -k -u elastic:<password> https://node1.evermight.net:9200/_cat/master?pretty
Step 5 - Configure node2.evermight.net [16:10]
Join with enrollment token
Go to node1.evermight.net and run this command to create an enrollment token:
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
Go to node2.evermight.net and run this command to accept the enrollment token:
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <paste the token>
Confirm that the /etc/elasticsearch/elasticsearch.yml of node2.evermight.net has a discovery.seed_hosts that mentions the IP address of node1.evermight.net.
Copy over SSL certificates:
mkdir -p /etc/elasticsearch/certs/node2.evermight.net/
Upload your certificates to the directory.
Edit elasticsearch.yml
Now you must make some final edits to the /etc/elasticsearch/elasticsearch.yml of node2.evermight.net. Edit these fields:
cluster.name: es-demo
node.name: node2
network.host: node2.evermight.net
http.port: 9200
transport.host: 0.0.0.0
xpack.security.http.ssl:
enabled: true
key: certs/node2.evermight.net/privkey1.pem
certificate: certs/node2.evermight.net/fullchain1.pem
Completed elasticsearch.yml available on github
Change ownership
chown -R elasticsearch:elasticsearch /etc/elasticsearch
Step 6 - Start node2.evermight.net [26:00]
Start elasticsearch with these commands:
systemctl enable elasticsearch;
systemctl daemon-reload;
systemctl start elasticsearch;
Confirm elasticsearch is working with this command:
curl -k -u elastic:<password> https://node1.evermight.net:9200/_cluster/health?pretty
Confirm there are two nodes in the cluster with this command:
curl -k -u elastic:<password> https://node1.evermight.net:9200/_cat/nodes?pretty
You can also ping node2.evermight.net for similar results.
Step 7 - Configure and start node3, node4 and node5 [27:26]
You can repeat step 5 and step 6 for node3.evermight.net, node4.evermight.net, node5.evermight.net.
Step 8 - Clean up node1.evermight.net - initialmasternodes and publicly signed SSL [30:10]
Go back to node1.evermight.net and edit the /etc/elasticsearch/elasticsearch.yml to use the public signed SSL.
xpack.security.http.ssl:
enabled: true
key: certs/node1.evermight.net/privkey1.pem
certificate: certs/node1.evermight.net/fullchain1.pem
Also comment out the cluster.initial_master_nodes because that should be entirely determined by the cluster upon each restart.
Here is the elasticsearch.yml for subsequent restarts of node1.evermight.net.
Then restart node1.evermight.net
systemctl restart elasticsearch.service
Step 9 - Update Seed Hosts [32:15]
In the /etc/elasticsearch/elasticsearch.yml of all nodes, update the discovery.seed_hosts to reference every node:
// replace with actual ip addresses
discovery.seed_hosts:
- 1.1.1.1
- 1.1.1.2
- 1.1.1.3
- 1.1.1.4
- 1.1.1.5
This way any node that leaves the cluster will know how to rejoin the cluster by looking up the discovery.seed_hosts.
Step 10 - Install Kibana [42:30]
Run this command on the kibana.evermight.net machine to install Kibana:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg;
echo 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main' | sudo tee /etc/apt/sources.list.d/elastic-8.x.list;
apt-get install -y apt-transport-https;
apt-get install -y kibana;
Step 11 - Configure Kibana [43:30]
Copy over SSL certificates:
Go to your kibana.evermight.net server and run this command:
mkdir /etc/kibana/certs/kibana.evermight.net
Then copy your SSL certificates into the /etc/kibana/cert/kibana.evermight.net.
Edit kibana.yml
Go to the /etc/kibana/kibana.yml file. Edit the following fields:
server.port: 5601
server.host: 0.0.0.0
server.publicBaseUrl: "https://kibana.evermight.net:5601"
server.ssl.enabled: true
server.ssl.key: /etc/kibana/certs/kibana.evermight.net/priv1.key
server.ssl.certificate: /etc/kibana/certs/kibana.evermight.net/fullchain1.pem
elasticsearch.hosts: ["https://node1.evermight.net:9200"]
elasticsearch.ssl.verificationMode: full
Note - Only add hosts that have the /etc/elasticsearch/service_tokens to the elasticsearch.hosts. Hosts without the service_tokens file will not be able to authenticate Kibana.
Create Service Token
Run this command from any server server:
curl -X POST -u elastic:<password> https://node1.evermight.net:9200/_security/service/elastic/kibana/credential/token/kibana_token
Copy the token that you see.
Run this command on the Kibana server:
/usr/share/kibana/bin/kibana-keystore add elasticsearch.serviceAccountToken
Paste in the token after the prompt.
Step 12 - Start Kibana [52:30]
systemctl enable kibana;
systemctl start kibana;
Now you can visit https://kibana.evermight.net:5601/ and login with elastic and the password from Step 4.