3 node Elasticseach cluster setup with x-pack

Skip to end of metadata

• Created by developers, last modified on May 30, 2018

Go to start of metadata

[1] Install OpenJDK 8 to configure Java development environment.

Install OpenJDK 8.
Oracle JDK includes compiler but compiler for OpenJDK 8 is included in openjdk-devel.

yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel

cat > /etc/profile.d/java8.sh <<EOF
export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac)))))
export PATH=\$PATH:\$JAVA_HOME/bin
export CLASSPATH=.:\$JAVA_HOME/jre/lib:\$JAVA_HOME/lib:\$JAVA_HOME/lib/tools.jar
EOF

[2] Install and Run Elasticsearch.

vi /etc/yum.repos.d/elasticsearch.repo

# create new
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum -y install elasticsearch
systemctl start elasticsearch
ystemctl enable elasticsearch

# verify working

curl http://127.0.0.1:9200

{
"name" : "bV0GuFK",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "JprAV7AzTtSNL0De6BWwEQ",
"version" : {
"number" : "6.2.1",
"build_hash" : "7299dc3",
"build_date" : "2018-02-07T19:34:26.990113Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

[3]Elastic Stack 6 : Three node Elasticsearch Cluster setup and configuration

Install and Run Elasticsearch on all Nodes.
Change settings on all Nodes like follows.

[root@915423-elastic1 ~]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: newfrontierdata
node.name: ${HOSTNAME}
network.host: 192.168.60.4
discovery.zen.ping.unicast.hosts: ["192.168.60.4", "192.168.60.5","192.168.60.5"]

discovery.zen.minimum_master_nodes: 2
path.data: /data/elkdata
path.logs: /data/elklogs
xpack.security.enabled: true
bootstrap.system_call_filter: false
[root@915423-elastic1 ~]#

[root@915453-elastic2 ~]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: newfrontierdata
node.name: ${HOSTNAME}
network.host: 172.24.32.217
discovery.zen.ping.unicast.hosts: ["192.168.60.4", "192.168.60.5,"192.168.60.6"]
discovery.zen.minimum_master_nodes: 2
path.data: /data/elkdata
path.logs: /data/elklogs
xpack.security.enabled: true
bootstrap.system_call_filter: false

[root@915454-elastic3 ~]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: newfrontierdata
node.name: ${HOSTNAME}
network.host: 172.24.32.218
discovery.zen.ping.unicast.hosts: ["192.168.60.4", "192.168.60.5","192.168.60.6"]
discovery.zen.minimum_master_nodes: 2
path.data: /data/elkdata
path.logs: /data/elklogs
xpack.security.enabled: true
bootstrap.system_call_filter: false
[root@915454-elastic3 ~]#

Verification methood

[root@915423-elastic1 ~]# curl http://172.24.32.218:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.24.32.218 12 96 64 3.45 3.78 3.72 mdi - 915454-elastic3.newfrontierdata.com
172.24.32.217 11 96 89 3.97 3.95 3.86 mdi * 915453-elastic2.newfrontierdata.com
172.24.32.209 8 94 70 3.70 3.53 3.65 mdi - 915423-elastic1.newfrontierdata.com
[root@915423-elastic1 ~]#

[root@915453-elastic2 ~]# curl http://172.24.32.218:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.24.32.218 9 96 79 3.39 3.73 3.71 mdi - 915454-elastic3.newfrontierdata.com
172.24.32.217 11 97 79 3.37 3.81 3.82 mdi * 915453-elastic2.newfrontierdata.com
172.24.32.209 11 97 87 3.44 3.48 3.63 mdi - 915423-elastic1.newfrontierdata.com
[root@915453-elastic2 ~]#

[root@915454-elastic3 ~]# curl http://172.24.32.218:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.24.32.218 10 97 83 3.03 3.60 3.66 mdi - 915454-elastic3.newfrontierdata.com
172.24.32.217 5 98 77 3.25 3.73 3.79 mdi * 915453-elastic2.newfrontierdata.com
172.24.32.209 11 98 77 3.33 3.46 3.62 mdi - 915423-elastic1.newfrontierdata.com
[root@915454-elastic3 ~]#

[4] Elastic Stack 6 : Install Kibana

yum -y install kibana
vi /etc/kibana/kibana.yml
# line 7: uncomment and change (listen all)
server.host: "0.0.0.0"
# line 18: uncomment and change (specify own hostname)
server.name: "915454-elastic3.newfrontierdata.com"

systemctl start kibana
systemctl enable kibana

[5]Elastic Stack 6 : Install Logstash

yum -y install logstash
systemctl start logstash
systemctl enable logstash

[6] Installing X-pack in Elasticsearch 

The following diagram provides an overview of the steps that are required to set up X-Pack on Elasticsearch:

To install X-Pack in Elasticsearch:

Install X-Pack for Elasticsearch on all Elasticsearch Nodes in Cluster

/usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack

Confirm that you want to grant X-Pack additional permissions.

Tip
Specify the --batch option when running the install command to automatically grant these permissions and bypass these install prompts.

X-Pack needs these permissions to set the threat context loader during install so Watcher can send email notifications.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.security.SecurityPermission createPolicy.JavaPolicy
* java.security.SecurityPermission getPolicy
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setPolicy
* java.util.PropertyPermission * read,write
* java.util.PropertyPermission sun.nio.ch.bugLevel write
* javax.net.ssl.SSLPermission setHostnameVerifier
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
X-Pack requires permissions to enable Elasticsearch to launch the machine learning analytical engine. The native controller ensures that the launched process is a valid machine learning component. Once launched, communications between the machine learning processes and Elasticsearch are limited to the operating system user that Elasticsearch runs as.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin forks a native controller @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
This plugin launches a native controller that is not subject to
the Java security manager nor to system call filters.

Continue with installation? [y/N]y

Start Elasticsearch.

/usr/share/elasticsearch/bin/elasticsearch

Set the passwords for all built-in users. The setup-passwords command is the simplest method to set the built-in users' passwords for the first time.

For example, you can run the command in an "interactive" mode, which prompts you to enter new passwords for the elastic, kibana, and logstash_system users:

/usr/share/elasticsearch/bin/x-pack/setup-passwords interactive

[7] Installing X-pack in Kibana

The following diagram provides an overview of the steps that are required to set up X-Pack on Kibana:

To install X-Pack in Kibana:

/usr/share/kibana/bin/kibana-plugin install x-pack

Update Kibana to use the new password for the built-in kibana user, which you set up along with the other built-in users when you installed X-Pack on Elasticsearch. You must configure the elasticsearch.password setting in the kibana.yml configuration file with the new password for the kibana user.

elasticsearch.username: "kibana"
elasticsearch.password: "kibanapassword"

Start Kibana.

/usr/share/kibana/bin/kibana-plugin install x-pack

[8] Installing X-pack in logstash

The following diagram provides an overview of the steps that are required to set up X-Pack on Logstash:

To install X-Pack on Logstash:

/usr/share/logstash/bin/logstash-plugin install x-pack 

This guide prepared by Nuwan Vithanage