Configuring External Elasticsearch Cluster

Before you start, ensure that the external Elasticsearch is not started after API Gateway installation.

1. If you have started API Gateway before setting up the external Elasticsearch cluster configuration, perform the following steps before proceeding with the configuration:

2. Open elasticsearch.yml from <Elastic_Dir>/config/elasticsearch.yml in any node that you want to cluster.

cluster.name:"SAG_EventDataStore"
node.name: node1
path.logs: <Elastic_Dir>/logs
network.host:0.0.0.0
http.port:9240
discovery.seed_hosts: ["node1:9340"]
transport.port:9340
path.repo:['<Elastic_Dir>/archives']
cluster.initial_master_nodes:["node1"]
xpack.ml.enabled: false
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false
xpack.security.http.ssl.enabled: false
action.destructive_requires_name: false

discovery.seed_hosts. Provide a list of nodes to the external Elasticsearch that it should try to contact. Once the node contacts a member of the unicast list, it receives a full cluster state that lists all nodes in the cluster. It then proceeds to contact the master and join the cluster.

path.repo. This is a mandatory configuration for performing backup and restore. This is the location where the external Elasticsearch writes the snapshots to. Hence, it is important to have a location that is accessible to all the nodes. The location must be network file system, S3 or Azure in the clustered setup.

cluster.initial_master_nodes. This parameter must be set so that when you start a cluster for the first time cluster bootstrapping is performed. The parameter must contain the names of the master-eligible nodes in the initial cluster and must be defined on every master-eligible node in the cluster. This setting helps prevent split-brain, the existence of two masters in a single cluster.

action.destructive_requires_name. This parameter is set to be false so that the migration and backup processes can be performed.

External Elasticsearch provides an option to configure the locations where you would want to store your data and logs. Ensure that you specify the locations that are accessible and have enough disk space. It is also important to monitor and ensure basic house keeping of the data location by planning an effective data retention strategy. You can change the defaults using the following configuration:

path.data: /var/lib/elasticsearch
path.logs: <<Elastic_Dir>/logs
// These values can be changed

External Elasticsearch, by default, binds to loop back address and hence it is important to change it for a production deployment. For more details on this configuration, see https://www.elastic.co/guide/en/elasticsearch/reference/8.2/modules-network.html

Nodes with same cluster names form a cluster. That is, if there are three nodes in the cluster, the value in the cluster.name property must be same across all three nodes. In other words, Elasticsearch forms a cluster with nodes that have the same cluster.name.

cluster.name:"SAG_EventDataStore"

4. Provide the names of all participating nodes, as seen in the node.name property, and the ports they use, as seen in the http.port property, in the discovery.seed_hosts property in the following format:

If there are three nodes in the cluster, the value in the discovery.seed_hosts property is as in the following example:

discovery.seed_hosts: ["node1:9340","node2:9340","node3":"9340"]

The names of all nodes appear in the cluster.initial_master_nodes property. The node name displayed in this property is same as seen in the node.name property.

cluster.name:"SAG_EventDataStore"
node.name: node1
path.logs: <Elastic_Dir>/logs
network.host:0.0.0.0
http.port:9240
discovery.seed_hosts: ["hostname1:9340","hostname2:9340","hostname3:9340"]
transport.port:9340
path.repo:['<Elastic_Dir>/archives']
cluster.initial_master_nodes:["node1","node2","node3"]
xpack.ml.enabled: false
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false
xpack.security.http.ssl.enabled: false
action.destructive_requires_name: false