All Articles

Elasticsearch EC2 Discovery on AWS

The gist

The Hive and Cortex both uses Elasticsearch for its data store. I wanted to make sure that I could deploy a resilient cluster. However, there was not much information on how to do this in an AWS environment from the documentation. After researching and a bit of Google-fu-ing, I’ve got it to work. I’m assuming if you’re reading this, you’re trying to do the same. Otherwise, you might be lost :O

I plan to write another blog about my set up for The Hive/Cortex. Stay tuned, once it is out, I will link it through this blog as well! But for now, their Github have a great documentation section that does a really good job of what it does, and how to deploy it!

Set up ES 6.8

Note: This is the version I am using with The Hive. If you are doing this for a different project, then ignore this.


  • Set up an IAM role with EC2 discovery policy attached
  • Make note of es-access-user KeyID and AWS Secret
  • Attach policy to instance
  "Statement": [
      "Action": [
      "Effect": "Allow",
      "Resource": [
  "Version": "2012-10-17"
sudo apt-get update
sudo apt-get upgrade

sudo apt-get install openjdk-11-jre-headless

wget -qO - | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install elasticsearch

Just for debugging purposes, I also like to install these 2 packages:

sudo apt-get install curl
sudo apt-get install jq

Set $JAVA_HOME in /etc/default/elasticsearch:

sudo vi /etc/default/elasticsearch

Now set the JVM memory options to at least ½ of the memory on the machine. ES says anything less than that will result in poor performance. (On 8 gb machine, 4 is half.)

sudo vim /etc/elasticsearch/jvm.options

Set up EC2-discovery for AWS:

cd /usr/share/elasticsearch/bin
sudo ./elasticsearch-plugin install discovery-ec2

Set up Elasticsearch Keystore (for both Master + Node):

cd /usr/share/elasticsearch/bin
sudo ./elasticsearch-keystore create
sudo ./elasticsearch-keystore list
sudo ./elasticsearch-keystore add discovery.ec2.access_key (enter key when prompts)
sudo ./elasticsearch-keystore add discovery.ec2.secret_key (enter key when prompts)
sudo ./elasticsearch-keystore list

Once you’re done, it should look like this:

ubuntu@ip-x-x-x-x:/usr/share/elasticsearch/bin$ sudo ./elasticsearch-keystore list

If you have ES running already:

Disable shard allocation

curl -H "Content-Type: application/json" -XPUT 'localhost:9200/_cluster/settings' -d '{ "persistent": { "cluster.routing.allocation.enable": "none" } }'

Stop ES & Kibana:

sudo systemctl stop elasticsearch
sudo systemctl stop kibana

Edit Elasticsearch settings:

sudo vi /etc/elasticsearch/elasticsearch.yml

Master Node Setting: hive hive

node.master: true true
node.ingest: true

# /var/lib/elasticsearch
# path.logs: /var/log/elasticsearch [_ec2_,_local_]

discovery.zen.hosts_provider: ec2 ["x.x.x.x", "x.x.x.x"]
discovery.zen.minimum_master_nodes: 1

discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip

cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.tag.es_cluster: "hive-prod-elasticsearch"

thread_pool.index.queue_size: 100000 100000
thread_pool.bulk.queue_size: 100000

Data/Ingest/Master-eligible Node Setting: Use the same config above. Edit these as you see fit for your environment

node.master: true true
node.ingest: true

Start ES and Kibana back up:

sudo systemctl start elasticsearch
sudo systemctl start kibana

Check system health:

curl -XGET http://localhost:9200/_cluster/health?pretty=true

It should said there are 2 nodes running.

If anything is wrong, check the log:

sudo cat /var/log/elasticsearch/hive.log

Re-enable shard allocation:

curl -H "Content-Type: application/json" -XPUT 'localhost:9200/_cluster/settings' -d '{ "persistent": { "cluster.routing.allocation.enable": null } }'


  • Data nodes - stores data and executes data-related operations such as search and aggregation.
  • Master nodes - in charge of cluster-wide management and configuration actions such as adding and removing nodes.
  • Client nodes - forwards cluster requests to the master node and data-related requests to data nodes.
  • Ingest nodes - for pre-processing documents before indexing.
  • - Any name. Must be the same across all nodes.
  • - Any name.
  • node.master - Whether or not this is a master node, meaning one that coordinates the activity of data nodes. In our example, we have 1 master and 2 data nodes. The master can also serve as a data node.
  • - Indicates whether this node can store data.
  • node.ingest - Used to pre-process documents before they are indexed.
  • discovery.zen.hosts_provider - Use ec2 for node discovery.
  • - List the IP address of all master and data nodes in the cluster. Enclose in commas.
  • - The private IP address of this machine, i.e., the one shown by the ifconfig -a command. (Not the public address that you use to ssh into)

More info about IAM role set up: