AWS Elasticsearch frequently getting into yellow state

Question

• I have AWS Elasticsearch running with 3 Master nodes(C4.large) and 10 data nodes(C5.large). Recently I am experiencing frequently that my domain gets into yellow state for some time around 30 mins and then i do nothing it changes to green.
• when i used the query GET /_cluster/allocation/explain?pretty this is what i see

{
  "index" : "lgst-",
  "shard" : 4,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2021-01-06T13:15:38.721Z",
    "details" : "node_left [**************]",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "yes",
  "allocate_explanation" : "can allocate the shard",
  "target_node" : {
    "id" : "****************",
    "name" : "********************"
  },

• I couldn't understand what does it mean and how do i over come it. Any help please would be appreciated.

Answer 1

Looks like you are using spot instances in your cluster and cause for this is that nodes in your AWS is not stable as shown clearly in the unassigned_info

"unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2021-01-06T13:15:38.721Z",
    "details" : "node_left [**************]",
    "last_allocation_status" : "no_attempt"
  },

I would suggest chaning the instance types if you are using ec2 spot instances and check why nodes are getting disconnected in your cluster with AWS support.