elasticsearch - What is the best way to implement Email Alerts in Elastisearch?

Question

We will be building a new job board type site that runs in AWS and we are using Elastisearch for all the job and candidate search functionality.

The site will have email alerts. 1) Candidates can set an alert so that a new job that is posted that matches certain keywords and is within X miles of a certain zipcode will be emailed to them. 2) Recruiters will be able to set alerts so that a resume with certain keywords within X miles of a certain zipcode will be emailed to them

Is there opensource code that will get us started?

I have read a bit on Watcher by Elastic.co but we are bootstrapping and trying to find a low cost solution. I will ping Elastic to get a feel for their up front costs...

Has anyone built a scalable web application that can handle 10's of thousands of alerts per day using Watcher or some other Alerting Tool?

Thx,

Brent Byers

Answer 1

If you don't want to use Watcher (yet), the next available option is to dig into the percolate API (+ older blog post on the subject and a more recent one) in order to implement the "watch" part and handle the "alert" part in your own code.

Some examples of what people out there are doing:

1. You might want to check ElastAlert created by Yelp and which should probably get you going.

2. Toplog uses the percolate API in order to get alerts when specific log patterns are being index. They've also created the percolator Logstash output in order to percolate a document just after it's been indexed and store the matching percolated queries into redis (or what have you). Some Logstash folks have had the same idea and might be considreing that plugin as a viable option.

3. There's another example where an email is sent whenever a new product is being indexed which matches a given query.

If you dig a little more, you'll certainly find plenty more examples. But the overall idea is always the same:

1. Store job/candidate queries using the percolate API
2. Percolate new job/candidates documents
3. If matching queries are found in step 2, send alerts.

Finally, there's an open issue in Kibana for integrating an alerting system that you might also want to watch.

UPDATE (2016/05/04)

I've decided to create a Logstash filter that supports the ES Percolate API. It is similar to what the Toplog guys do, but it runs as a filter to enrich the event instead of running as an output to redis.

You can find more info at: https://github.com/elastic/logstash/issues/2187#issuecomment-216760668