Makes searching logs stored in elasticsearch from the command line easy.
Go to file
Zane C. B-H ed247fdbbd add repo to Makefile.PL 2021-11-08 20:32:55 -06:00
bin learned about make dist 2021-10-28 06:32:44 -05:00
lib/Search fix srcx and hostx for syslog and bump for release 2021-11-08 20:23:42 -06:00
logstash ready to release 0.4.3 2021-11-04 04:01:24 -05:00
t learned about make dist 2021-10-28 06:32:44 -05:00
.gitignore meh 2019-12-06 12:15:13 -06:00
Changes fix srcx and hostx for syslog and bump for release 2021-11-08 20:23:42 -06:00
LICENSE Initial commit 2019-05-24 22:52:40 -05:00
MANIFEST add back in postfix geoip processing 2021-11-04 03:57:32 -05:00
Makefile.PL add repo to Makefile.PL 2021-11-08 20:32:55 -06:00
README learned about make dist 2021-10-28 06:32:44 -05:00
README.md remove a extra ` 2021-10-21 11:55:35 -05:00
essearcher.png Add files via upload 2019-06-05 01:51:36 -05:00

README.md

About

essearcher

It provides a dynamic system for searching logs stored in Elasticsearch. Currently it has out of the box support for the items below.

Configuring

If elasticsearch is not running on the same machine, then you will need to setup the elastic file. By default this is ~/.config/essearcher/elastic/default . If not configured, the default is as below.

{ "nodes": [ "127.0.0.1:9200" ] }

So if you want to set it to use ES on the server foo.bar, it would be as below.

{ "nodes": [ "foo.bar:9200" ] }

The elastic file is JSON that will be passed to hash and passed to Search::Elasticsearch->new.

As A Nagios Style Check

This requires three options, -n, -w, -c.

-n <check>
-w <warn>
-c <critical>

Check is the equality to use when comparing the number of hits found
for the search.

gt >
gte >=
lt <
lte <=

Critical and warn are the thresholds to use.

So for example for httpAccess if we want to alert for number of times robots.txt is requested, we would do it like below.

essearcher -m httpAccess --dgte -5m --req robots.txt -n gt -w 2 -c 5

This will search for requests with 'robots.txt' in it within the last 5 minutes and will warn if the number of hits are great than 2 and go critical if greater than 5.

Extending

It has 5 parts that are listed below.

  • options : Getopt::Long options that are parsed after the initial basic options. These are stored and used with the search and output template.
  • elastic : This is a JSON that contains the options that will be used to initialize Search::Elasticsearch.
  • search : This is a Template template that will be fed to Search::Elasticsearch->search.
  • output : This is a Template template that will be be used on each found item.

It will search for those specified in the following order.

  1. $ENV{'HOME'}.'/.config/essearcher/'.$part.'/'.$name
  2. $base.'/etc/essearcher/'.$part.'/'.$name
  3. Search::ESsearcher::Templates::$name->$part (except for elastic)

INSTALLING

FreeBSD

pkg install perl5 p5-JSON p5-Error-Helper p5-Template p5-Template-Plugin-JSON p5-Time-ParseDate p5-Term-ANSIColor p5-Data-Dumper
cpanm Search::ESsearcher

Linux

CentOS

yum install cpanm
cpanm Search::ESsearcher

Debian

apt install perl perl-base perl-modules make cpanminus
cpanm Search::ESsearcher

Caveat

Please be aware that if a similar search has not been ran for awhile, Elasticsearch will likely return a buggy/empty result that can't be used. The usual return when this happens is empty JSON just containing the key 'hits', which can be viewed via switch -R. When this happens, just wait a few minutes or so to try again and Elasticsearch should have reindex/cached/etc.