2019-06-05 03:59:16 -05:00
|
|
|
# About
|
|
|
|
|
|
|
|
![essearcher](essearcher.png)
|
|
|
|
|
2021-10-20 20:33:11 -05:00
|
|
|
It provides a dynamic system for searching logs stored in
|
|
|
|
Elasticsearch. Currently it has out of the box support for the items below.
|
2019-06-05 03:59:16 -05:00
|
|
|
|
|
|
|
* [syslog](https://metacpan.org/pod/Search::ESsearcher::Templates::syslog)
|
|
|
|
* [postfix](https://metacpan.org/pod/Search::ESsearcher::Templates::postfix)
|
|
|
|
* [fail2ban via filebeat](https://metacpan.org/pod/Search::ESsearcher::Templates::bf2b)
|
|
|
|
* [HTTP access via filebeat](https://metacpan.org/pod/Search::ESsearcher::Templates::httpAccess)
|
|
|
|
|
2021-10-20 20:33:11 -05:00
|
|
|
# Configuring
|
|
|
|
|
|
|
|
If elasticsearch is not running on the same machine, then you will
|
|
|
|
need to setup the elastic file. By default this is
|
|
|
|
~/.config/essearcher/elastic/default . If not configured, the default
|
|
|
|
is as below.
|
|
|
|
|
|
|
|
```
|
|
|
|
{ "nodes": [ "127.0.0.1:9200" ] }
|
|
|
|
```
|
|
|
|
|
|
|
|
So if you want to set it to use ES on the server foo.bar, it would be
|
|
|
|
as below.
|
|
|
|
|
|
|
|
```
|
|
|
|
{ "nodes": [ "foo.bar:9200" ] }
|
|
|
|
```
|
|
|
|
|
|
|
|
The elastic file is JSON that will be passed to hash and passed to
|
|
|
|
[Search::Elasticsearch](https://metacpan.org/pod/Search::Elasticsearch)->new.
|
|
|
|
|
2021-10-20 21:40:05 -05:00
|
|
|
# As A Nagios Style Check
|
|
|
|
|
|
|
|
This requires three options, -n, -w, -c.
|
|
|
|
|
|
|
|
```
|
|
|
|
-n <check>
|
|
|
|
-w <warn>
|
|
|
|
-c <critical>
|
|
|
|
|
|
|
|
Check is the equality to use when comparing the number of hits found
|
|
|
|
for the search.
|
|
|
|
|
|
|
|
gt >
|
|
|
|
gte >=
|
|
|
|
lt <
|
|
|
|
lte <=
|
|
|
|
|
|
|
|
Critical and warn are the thresholds to use.
|
|
|
|
```
|
|
|
|
|
|
|
|
So for example for httpAccess if we want to alert for number of times
|
|
|
|
robots.txt is requested, we would do it like below.
|
|
|
|
|
|
|
|
```
|
|
|
|
essearcher -m httpAccess --dgte -5m --req robots.txt -n gt -w 2 -c 5
|
|
|
|
```
|
|
|
|
|
2021-10-20 21:41:44 -05:00
|
|
|
This will search for requests with 'robots.txt' in it within the last
|
|
|
|
5 minutes and will warn if the number of hits are great than 2 and go
|
|
|
|
critical if greater than 5.
|
|
|
|
|
2021-10-20 20:33:11 -05:00
|
|
|
# Extending
|
|
|
|
|
2019-06-05 03:59:16 -05:00
|
|
|
It has 5 parts that are listed below.
|
|
|
|
|
2021-10-20 20:33:11 -05:00
|
|
|
* options : [Getopt::Long](https://perldoc.perl.org/Getopt/Long.html)
|
|
|
|
options that are parsed after the initial basic options. These are
|
|
|
|
stored and used with the search and output template.
|
|
|
|
* elastic : This is a JSON that contains the options that will be used
|
|
|
|
to initialize [Search::Elasticsearch](https://metacpan.org/pod/Search::Elasticsearch).
|
|
|
|
* search : This is a [Template](https://metacpan.org/pod/Template)
|
|
|
|
template that will be fed to [Search::Elasticsearch](https://metacpan.org/pod/Search::Elasticsearch)->search.
|
|
|
|
* output : This is a [Template](https://metacpan.org/pod/Template)
|
|
|
|
template that will be be used on each found item.
|
2019-06-05 03:59:16 -05:00
|
|
|
|
|
|
|
It will search for those specified in the following order.
|
|
|
|
|
|
|
|
1. $ENV{'HOME'}.'/.config/essearcher/'.$part.'/'.$name
|
2021-10-20 20:33:11 -05:00
|
|
|
1. $base.'/etc/essearcher/'.$part.'/'.$name
|
2019-06-05 03:59:16 -05:00
|
|
|
1. Search::ESsearcher::Templates::$name->$part (except for elastic)
|
|
|
|
|
2019-06-02 04:42:01 -05:00
|
|
|
# INSTALLING
|
2019-05-24 22:52:40 -05:00
|
|
|
|
2021-10-20 20:33:11 -05:00
|
|
|
## FreeBSD
|
2019-06-02 04:42:01 -05:00
|
|
|
|
2021-10-21 11:55:35 -05:00
|
|
|
```
|
2021-10-20 20:33:11 -05:00
|
|
|
pkg install perl5 p5-JSON p5-Error-Helper p5-Template p5-Template-Plugin-JSON p5-Time-ParseDate p5-Term-ANSIColor p5-Data-Dumper
|
|
|
|
cpanm Search::ESsearcher
|
|
|
|
```
|
2019-08-05 23:39:43 -05:00
|
|
|
|
2019-06-02 04:42:01 -05:00
|
|
|
## Linux
|
|
|
|
|
|
|
|
### CentOS
|
|
|
|
|
2021-10-20 20:33:11 -05:00
|
|
|
```
|
|
|
|
yum install cpanm
|
|
|
|
cpanm Search::ESsearcher
|
|
|
|
```
|
2019-06-02 04:42:01 -05:00
|
|
|
|
|
|
|
### Debian
|
|
|
|
|
2021-10-20 20:33:11 -05:00
|
|
|
```
|
|
|
|
apt install perl perl-base perl-modules make cpanminus
|
|
|
|
cpanm Search::ESsearcher
|
|
|
|
```
|
2021-10-21 09:11:10 -05:00
|
|
|
|
|
|
|
# Caveat
|
|
|
|
|
|
|
|
Please be aware that if a similar search has not been ran for awhile,
|
|
|
|
Elasticsearch will likely return a buggy/empty result that can't be
|
|
|
|
used. The usual return when this happens is empty JSON just containing
|
|
|
|
the key 'hits', which can be viewed via switch -R. When this happens,
|
|
|
|
just wait a few minutes or so to try again and Elasticsearch should
|
|
|
|
have reindex/cached/etc.
|
|
|
|
|