
=========================================
Indexed Logging Output with ElasticSearch
=========================================

Intro
-----

Bro's default ASCII log format is not exactly the most efficient way
for searching large volumes of data. ElasticSearch is a new data
storage technology for dealing with tons of data. It's also a search
engine built on top of Apache's Lucene project. It scales very well,
both for distributed indexing and distributed searching.

.. contents::

Warning
-------

This writer plugin is still in testing and is not yet recommended for
production use!  The approach to how logs are handled in the plugin is "fire
and forget" at this time, there is no error handling if the server fails to
respond successfully to the insertion request.

Installing ElasticSearch
------------------------

Download the latest version from: http://www.elasticsearch.org/download/.
Once extracted, start ElasticSearch with::

# ./bin/elasticsearch

For more detailed information, refer to the ElasticSearch installation
documentation: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html

Installing the ElasticSearch Plugin
-----------------------------------

First, ensure that you have libcurl (headers and library) installed. Then the
following will compile and install the plugin alongside Bro::

    # ./configure && make && make install

See the output of ``./configure --help`` for additional options if it
can't find any of the prerequisites.

If everything built and installed correctly, you should see this::

    # bro -N Bro::ElasticSearch
    Bro::ElasticSearch - ElasticSearch log writer (dynamic, version 1.0)

Activating ElasticSearch
------------------------

The easiest way to enable ElasticSearch output is to load the
plugin's ``logs-to-elasticsearch.bro`` script.  If you are using BroControl,
the following line in local.bro will enable it:

.. console::

    @load Bro/ElasticSearch/logs-to-elasticsearch.bro

With that, Bro will now write most of its logs into ElasticSearch in addition
to maintaining the Ascii logs like it would do by default.  That script has
some tunable options for choosing which logs to send to ElasticSearch, refer
to the autogenerated script documentation for those options.

There is an interface being written specifically to integrate with the data
that Bro outputs into ElasticSearch named Brownian.  It can be found here::

    https://github.com/grigorescu/Brownian

Tuning
------

A common problem encountered with ElasticSearch is too many files being held
open.  The ElasticSearch documentation has some suggestions on this and other
issues.

  - http://www.elastic.co/guide/en/elasticsearch/reference/1.3/setup-configuration.html

TODO
----

Lots.

- Perform multicast discovery for server.
- Better error detection.
- Better defaults (don't index loaded-plugins, for instance).

