[Snort-devel] To discuss: FLoP and missing database
Dirk at ...972...
Tue Dec 9 14:52:02 EST 2003
I received some concerns about the problem what would
happen if the database is not available.
Maybe someone has some good ideas how this should
be handled. (Note: As far as I can oversee it, neither
the database output plugin of snot nor barnyard did
really count on this problem. The database is assumed
to run every time and forever. BTW: FLoP does the same
up to now...)
The actual problem is:
There is no check for the availability of the database.
So: What happens if the database is not available?
Up to FLoP-1.0:
It is assumed that the database is available. If it fails,
all errors are ignored. An error message is printed but
So if noone implements checks for this messages all
works and nothing will be logged. Not a good solution.
FLoP-1.0.1: (ready, but not released yet -> needs testing)
There is a check for the availability of the database
if a remote sensor (sockserv) connects.
A handshake is implemented, if the database is not
available an error code is sent to the sockserv
On an error: Depending on the return code the sockserv
process will retry a connection later on (recoverable
error) or will exit (unrecoverable database error).
(Further we check the endianess. These must now be
identical on server and remote sensor. This is the
first step towards a mixed environment but is not
related to the database...)
Version FLoP-1.xxx: (not started yet)
Real Problem: What happens if the database is dying
during a session (after the initialization)?
Up to now (Version 1.0.1) we check only for the database
on a connect of a remote sensor. If the database fails
after this time we are at the old problem...
+ We check for the database availabilty or failure
of any INSERT statement (this is ignored yet).
+ If a INSERT fails we close the connection to the
remote sensor (aka sockserv). So no further alerts
will be accepted in this session.
+ The sockserv process will try to reconnect. This
may fail for two reasons:
1. There are still some old alerts which have to be
INSERTed first (yes this is checked!)
2. The database is not available.
+ Depending on the problem (see above) sockserv will
try several times to reconnect.
So far all seems to be easy. Now we have to think
about the alerts already spooled to servsock which
may be still in memory but not INSERTed in the
database. Here we could implement:
+ Open a file: database.sensorname (maybe should be done
+ Write all INSERT statements of the failing alert to
+ Write all INSERT statements of the remaining alerts
to this file.
+ Close this file and do an exit.
+ Any further connects of sockserv processes are handeld
as in version FLoP-1.0.1 (see above)
The advantage of this method:
+ We get an ASCII file of statements which can be sourced
manually into the database.
and the disadvantage:
+ The cid is counted for each alert and is part of the
INSERT statement (the cid is part of the combined
primary index). If the database is available again,
the sockserv process re-connects and inserts alerts
these cid values will overlap.
+ So: We have to source this file *before* the correspondent
sockserv is starting to insert new alerts.
+ Big Question: how and when?
+ Ok, we can check the file manually and then remove
it if no important alert is found.
Another solution would be in writing the data binary
to a file. This data can be read in if the client
connects and the database is available. Thus we
first read in the data from the file, remove this
file and then start the normal, threaded, process.
+ we don't have to care about the cid
+ the data are inserted in right order
+ it's more difficult to implement.
Does anyone have a better idea, some comments
or simply a good argument what should be implemented?
More information about the Snort-devel