[Snort-devel] Re: Announce: FLoP-1.0 --- Fast Logging Project for snort
bamm at ...101...
Mon Dec 1 21:45:10 EST 2003
On Mon, Dec 01, 2003 at 11:05:07PM +0100, Dirk Geschke wrote:
> Hi Bamm,
> > I hope you don't think I was attacking flop. I actually want
> > to take a close look at what you've done and also talk to you
> > about supporting sguil (http://sguil.sf.net) and its DB schema.
> I did not assume that you wanted to attack the project but there
> were some points which needed some comments.
> > My only intent was to start a discussion of the main disadvantage
> > of using the unix socket plugin with snort: that if for any reason,
> > the process reading the unix socket dies, any alerts that snort
> > writes while the proc is down, is gone for ever. In short, by using
> > the unix socket, you've added another potential point of failure to
> > your IDS. Please understand, that I believe that risk is fairly low
> > (at least for a well written/tested program), but there are those
> > that may not be willing to accept that risk, just like there are those
> > that won't entertain the idea of an inline IDS (versus passive monitoring)
> > for the sole reason a inline IDS brings another point of failure into the
> > network.
> With any solution you introduce some possible problems. Think either of
> a hanging snort waiting for the output plugins and therefore simply
> missing some packets to sniff. Or think of a filled up filesystem where
> no process is able to write any byte which could have just more and
> worser side effects. With unix domain sockets you can add an extra
> layer of security. I just mentioned it: The output-plugin realizes
> if a process is reading on the socket or not. So if writing to the
> socket fails you can start another output plugin or simply write
> the data in a file.
> The problem here is: You need a mechanism how you want to handle this
> special situation. And much worser: How to watch for the process which
> waits for data in this special file...
> > Now that that is said, can you answer a couple of questions (why RTFM when
> > I have your ear ;) ).
> > From the documentation I did read, it looks like if servsock (the process
> > that recieves data from the sockserv) dies or becomes unreachable for an
> > extended period of time, then sockserv (the proc that reads the unix sock)
> > will die. I saw the experimental 'drop' feature, but I am wondering if you
> > had any plans to instead start writing the events to disk and then be able
> > to read those alerts back into sockserv and send them off once servsock
> > becomes available again?
> Okay, this can lead to problems in two sitution. Either when sockserv is
> started or if an alert arises. If there is no servsock process listening
> the program tries MaxTry times to connect to servsock. Between two tries
> the process waits WaitTime seconds. This values are set by default to
> 10, which is quite small but can be adjusted with the command line
> options -M and -W.
> The reason why this values are so small by default is to make it easier
> to test it. And the main reason why I don't use an infinite loop is to
> have the possibility to inform someone of the problem.(I simply assume
> that there is a job in the background surveying the processes. How else
> will you notice that there is a problem? But in principle you can choose
> high values for this parameters.)
> The drop feature was introduced as a "stability" factor. Think of a lot
> of buffered alerts and no servsock. Then the memory usage will increase
> until malloc() fails. The result would be a dying sockserv and all
> alerts are lost. With drop you can get a short information of wich
> alerts are logged. This contains of the signature, the IP addresses
> and ports (if ther are any) involved in the process together with the
> timestamp. This values are either send to a list of recipients and if
> this is not possible the alerts are written to syslog or stdout. (Think
> of a network problem: No E-Mail can be sent so we write to stdout or
> syslog.) So effectively you only loose some network headers and the
> And finally: Yes, first I thought on a temporarily buffering of alerts
> by writing them to disk. But then you have the same problems we already
> discussed with barnyard. Additionally there is a further problem: When
> and how will you reload these alerts? Will you stop reading from the
> unix socket until all data is read in from disk? Or should we create
> a third thread, mixing up with the other by storing the alerts again
> in memory? Think of a disturbed network where you can't reach the
> cental server. All alerts will fill up memory, where dumped to disk
> and reread until you have to redump them again? And what is with the
> alerts ariving from snort inbetween?
> So there are a lot of problems with this approach.
> If you assume that sockserv will not die due to an unknown behaviour
> like power shortage or something else. Then let us start a small
> calculation: The alert information together with the payload of a
> full saturated (MTU) packet is less than 3 kB of memory (to be more
> precisely 1360 Byte + 1514 Bytes). Now memory is quite cheap so I
> assume you may have 300 MB (it's easier to calculate) of memory
> available solely for sockserv. Then you can buffer up to 100,000
> alerts until you are running into problems. And this is for the case
> that all alerts use the maximum MTU size of Ethernet...
I've learned to expect the unexpected. For me, many of my sensors are remote, and I don't control any of the networks. So, if we take our 'central' network down for maintenance, my remote sensors wouldn't be able to connect to the central servsock until it's back up. Usually this is no more than two to four hours, but every once in a while it can go down for longer. Yes, we write alerts to disk after they use X amount of mem (set in a conf file). They are written one alert per file and then read back in order once the backend becomes available again. There are numerous ways to accomplish this and I like the extra reliability it provides. I don't think it's right to compare it to barnyard since we are only writing to disk when we can't send the alerts up. I'd rather take a small hit in my snort performance rather than losing alerts altogether.
> A problem could arise if you have to restart or replace sockserv.
> But I guess you won't do this on a heavy attack...
> > Also, any plans to do much of the same thing with servsock (like
> > if the DB somehow become unavailable for an extended length of time)?
> > I can think of a couple of circimstances that may cause either the DB
> > and/or servsock to become 'unavailable' (such as a server reboot, DB
> > cleaning, etc). This also adds more points of failure, where if one
> > component dies, events could be lost to /dev/null.
> Hmm, this is a little bit more complicated. I don't know how I can
> identify problems with the database connection within servsock. But
> the good thing is: Good databases will never die ;-)
We can dream can't we ;)
> In fact databases will rarely die and my project assumes that the
> database is running on the same host as servsock. Otherwise we can't
> feed the database via an unix socket... So a reboot is no issue. DB
> cleaning could be: but this can be done online. Then maybe servsock
> will work a little bit slower...
Depends on the database. We use postgres and while a 'normal' vacuum is good for routine maintenance, once you postgres DB gets huge (tens of millions of rows) you probably need to run 'VACUUM FULL' which will totally lock the DB until its done (were talking hours here). I'd be interested to see how servsock respondes to a locked DB. Generally, I restart the DB and only allow local connections when I need to do a 'VACUUM FULL', this causes the remote agents to start buffering their events locally (our version of servsock isn't run on the DB machine).
> > Anyway, the project definately looks cool and I am glad see there
> > is a real alternative to unified out and barnyard out there.
> Fine, give it a try and give me a note how it worked...
> Finally one note to sguil: I just took a look at the project site
> and TCL/TK is not really my world. But if I understand it the right
> way than it seems to be a more complex system than FLoP...
> Which database design are you using? I think it is the same one as
> of snort/ACID? Did you ever think about the possibility to store the
> whole pcap data in the database? This should make it unnecessary to
> store them on disk on a separate way.
No, sguil does not use the standard snort/acid schema. There have been many discussions about this, but basically the standard schema isn't as scalable as I and others need/want it to be. I can send you a diagram (once I create it) otherwise you can just take a peek at the create_sguildb.sql script in the source. Pcap data for each alert is stored in the DB. The pcap that is stored on the sensors is for entire streams (think 'log ip any any -> any any'). It's hard enought to store that data locally, I can't imagine trying to push that data up to a DB (let alone dealing with it once it's there). Sguil uses barnyard and the op_sguil plugin for recieving RT events and INSERTing them into the DB right now. Are you interested in discussing (off this list) an option for FLoP to use the sguil schema and to send alerts directly to sguild?
> Best regards
More information about the Snort-devel