[Snort-devel] Plugin API Feature Request

Thomas.Seiler at ...2736... Thomas.Seiler at ...2736...
Mon Feb 13 07:57:04 EST 2006


Hi Jeff,
Hi list, 

> I don't completely understand your issue. I still suspect that a
> callback within the main pcap loop would be sub-optimal.

Yes, I was not very clear in my last mail, sorry for that.
I will try to be clearer in this mail even if this means that the mail
will get long.

I agree that it is not very optimal to break the pcap loop for
performance reasons, but I am not sure how one could address the issues
in another way. That is why I wanted to start a discussion. I think the
best way currently is to have two version of the main loop:
- one with pcap_dispatch if there is a plug-in loaded who benefits from
it 
- one with pcap_loop for those who only need the speed.

Maybe I should first tell you something about the context. We are trying
to build a new and highly experimental output plug-in for snort. The
idea is to fork a child process or second thread depending on platform
which handles the output-plugin so that the snort decoder is not blocked
by the output. The goal is to reduce sensor deployment complexity.

While working on this plug-in, I found these issues with the current
snort plug-in API:

(issue 1 : callbacks in signal handler context)

CleanExit() callback is currently called from a signal handler context.
On many platforms this means that it is not safe to call any libc
function that is not known to be re-entrant (thread-safe). This can lead
worst case to a segfault killing snort. The problem is, that CleanExit()
is also called when snort receives a SIGHUP - to reload its
configuration.

We wanted to log the reload / shutdown of snort in the database as a
kind of meta-alarm, but I cannot execute an SQL statement in my
CleanExit() callback, because it is called asynchronously to the packet
logging. It is therefore quite probable that the database connection is
midway in the execution of another statement from the packet processing
callbacks, and logging the reload/shutdown event will result in
desynchronizing the db connection instead of logging anything.

Also, if I start threads or child processes from my output plug-in in
order to decouple logging from the detection engine of snort (i.e. see
prelude plug-in), then I cannot cleanly shutdown these threads / childs
because any call to libc is dangerous, and could kill snort when
reloading the config.

Currently the prelude plug-in is suffering from this as well. If you
have prelude logging enabled and signal SIGHUP to snort, you will end up
with zombie processes.

This is fixable by a pcap_dispatch instead of pcap_loop in the main
loop. The signal handler would just set a global variable which is
queried to exit the main loop. After this the CleanExit() handlers can
be called.

This might however have an impact on performance.

Steven Sturges [steve.sturges at ...402...] wrote on 09.02.06 14:52
CET
> We are already working on rearranging the main loop and using pcap 
> dispatch.  We are also addressing the non-reentrant nature of the
2.4.x 
> signal handlers along with this set of changes.

I was very glad to hear that. So hopefully, the next snort release will
already have solved the CleanExit() issues.

(issue 2 : no regular callback for heartbeat functionality available) 

It was also one of the goals to have a kind of heartbeat from the
sensors. Every N seconds, it should send a heartbeat to the database,
saying that it is up and running, and also giving some status about the
system (i.e. total number of pakets, number of dropped packets and CPU
usage) It is important to get this heartbeat from the detection thread /
process, because then we can supervise the whole logging path, from the
detector, via logging thread to the database.

I can currently do this in the packet processing callbacks and check
each time if I should send the next heartbeat. But what if no packet
arrives for more than N seconds? At the moment I can't tell a dead
sensor from a sensor without traffic, which somehow makes the whole
heartbeat idea somehow useless.

It would also be interesting to use heartbeat callback to feed commands
to a sensor via the database, i.e. the heartbeat callback could poll for
a pending command. (i.e reload,stop,...)

Jeff Nathan [jeff at ...835...] wrote on 13.02.06 13:03
> If you traverse a list of plugin callbacks and one of them blocks,  
> then all of snort is blocked.  Perhaps this is a circular argument,  
> it's not intended to be so, however...

Well the same is true for the alert logging callbacks, nevertheless they
are very useful. The overhead for this should be in the order of
magnitude for one additional alert logged every N seconds, otherwise the
idea is not practical. I think nobody will have to low-level-format a
hard drive in this callback ;)

The problem is that currently pcap_loop is blocking when there is no
packet to sniff. To execute a callback every N seconds, one would have
to exit the pcap_loop from time to time, check if its time to heartbeat
and then call the callbacks. It is important that such a heartbeat
callback happens in sync with the packet processing, because only then
it can use whatever resources (db handles, sockets, etc) the normal
alert logging uses.

Jeff Nathan [jeff at ...835...] wrote on 13.02.06 13:03
> Perhaps your database provides you a handle that is selectable or
> similar?  In that case you could register an event for the database
> handle and its callback could call your intended callback, provided
> the database was in a suitable state.

I am not sure I understand your solution correctly. I think you are
calling libevent's event_loop() whenever there is an alert to be logged.
But when there is no alert for a long period of time, and a heartbeat
should be sent, how (from where ?) is the callback then called ? This is
the part that I don't understand... I think it has to be called form an
alarm handler, and then I have the same issues as in (1).
 
If snort will change in the near future to a pcap_dispatch main loop
exit the packet handling from time to time (i.e. once a second) to
handle signals, there is another way to add heartbeats to snort.

time_t last_heartbeat;
time_t between_heartbeats;

while(running) {
   // ... snort main loop ...
   
   if (last_hearbeat + between_hearbeats < time()) {
     call_plugins()
   }

   // ... snort main loop ...

   pcap_dispatch(...);

   // ... snort main loop ...

}

(issue 3 : callbacks at end of initialisation )

This is a just a minor issue. We wanted to be able to save the whole
ruleset of snort at startup to the database, instead of checking for
each alert if the corresponding signature was already saved in the
database as the current spo_database.c does it. But the problem is that
at initialization time, the plugin might not see the whole
configuration. The order at which plugins are initialized / rules are
parsed depends on the order in which they are in the config file. If the
plugin gets loaded before any rules, it will see nothing.

So we had to wait for the first packet before we could synchronize the
signatures with the database. The question was, if there is a way to be
called after all rules are parsed, so that the plug-in can have a look
at the rules.

I hope I was able to clarify my issues. Feel free to ask if there are
any questions left open. My intention was to start a discussion about
the mentioned issues and possible solutions...

Best Regards,
Thomas Seiler


-----------------------------
Thomas Seiler
Ing. sys. com. dipl. EPFL
SWISSCOM AG
Innovations
Security and Service Management
Ostermundigenstrasse 93
CH - 3050 Bern
SWITZERLAND


Phone:  +41 (0)31 342 42 69
Mobile: +41 (0)79 427 97 26
Fax:    +41 (0)31 892 62 27


thomas.seiler at ...2736...
http://www.swisscom.com





More information about the Snort-devel mailing list