[Snort-devel] Re: A Proposal for Reengineering Snort's Packet Matching System - Com ments

Todd Lewis tlewis at ...255...
Fri Apr 6 01:40:01 EDT 2001


Abe said that I could respond to his message in public, and so I do so.

On Fri, 6 Apr 2001 agetchel at ...358... wrote:

> Hi Todd,
> 	Great document!  One thing this got me thinking about is the idea of
> a self-tuning rule matching system.  This would provide a quick and dirty
> automatic mechanism for Snort itself to automatically optimize the order of
> the rules which each packet is compared too.
> 	The way that the Snort rules are organized now is primarily by
> layer-7 protocol type (HTTP, FTP, SMTP, SQL, etc).  The rules contained in
> these files could be ranked by Snort, during run-time, based on how much
> traffic that Snort sees of these various protocol types.  For instance, if
> Snort sees that 70% of all traffic it is processing is HTTP, then Snort
> would rank all of the rules dealing with HTTP higher, and matcher would
> first compare all traffic it receives to this set of rules hoping to get a
> match and not have to do any more compare operations.  This optimization
> could have a _very_ big impact on the performance of Snort.  Maybe call this
> subsystem 'ranker'?  It could be the core of the rule tuning system as well
> as the run-time rule manipulation/addition/deletion engine.

This would definitely be a big improvement over the status quo.  One of
the late additions to my proposal was the idea of cleanly segmenting the
matcher engine, for the express purpose of allowing people to experiment
easily with different optimization strategies.

If I were in the business of writing optimizers, and with any luck
so many other people will that I won't have to, then I would try to
generalise this strategy by somehow associating protocol prerequisites
to rules and then culling all rules whose protocol prereqs are not met.
(This is basically your proposal, only shifted a little.)  Right away,
you automatically reduce your workload by eliminating, e.g., all
DNS-specific rules that you know can't be met, since it's not even DNS
traffic.

It is the combination of several optimization strategies like this one
that will end up giving us just _unbelieveable_ performance gains in
rule matching.  I really hope and believe that this is possible.

> 	Some of the comments you make in the 'Design Principles' section of
> this document further supports this idea, as you mention that there is a
> need to be able support _all_ protocol types, not just IP, TCP, UDP, and
> ICMP.  This, I assume, would include the ability to identify the various
> layer-7 protocol types such as the ones mentioned above as well as some of
> the other application layer protocols you mention in the doc such as DNS and
> OSPF.

If you can send it on the wire, then you should be able to write a snort
rule to describe it.  I did not mention one single network protocol in
my proposal that I don't think one could write a protocol engine for.

> 	Another great point you make is the need for Snort to be
> multi-threaded.  If Snort is to be used to monitor very high bandwidth
> networks, such as Gigabit Ethernet or similar, then having the process run
> on one processor isn't going to cut it for much longer.  An 800MHz PIII
> stays buries on a saturated T3.  That ain't good. =)  Could this possibly be
> accomplished by using a threadpool with a user configurable (by the means of
> a config option in the snort.conf file) number of threads?

I don't think that it's so important that snort be threaded, necessarily,
as that it be _possible_ to thread snort.  Just as we shouldn't make
snort linux-specific, we shouldn't make snort SMP-hostile.  Now, does
this mean that us Linux folks have to do the OpenBSD port ourselves?
No, just that we shouldn't step on the toes of the people doing the
OpenBSD port.  Similarly, while I don't think that the core snort project
should necessarily spend a lot of effort on making snort multi-threaded,
I do think that we should pay a lot of attention to making our code
reentrant and otherwise making sure that those who need snort to be
threaded can make snort threaded.

As for the details of how you thread it, there are a thousand ways to skin
a cat.  Personally, I would use a threadpool, as you suggest.  However,
too often in the past I've made the wrong decisions on how to skin this
particular cat (how to divide up the workload in a threading model), so
caveat emptor.

> 	Again, great article.  Hope my comments are useful.  I'm sure I
> think of more after sleeping on it, and would love to get an eye on any
> updated versions you come up with.

Hey, thanks a lot!  Feedback like this is really helpful.  I'm glad that
you liked it and hope that future versions will be even better.

--
Todd Lewis
tlewis at ...255...





More information about the Snort-devel mailing list