[Snort-users] Anyones doomsday machine running low on IDS analyst tears?

Steven Sturges steve.sturges at ...1935...
Wed Oct 6 23:01:22 EDT 2010


The max_pattern_len is a memory option that is used in conjunction
with ac-split (introduced in 2.8.6).  In terms of the pattern matcher
itself, it gets near AC-FULL performance at the AC-BNFA memory.
Pretty significant when there are 10k+ rules in your configuration.
Performance-wise, reducing the size of the pattern matcher is two-fold.

-- Use of the faster AC algorithm
-- Fewer CPU cache misses during pattern matching phase

The whole idea of the fast pattern matcher is to use the most unique
patterns to select a set of "most likely" rules for further evaluation.

One could argue that of the pattern "User-Agent|3a| Mozilla/3.0
(compatible|3b| Indy Library)", the real unique part starts with
"Mozilla", so the pattern itself could be shortened.  Every HTTP
request has a User-Agent header these days, so that really isn't buying
you all that much in terms of uniqueness of the pattern itself.  And
when it comes to evaluation of the content, that's an extra 12
characters that need to be compared via boyer-moore.

In the specific rule example, if "User-Agent:" is a required part
of the pattern, using an offset of 13, length 20 would get you
"Mozilla/3.0 (compati" -- eg: content:"User-Agent|3a| Mozilla/3.0
(compatible|3b| Indy Library)"; fast_pattern:13,20;
Pretty unique & short.  Therefore less memory in pattern matcher --
that can now be used for Stream reassembly, gzip decompression, etc.

Will's solution in the follow-on post of fast_pattern:30,20 also works
and gives you the tail of the above pattern.

Or, split the two contents:

content:"User-Agent|3A| "; http_header; content:"Mozilla/3.0
(compatible|3b| Indy Library)"; distance:0; fast_pattern;

Or, increase the max pattern length in the default config.

Rule writers know the most unique part of the content patterns
they are using, especially as they relate to patterns in other
rules, and the parameters to fast_pattern give them the tools
that are needed.

Snort is highly configurable and users have so many options to work
with here to suit their specific ruleset, sensor, available memory,
network speed, etc.

Or, you can simply revert to using the slower AC-BNFA configuration
of yore.

Cheers.
-steve

> From: *Will Metcalf* <william.metcalf at ...11827...>
> Date: Wed, Oct 6, 2010 at 3:10 PM
> To: Emerging Threats Signatures <emerging-sigs at ...14333...>
> Cc: Snort Users <snort-users at lists.sourceforge.net>
> 
> 
> I kid... I kid.. But seriously ran into something sort of interesting
> playing snort conf options.  It seems that the default pm in snort has
> changed the the default pattern matcher to be ac-split via...
> 
> # Configure the detection engine  See the Snort Manual, Configuring
> Snort - Includes - Config
> config detection: search-method ac-split search-optimize max-pattern-len 20
> 
> If you are using your own custom rule sets/ET rules this means that if
> you previously had an override for fast_pattern's default cut-off of
> 20 via offset and length options via fast_pattern:<offset>,<length>;
> or you expected the entire unique pattern specified via fast_pattern,
> You are now limited to only 20 bytes for fast_pattern even if you set
> offset/length. So for example let us consider the following ET
> signature optimized for 2.8.6 using the default pm in 2.8.6 of
> ac-bnfa.
> 
> alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET TROJAN
> Delf Checkin via HTTP (8)"; flow:established,to_server;
> content:"POST"; http_method; content:".php"; http_uri; nocase;
> content:"User-Agent|3a| Mozilla/3.0 (compatible|3b| Indy Library)";
> http_header; fast_pattern; content:"name="; http_client_body; depth:5;
> classtype:trojan-activity;
> reference:url,doc.emergingthreats.net/2008268;
> reference:url,
> www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Delf;
> sid:2008268; rev:5;)
> 
>  Fast pattern matcher: HTTP Header content
>  Fast pattern set: yes
>  Fast pattern only: no
>  Negated: no
>  Pattern offset,length: none
>  Pattern truncated: no
>  Original pattern
>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>  Final pattern
>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
> 
> timestamp: 1286391049
> Rule Profile Statistics (all rules)
> ==========================================================
>   Num      SID GID Rev     Checks   Matches    Alerts
> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>   ===      === === ===     ======   =======    ======
> =========  =========  ========= ============
>     1  2008268   1   5         18         4         4
>         55        3.1        3.8          2.9
> 
> timestamp: 1286391054
> Rule Profile Statistics (all rules)
> ==========================================================
>   Num      SID GID Rev     Checks   Matches    Alerts
> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>   ===      === === ===     ======   =======    ======
> =========  =========  ========= ============
>     1  2008268   1   5         18         4         4
>          52        2.9        3.8          2.7
> 
> timestamp: 1286391058
> Rule Profile Statistics (all rules)
> ==========================================================
>   Num      SID GID Rev     Checks   Matches    Alerts
> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>   ===      === === ===     ======   =======    ======
> =========  =========  ========= ============
>     1  2008268   1   5         18         4         4
>        54        3.0        3.9          2.8
> 
> 
> Ok this is what I expect. I have a long match that should be fairly unique.
> 
> Lets modify the pm to be the one included in the VRT version of the
> default snort.conf and/or the 2.9.0 snort.conf.  We now get
> "User-Agent:|20|Mozilla/" as the match added to fast_pattern.  Uh
> oh... This isn't a very unique pattern is it, this will cause all
> packets that contain a firefox UA to be at least partially evaluated.
> 
> 1:2008268
>  Fast pattern matcher: HTTP Header content
>  Fast pattern set: yes
>  Fast pattern only: no
>  Negated: no
>  Pattern offset,length: none
>  Pattern truncated: 50 to 20 bytes
>  Original pattern
>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>  Final pattern
>    "User-Agent:|20|Mozilla/"
> 
> timestamp: 1286390771
> Rule Profile Statistics (all rules)
> ==========================================================
>   Num      SID GID Rev     Checks   Matches    Alerts
> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>   ===      === === ===     ======   =======    ======
> =========  =========  ========= ============
>     1  2008268   1   5      17381         4         4
> 10212        0.6        4.7          0.6
> 
> timestamp: 1286390775
> Rule Profile Statistics (all rules)
> ==========================================================
>   Num      SID GID Rev     Checks   Matches    Alerts
> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>   ===      === === ===     ======   =======    ======
> =========  =========  ========= ============
>     1  2008268   1   5      17381         4         4
> 9516        0.5        4.7          0.5
> 
> timestamp: 1286390779
> Rule Profile Statistics (all rules)
> ==========================================================
>   Num      SID GID Rev     Checks   Matches    Alerts
> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>   ===      === === ===     ======   =======    ======
> =========  =========  ========= ============
>     1  2008268   1   5      17381         4         4
> 9347        0.5        4.5          0.5
> 
> timestamp: 1286390784
> Rule Profile Statistics (all rules)
> ==========================================================
>   Num      SID GID Rev     Checks   Matches    Alerts
> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>   ===      === === ===     ======   =======    ======
> =========  =========  ========= ============
>     1  2008268   1   5      17381         4         4
> 9266        0.5        4.1          0.5
> 
> Evaluating the same pcap we now spend 20x more ticks evaluating the
> same rule.  I doubt that specifying a fast pattern offset and length
> will allow us to override the max-pattern-len 20 but lets give it a
> shot by specifying 0,50 as the offset and length.
> 
> snaplen = 65535
> 1:2008268
>  Fast pattern matcher: HTTP Header content
>  Fast pattern set: yes
>  Fast pattern only: no
>  Negated: no
>  Pattern <offset,length>: 0,50
>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>  Pattern truncated: 50 to 20 bytes
>  Original pattern
>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>  Final pattern
>    "User-Agent:|20|Mozilla/"
> 
> No dice.. So I guess the take away here is that if you are moving to a
> VRT snort.conf or a 2.9.0 ruleset and you are running custom rules I
> would pay real close attention to debug-print-fast-pattern output.  We
> are going through the poor performers now and making modifications
> where appropriate for ET rules, just thought folks might want to know
> ;-)...
> 
> Regards,
> 
> Will




More information about the Snort-users mailing list