[Snort-users] Anyones doomsday machine running low on IDS analyst tears?

Will Metcalf william.metcalf at ...11827...
Thu Oct 7 06:19:12 EDT 2010


Steve,

Thanks for the additional info.  It was more of a heads up for folks
switching to a VRT snort/2.9.0 conf to review their existing
rule-sets. If people have custom rules updating to this version of the
conf can have a significantly negative performance impact for some
sigs.  Of course if you don't update to include other portions of the
VRT/snort-2.9.0 config like extended_response_inspection you don't
have access to file_data so VRT sigs that rely on this won't fire.

Regards,

Will


On Wed, Oct 6, 2010 at 10:01 PM, Steven Sturges
<steve.sturges at ...1935...> wrote:
> The max_pattern_len is a memory option that is used in conjunction
> with ac-split (introduced in 2.8.6).  In terms of the pattern matcher
> itself, it gets near AC-FULL performance at the AC-BNFA memory.
> Pretty significant when there are 10k+ rules in your configuration.
> Performance-wise, reducing the size of the pattern matcher is two-fold.
>
> -- Use of the faster AC algorithm
> -- Fewer CPU cache misses during pattern matching phase
>
> The whole idea of the fast pattern matcher is to use the most unique
> patterns to select a set of "most likely" rules for further evaluation.
>
> One could argue that of the pattern "User-Agent|3a| Mozilla/3.0
> (compatible|3b| Indy Library)", the real unique part starts with
> "Mozilla", so the pattern itself could be shortened.  Every HTTP
> request has a User-Agent header these days, so that really isn't buying
> you all that much in terms of uniqueness of the pattern itself.  And
> when it comes to evaluation of the content, that's an extra 12
> characters that need to be compared via boyer-moore.
>
> In the specific rule example, if "User-Agent:" is a required part
> of the pattern, using an offset of 13, length 20 would get you
> "Mozilla/3.0 (compati" -- eg: content:"User-Agent|3a| Mozilla/3.0
> (compatible|3b| Indy Library)"; fast_pattern:13,20;
> Pretty unique & short.  Therefore less memory in pattern matcher --
> that can now be used for Stream reassembly, gzip decompression, etc.
>
> Will's solution in the follow-on post of fast_pattern:30,20 also works
> and gives you the tail of the above pattern.
>
> Or, split the two contents:
>
> content:"User-Agent|3A| "; http_header; content:"Mozilla/3.0
> (compatible|3b| Indy Library)"; distance:0; fast_pattern;
>
> Or, increase the max pattern length in the default config.
>
> Rule writers know the most unique part of the content patterns
> they are using, especially as they relate to patterns in other
> rules, and the parameters to fast_pattern give them the tools
> that are needed.
>
> Snort is highly configurable and users have so many options to work
> with here to suit their specific ruleset, sensor, available memory,
> network speed, etc.
>
> Or, you can simply revert to using the slower AC-BNFA configuration
> of yore.
>
> Cheers.
> -steve
>
>> From: *Will Metcalf* <william.metcalf at ...11827...>
>> Date: Wed, Oct 6, 2010 at 3:10 PM
>> To: Emerging Threats Signatures <emerging-sigs at ...14333...>
>> Cc: Snort Users <snort-users at lists.sourceforge.net>
>>
>>
>> I kid... I kid.. But seriously ran into something sort of interesting
>> playing snort conf options.  It seems that the default pm in snort has
>> changed the the default pattern matcher to be ac-split via...
>>
>> # Configure the detection engine  See the Snort Manual, Configuring
>> Snort - Includes - Config
>> config detection: search-method ac-split search-optimize max-pattern-len 20
>>
>> If you are using your own custom rule sets/ET rules this means that if
>> you previously had an override for fast_pattern's default cut-off of
>> 20 via offset and length options via fast_pattern:<offset>,<length>;
>> or you expected the entire unique pattern specified via fast_pattern,
>> You are now limited to only 20 bytes for fast_pattern even if you set
>> offset/length. So for example let us consider the following ET
>> signature optimized for 2.8.6 using the default pm in 2.8.6 of
>> ac-bnfa.
>>
>> alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET TROJAN
>> Delf Checkin via HTTP (8)"; flow:established,to_server;
>> content:"POST"; http_method; content:".php"; http_uri; nocase;
>> content:"User-Agent|3a| Mozilla/3.0 (compatible|3b| Indy Library)";
>> http_header; fast_pattern; content:"name="; http_client_body; depth:5;
>> classtype:trojan-activity;
>> reference:url,doc.emergingthreats.net/2008268;
>> reference:url,
>> www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Delf;
>> sid:2008268; rev:5;)
>>
>>  Fast pattern matcher: HTTP Header content
>>  Fast pattern set: yes
>>  Fast pattern only: no
>>  Negated: no
>>  Pattern offset,length: none
>>  Pattern truncated: no
>>  Original pattern
>>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>>  Final pattern
>>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>>
>> timestamp: 1286391049
>> Rule Profile Statistics (all rules)
>> ==========================================================
>>   Num      SID GID Rev     Checks   Matches    Alerts
>> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>>   ===      === === ===     ======   =======    ======
>> =========  =========  ========= ============
>>     1  2008268   1   5         18         4         4
>>         55        3.1        3.8          2.9
>>
>> timestamp: 1286391054
>> Rule Profile Statistics (all rules)
>> ==========================================================
>>   Num      SID GID Rev     Checks   Matches    Alerts
>> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>>   ===      === === ===     ======   =======    ======
>> =========  =========  ========= ============
>>     1  2008268   1   5         18         4         4
>>          52        2.9        3.8          2.7
>>
>> timestamp: 1286391058
>> Rule Profile Statistics (all rules)
>> ==========================================================
>>   Num      SID GID Rev     Checks   Matches    Alerts
>> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>>   ===      === === ===     ======   =======    ======
>> =========  =========  ========= ============
>>     1  2008268   1   5         18         4         4
>>        54        3.0        3.9          2.8
>>
>>
>> Ok this is what I expect. I have a long match that should be fairly unique.
>>
>> Lets modify the pm to be the one included in the VRT version of the
>> default snort.conf and/or the 2.9.0 snort.conf.  We now get
>> "User-Agent:|20|Mozilla/" as the match added to fast_pattern.  Uh
>> oh... This isn't a very unique pattern is it, this will cause all
>> packets that contain a firefox UA to be at least partially evaluated.
>>
>> 1:2008268
>>  Fast pattern matcher: HTTP Header content
>>  Fast pattern set: yes
>>  Fast pattern only: no
>>  Negated: no
>>  Pattern offset,length: none
>>  Pattern truncated: 50 to 20 bytes
>>  Original pattern
>>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>>  Final pattern
>>    "User-Agent:|20|Mozilla/"
>>
>> timestamp: 1286390771
>> Rule Profile Statistics (all rules)
>> ==========================================================
>>   Num      SID GID Rev     Checks   Matches    Alerts
>> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>>   ===      === === ===     ======   =======    ======
>> =========  =========  ========= ============
>>     1  2008268   1   5      17381         4         4
>> 10212        0.6        4.7          0.6
>>
>> timestamp: 1286390775
>> Rule Profile Statistics (all rules)
>> ==========================================================
>>   Num      SID GID Rev     Checks   Matches    Alerts
>> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>>   ===      === === ===     ======   =======    ======
>> =========  =========  ========= ============
>>     1  2008268   1   5      17381         4         4
>> 9516        0.5        4.7          0.5
>>
>> timestamp: 1286390779
>> Rule Profile Statistics (all rules)
>> ==========================================================
>>   Num      SID GID Rev     Checks   Matches    Alerts
>> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>>   ===      === === ===     ======   =======    ======
>> =========  =========  ========= ============
>>     1  2008268   1   5      17381         4         4
>> 9347        0.5        4.5          0.5
>>
>> timestamp: 1286390784
>> Rule Profile Statistics (all rules)
>> ==========================================================
>>   Num      SID GID Rev     Checks   Matches    Alerts
>> Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
>>   ===      === === ===     ======   =======    ======
>> =========  =========  ========= ============
>>     1  2008268   1   5      17381         4         4
>> 9266        0.5        4.1          0.5
>>
>> Evaluating the same pcap we now spend 20x more ticks evaluating the
>> same rule.  I doubt that specifying a fast pattern offset and length
>> will allow us to override the max-pattern-len 20 but lets give it a
>> shot by specifying 0,50 as the offset and length.
>>
>> snaplen = 65535
>> 1:2008268
>>  Fast pattern matcher: HTTP Header content
>>  Fast pattern set: yes
>>  Fast pattern only: no
>>  Negated: no
>>  Pattern <offset,length>: 0,50
>>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>>  Pattern truncated: 50 to 20 bytes
>>  Original pattern
>>    "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
>>  Final pattern
>>    "User-Agent:|20|Mozilla/"
>>
>> No dice.. So I guess the take away here is that if you are moving to a
>> VRT snort.conf or a 2.9.0 ruleset and you are running custom rules I
>> would pay real close attention to debug-print-fast-pattern output.  We
>> are going through the poor performers now and making modifications
>> where appropriate for ET rules, just thought folks might want to know
>> ;-)...
>>
>> Regards,
>>
>> Will
>




More information about the Snort-users mailing list