[Snort-users] Snort spikes to 100% CPU followed by network latency

Cody Brugh cbrugh at ...11827...
Wed May 28 17:47:55 EDT 2014


Ok, who can examine the core files once I have them?



On Wed, May 28, 2014 at 5:43 PM, Russ Combs (rucombs) <rucombs at ...589...>wrote:

>
>  ------------------------------
> *From:* Cody Brugh [cbrugh at ...11827...]
> *Sent:* Wednesday, May 28, 2014 5:40 PM
>
> *To:* Russ Combs (rucombs)
> *Cc:* snort-users at lists.sourceforge.net
> *Subject:* Re: [Snort-users] Snort spikes to 100% CPU followed by network
> latency
>
>   Also note that when we see these CPU/latency spikes we have no alerts
> or drops that would easily tell us what is causing the problem. If it's not
> a rule what should I start turning off to try eliminate possible causes?
>  It's something that doesn't log or anything.
>
>  * Another option you have is to compile with debug and generate a core
> by sending a sig abort to the Snort process when it is in the 100% CPU
> state.  And you should capture a few to ensure they are in the same basic
> area.  Then the cores need to be examined for clues.
>
>
> On May 28, 2014, at 5:12 PM, "Russ Combs (rucombs)" <rucombs at ...589...>
> wrote:
>
>
>  ------------------------------
> *From:* Cody Brugh [cbrugh at ...11827...]
> *Sent:* Tuesday, May 27, 2014 6:30 PM
> *To:* Russ Combs (rucombs)
> *Cc:* snort-users at lists.sourceforge.net
> *Subject:* Re: [Snort-users] Snort spikes to 100% CPU followed by network
> latency
>
>   Russ,
>
>  I am still having latency/CPU spike issues even after enabling PPM
> configuration.... In the below logs where does it tell me which rule SIG ID
> so I can disable the rules that are causing me slowness?
>
>  * Actually, it doesn't tell you which rule.  Only which rule tree, which
> isn't terribly helpful.  That's because the rules are compiled into a
> different form to eliminate redundant checks.
>
>  It does tell you that the rule trees that are triggering the PPM events
> are around 55 usec although you have one shown which is higher.  What is
> your threshold set to?
>
>  Also, did you analyze the packets logged with the events to see what
> type of traffic this is?  You may be able to narrow it down and capture a
> whole session that triggers the problem and go from there.
>
>
> PPM: Rule-Event Pkt[20570] address=0x0x8200f80 re-enabled
> 05/27-18:26:27.989936
> PPM: Rule-Event Pkt[20570] suspended (72.194.88.31:53484 -> 10.2.14.21:80
> ).
> PPM: Rule-Event Pkt[20570] address=0x0x8200f80 used=52.7766 usecs
> suspended 05/27-18:26:27.989936
> PPM: Rule-Event Pkt[30776] address=0x0x3c28fb0 re-enabled
> 05/27-18:26:36.195293
> PPM: Rule-Event Pkt[36115] suspended (10.2.13.17:80 ->
> 164.113.217.51:41204).
> PPM: Rule-Event Pkt[36115] address=0x0x3c28fb0 used=54.8616 usecs
> suspended 05/27-18:26:40.813898
> PPM: Rule-Event Pkt[47155] address=0x0x8200f80 re-enabled
> 05/27-18:26:48.038002
> PPM: Rule-Event Pkt[48185] suspended (66.87.133.205:4163 -> 10.2.2.4:80).
> PPM: Rule-Event Pkt[48185] address=0x0x8200f80 used=53.992 usecs suspended
> 05/27-18:26:49.081371
> PPM: Rule-Event Pkt[60578] address=0x0x3c28fb0 re-enabled
> 05/27-18:27:00.867186
> PPM: Rule-Event Pkt[71509] suspended (10.2.13.48:80 -> 96.44.123.180:1046
> ).
> PPM: Rule-Event Pkt[71509] address=0x0x3c28fb0 used=54.4075 usecs
> suspended 05/27-18:27:08.522285
> PPM: Rule-Event Pkt[72989] address=0x0x8200f80 re-enabled
> 05/27-18:27:09.766234
> PPM: Rule-Event Pkt[76364] suspended (10.2.13.1:35718 -> 66.135.58.62:80).
> PPM: Rule-Event Pkt[76364] address=0x0x8200f80 used=76.2302 usecs
> suspended 05/27-18:27:15.130825
> PPM: Rule-Event Pkt[77634] suspended (66.8.180.174:64983 -> 10.2.13.17:80
> ).
> PPM: Rule-Event Pkt[77634] address=0x0x820c0b0 used=53.126 usecs suspended
> 05/27-18:27:17.180899
>
>
>
>
> On Fri, May 23, 2014 at 8:09 AM, Russ Combs (rucombs) <rucombs at ...589...>wrote:
>
>>
>>  ------------------------------
>> *From:* Cody Brugh [cbrugh at ...11827...]
>> *Sent:* Thursday, May 22, 2014 8:13 PM
>> *To:* snort-users at lists.sourceforge.net
>> *Subject:* [Snort-users] Snort spikes to 100% CPU followed by network
>> latency
>>
>>       Hello,
>>
>>  We have been running snort in-line for over a year now with no issues in
>> terms of latency or CPU usage.  Recently (over the past month) snort will
>> all of the sudden spike CPU usage up to 100% and network latency becomes
>> real bad, 1000+ms.
>>
>>  I am really not sure where to start on figuring out what is causing
>> this.  I am starting snort so it prints the alerts/drops on the console and
>> don't see any specific rule that would be causing this.
>>
>>  Any advise on this issue?
>>
>>  * Did you change your Snort version or configuration around the time
>> you started seeing the issue?  How frequently does this occur?  And when it
>> happens does it resolve itself or do you restart or what?
>>
>>  You can turn on PPM (config ppm ...) and enable the PPM rules (gid
>> 134).  That may catch the problem packet which you can log and examine for
>> clues.
>>
>>  Without any clues I'd first check for SDF and PCRE.  If you have SDF
>> (preprocessor sensitive_data) configured you can try commenting that out.
>>  If you have any pcre/O rules (PCRE override) you can try commenting those
>> out too.
>>
>>  Snort OS: CentOS, 64-bit
>>
>>   o"  )~   Version 2.9.6.1 GRE (Build 56)
>>    ''''    By Martin Roesch & The Snort Team:
>> http://www.snort.org/snort/snort-team
>>            Copyright (C) 2014 Cisco and/or its affiliates. All rights
>> reserved.
>>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>            Using libpcap version 1.0.0
>>            Using PCRE version: 7.8 2008-09-05
>>            Using ZLIB version: 1.2.3
>>
>> DAQ version: 2.0.2
>>
>>  Thanks!
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20140528/09cd2340/attachment.html>


More information about the Snort-users mailing list