[Snort-devel] Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

Hui Cao (huica) huica at ...3461...
Thu Jun 4 11:23:57 EDT 2015


Try 

Assume snort pid is 1222

gdb /path/to/snort 1222

Best,
Hui.
On 6/4/15, 10:37 AM, "elof at ...969..." <elof at ...969...> wrote:

>
>An update:
>
>On a sensor where snort crashed with signal 6 three times, I downgraded
>daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem
>to 
>be in 'snort' or 'daq'.
>
>With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.
>
>
>This make me believe that there's something wrong in snort 2.9.7.3 and
>not in daq 2.0.5.
>
>
>
>On this sensor I have now done the opposite, upgraded daq to 2.0.5 and
>downgraded snort to 2.9.7.2 to see if I get any more signal 6.
>
>On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
>2.0.5 
>without chroot and uid/gid change, i.e. running as root, in order to
>create a core file, if the problem happen again.
>(if it doesn't happen on this sensor, I guess the problem lies somewhere
>in the chrooting code in snort. I know it has been updated between
>2.9.7.2 
>and 2.9.7.3)
>
>
>
>Russ C also wrote:
>> Elof - since this is happening frequently, you could try attaching the
>> debugger to one of your Snort processes and wait wait for segfault.
>
>I know too little about debugging. :-/ Can you give me instructions or
>point me to a guide that describes the steps I should take?
>
>
>
>/Elof
>
>
>On Thu, 4 Jun 2015, elof at ...969... wrote:
>
>>
>> Five different sensors have now had bus errors (signal 10), segmentation
>> faults (signal 11) and even signal 6 (SIGABRT).
>>
>> My snort config uses both chroot and dropping user privileges, so even
>>if
>> I start out as root with ulimit unlimited, this doesn't seem to be in
>>effect
>> after the chroot/uid-change.
>>
>> So currently I have no core-file to debug. :-/
>>
>> Anyone know how to set the ulimits for a chrooted and uid/gid-changed
>> process in FreeBSD?
>>
>> /Elof
>>
>>
>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>
>>>
>>> Hi Hui!
>>>
>>> Yes, the dynamic engine/preproc files are updated as well.
>>>
>>> Last night the problem reocurred, so this seem to be reproduceable.
>>>Good.
>>> Then there's a good chance this problem can be sorted out.
>>>
>>>
>>> A few minutes ago a signal 10 happened on another sensor (running
>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort
>>> 2.9.7.3 and not in the hardware nor in FreeBSD.
>>>
>>>
>>> I will compile a debug-snort and try to generate core files.
>>> I'll let you know the outcome next week.
>>>
>>> /Elof
>>>
>>>
>>> On Wed, 3 Jun 2015, Hui cao wrote:
>>>
>>>> Hi Elof,
>>>>
>>>> Are snort and snort dynamic preprocessors are in sync?
>>>>
>>>> If so, can you help us get a backtrace from the crush? You need
>>>> 1)  build snort with ./configure --enable-debug
>>>> 2)  allowing core dump (ulimit -c unlimited)
>>>> 3) run the snort
>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb command
>>>>line
>>>>
>>>> Best,
>>>> Hui.
>>>>
>>>>
>>>> On 06/03/2015 05:51 AM, elof at ...969... wrote:
>>>>> Hi all!
>>>>>
>>>>> This is just a report to inform that after I updated snort and DAQ
>>>>>to the
>>>>> latest versions, one of my sensors started throwing signal 10 (bus
>>>>>error)
>>>>> and signal 11 (segmentation fault).
>>>>>
>>>>> # uptime
>>>>> 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
>>>>> # dmesg | grep snort
>>>>> pid 1183 (snort), uid 100: exited on signal 11
>>>>> pid 16920 (snort), uid 100: exited on signal 11
>>>>> pid 17502 (snort), uid 100: exited on signal 11
>>>>> pid 18862 (snort), uid 100: exited on signal 11
>>>>> pid 20223 (snort), uid 100: exited on signal 11
>>>>> pid 20927 (snort), uid 100: exited on signal 11
>>>>> pid 1193 (snort), uid 100: exited on signal 11
>>>>> pid 2447 (snort), uid 100: exited on signal 11
>>>>> pid 3811 (snort), uid 100: exited on signal 10
>>>>> pid 7881 (snort), uid 100: exited on signal 11
>>>>> pid 9252 (snort), uid 100: exited on signal 10
>>>>> pid 25593 (snort), uid 100: exited on signal 11
>>>>> pid 26627 (snort), uid 100: exited on signal 11
>>>>> pid 56658 (snort), uid 100: exited on signal 11
>>>>> pid 57237 (snort), uid 100: exited on signal 10
>>>>> pid 58595 (snort), uid 100: exited on signal 11
>>>>> pid 68639 (snort), uid 100: exited on signal 11
>>>>> pid 70008 (snort), uid 100: exited on signal 11
>>>>> pid 71361 (snort), uid 100: exited on signal 10
>>>>> pid 72725 (snort), uid 100: exited on signal 11
>>>>>
>>>>> 20 crashes in a day...
>>>>> A reboot didn't help.
>>>>>
>>>>> This sensor has never behaved like this during its lifetime (1 year).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> FreeBSD 9.3 amd64
>>>>>
>>>>>      ,,_     -*> Snort! <*-
>>>>>     o"  )~   Version 2.9.7.3 (Build 217)
>>>>>      ''''    By Martin Roesch & The Snort Team:
>>>>> http://www.snort.org/contact#team
>>>>>              Copyright (C) 2014-2015 Cisco and/or its affiliates.
>>>>>All rights
>>>>> reserved.
>>>>>              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>>              Using libpcap version 1.4.0
>>>>>              Using PCRE version: 8.37 2015-04-28
>>>>>              Using ZLIB version: 1.2.8
>>>>>
>>>>> daq-2.0.5
>>>>>
>>>>>
>>>>>
>>>>> Bus errors are quite unusual in general, so I'll keep looking at
>>>>>this,
>>>>> trying to see if it is e.g. paging errors.
>>>>> It doesn't look like it though:
>>>>> # swapinfo
>>>>> Device          1K-blocks     Used    Avail Capacity
>>>>> /dev/mirror/swap   4194300        0  4194300     0%
>>>>>
>>>>> The machine doesn't seem to be overheated either:
>>>>> System Temp:	30 degrees C
>>>>> Peripheral Temp: 40 degrees C
>>>>> CPU Temp: Low
>>>>>
>>>>>
>>>>> If you need me to do something special to debug this further, let me
>>>>>know.
>>>>>
>>>>>
>>>>> PS. It is only one sensor, out of 20, that behaves like this. So
>>>>>perhaps
>>>>> it is something in the mirrored traffic that make DAQ or snort point
>>>>>at
>>>>> illegal memory addresses and crash.
>>>>> Or this particular machine is having hardware issues. However, it is
>>>>> strange that those hw-issues should suddenly start right after I
>>>>>updated
>>>>> the software on the machine...
>>>>>
>>>>> When I write this, the current snort process has been alive for 5
>>>>>hours.
>>>>> It's going to be interesting to see if the traffic tonight will
>>>>>cause it
>>>>> to crash many times again.
>>>>>
>>>>> /Elof
>>>>>
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>--------
>>>>> _______________________________________________
>>>>> Snort-devel mailing list
>>>>> Snort-devel at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>> Archive:
>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>
>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>
>>>>
>>>> 
>>>>-----------------------------------------------------------------------
>>>>-------
>>>> _______________________________________________
>>>> Snort-devel mailing list
>>>> Snort-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>> Archive:
>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>
>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>
>>>
>>> 
>>>------------------------------------------------------------------------
>>>------
>>> _______________________________________________
>>> Snort-devel mailing list
>>> Snort-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>> Archive:
>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>
>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>
>>
>> 
>>-------------------------------------------------------------------------
>>-----
>> _______________________________________________
>> Snort-devel mailing list
>> Snort-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>> Archive:
>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>
>> Please visit http://blog.snort.org for the latest news about Snort!
>>





More information about the Snort-devel mailing list