[Snort-devel] Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

Hui Cao (huica) huica at ...3461...
Thu Jun 4 11:40:34 EDT 2015


That¹s cool. All looks good to me. No need to do more things...

Best,
Hui

On 6/4/15, 11:35 AM, "elof at ...969..." <elof at ...969...> wrote:

>
>Hi Hui.
>
>That much I know. It is the debugging steps I'm curious about.
>
>(I think you forgot one important first command: continue )
>
>
>Is this a good start:
>
>gdb /path/to/snort 1222
>(gdb) set logging file gdb-snort.txt
>(gdb) set logging on
>(gdb) continue
>
><wait for it to crash>
>
>(gdb) backtrace full
>(gdb) info registers
>(gdb) x/16i $pc
>(gdb) thread apply all backtrace
>(gdb) quit
>
>Email the report.
>
>
>Should I prepare more stuff before the 'continue'?
>Like "handle SIG33 pass nostop noprint" or something?
>
>/Elof
>
>
>On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>
>> Try
>>
>> Assume snort pid is 1222
>>
>> gdb /path/to/snort 1222
>>
>> Best,
>> Hui.
>> On 6/4/15, 10:37 AM, "elof at ...969..." <elof at ...969...> wrote:
>>
>>>
>>> An update:
>>>
>>> On a sensor where snort crashed with signal 6 three times, I downgraded
>>> daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem
>>> to
>>> be in 'snort' or 'daq'.
>>>
>>> With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.
>>>
>>>
>>> This make me believe that there's something wrong in snort 2.9.7.3 and
>>> not in daq 2.0.5.
>>>
>>>
>>>
>>> On this sensor I have now done the opposite, upgraded daq to 2.0.5 and
>>> downgraded snort to 2.9.7.2 to see if I get any more signal 6.
>>>
>>> On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
>>> 2.0.5
>>> without chroot and uid/gid change, i.e. running as root, in order to
>>> create a core file, if the problem happen again.
>>> (if it doesn't happen on this sensor, I guess the problem lies
>>>somewhere
>>> in the chrooting code in snort. I know it has been updated between
>>> 2.9.7.2
>>> and 2.9.7.3)
>>>
>>>
>>>
>>> Russ C also wrote:
>>>> Elof - since this is happening frequently, you could try attaching the
>>>> debugger to one of your Snort processes and wait wait for segfault.
>>>
>>> I know too little about debugging. :-/ Can you give me instructions or
>>> point me to a guide that describes the steps I should take?
>>>
>>>
>>>
>>> /Elof
>>>
>>>
>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>
>>>>
>>>> Five different sensors have now had bus errors (signal 10),
>>>>segmentation
>>>> faults (signal 11) and even signal 6 (SIGABRT).
>>>>
>>>> My snort config uses both chroot and dropping user privileges, so even
>>>> if
>>>> I start out as root with ulimit unlimited, this doesn't seem to be in
>>>> effect
>>>> after the chroot/uid-change.
>>>>
>>>> So currently I have no core-file to debug. :-/
>>>>
>>>> Anyone know how to set the ulimits for a chrooted and uid/gid-changed
>>>> process in FreeBSD?
>>>>
>>>> /Elof
>>>>
>>>>
>>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>>
>>>>>
>>>>> Hi Hui!
>>>>>
>>>>> Yes, the dynamic engine/preproc files are updated as well.
>>>>>
>>>>> Last night the problem reocurred, so this seem to be reproduceable.
>>>>> Good.
>>>>> Then there's a good chance this problem can be sorted out.
>>>>>
>>>>>
>>>>> A few minutes ago a signal 10 happened on another sensor (running
>>>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort
>>>>> 2.9.7.3 and not in the hardware nor in FreeBSD.
>>>>>
>>>>>
>>>>> I will compile a debug-snort and try to generate core files.
>>>>> I'll let you know the outcome next week.
>>>>>
>>>>> /Elof
>>>>>
>>>>>
>>>>> On Wed, 3 Jun 2015, Hui cao wrote:
>>>>>
>>>>>> Hi Elof,
>>>>>>
>>>>>> Are snort and snort dynamic preprocessors are in sync?
>>>>>>
>>>>>> If so, can you help us get a backtrace from the crush? You need
>>>>>> 1)  build snort with ./configure --enable-debug
>>>>>> 2)  allowing core dump (ulimit -c unlimited)
>>>>>> 3) run the snort
>>>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb command
>>>>>> line
>>>>>>
>>>>>> Best,
>>>>>> Hui.
>>>>>>
>>>>>>
>>>>>> On 06/03/2015 05:51 AM, elof at ...969... wrote:
>>>>>>> Hi all!
>>>>>>>
>>>>>>> This is just a report to inform that after I updated snort and DAQ
>>>>>>> to the
>>>>>>> latest versions, one of my sensors started throwing signal 10 (bus
>>>>>>> error)
>>>>>>> and signal 11 (segmentation fault).
>>>>>>>
>>>>>>> # uptime
>>>>>>> 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
>>>>>>> # dmesg | grep snort
>>>>>>> pid 1183 (snort), uid 100: exited on signal 11
>>>>>>> pid 16920 (snort), uid 100: exited on signal 11
>>>>>>> pid 17502 (snort), uid 100: exited on signal 11
>>>>>>> pid 18862 (snort), uid 100: exited on signal 11
>>>>>>> pid 20223 (snort), uid 100: exited on signal 11
>>>>>>> pid 20927 (snort), uid 100: exited on signal 11
>>>>>>> pid 1193 (snort), uid 100: exited on signal 11
>>>>>>> pid 2447 (snort), uid 100: exited on signal 11
>>>>>>> pid 3811 (snort), uid 100: exited on signal 10
>>>>>>> pid 7881 (snort), uid 100: exited on signal 11
>>>>>>> pid 9252 (snort), uid 100: exited on signal 10
>>>>>>> pid 25593 (snort), uid 100: exited on signal 11
>>>>>>> pid 26627 (snort), uid 100: exited on signal 11
>>>>>>> pid 56658 (snort), uid 100: exited on signal 11
>>>>>>> pid 57237 (snort), uid 100: exited on signal 10
>>>>>>> pid 58595 (snort), uid 100: exited on signal 11
>>>>>>> pid 68639 (snort), uid 100: exited on signal 11
>>>>>>> pid 70008 (snort), uid 100: exited on signal 11
>>>>>>> pid 71361 (snort), uid 100: exited on signal 10
>>>>>>> pid 72725 (snort), uid 100: exited on signal 11
>>>>>>>
>>>>>>> 20 crashes in a day...
>>>>>>> A reboot didn't help.
>>>>>>>
>>>>>>> This sensor has never behaved like this during its lifetime (1
>>>>>>>year).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> FreeBSD 9.3 amd64
>>>>>>>
>>>>>>>      ,,_     -*> Snort! <*-
>>>>>>>     o"  )~   Version 2.9.7.3 (Build 217)
>>>>>>>      ''''    By Martin Roesch & The Snort Team:
>>>>>>> http://www.snort.org/contact#team
>>>>>>>              Copyright (C) 2014-2015 Cisco and/or its affiliates.
>>>>>>> All rights
>>>>>>> reserved.
>>>>>>>              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>>>>              Using libpcap version 1.4.0
>>>>>>>              Using PCRE version: 8.37 2015-04-28
>>>>>>>              Using ZLIB version: 1.2.8
>>>>>>>
>>>>>>> daq-2.0.5
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Bus errors are quite unusual in general, so I'll keep looking at
>>>>>>> this,
>>>>>>> trying to see if it is e.g. paging errors.
>>>>>>> It doesn't look like it though:
>>>>>>> # swapinfo
>>>>>>> Device          1K-blocks     Used    Avail Capacity
>>>>>>> /dev/mirror/swap   4194300        0  4194300     0%
>>>>>>>
>>>>>>> The machine doesn't seem to be overheated either:
>>>>>>> System Temp:	30 degrees C
>>>>>>> Peripheral Temp: 40 degrees C
>>>>>>> CPU Temp: Low
>>>>>>>
>>>>>>>
>>>>>>> If you need me to do something special to debug this further, let
>>>>>>>me
>>>>>>> know.
>>>>>>>
>>>>>>>
>>>>>>> PS. It is only one sensor, out of 20, that behaves like this. So
>>>>>>> perhaps
>>>>>>> it is something in the mirrored traffic that make DAQ or snort
>>>>>>>point
>>>>>>> at
>>>>>>> illegal memory addresses and crash.
>>>>>>> Or this particular machine is having hardware issues. However, it
>>>>>>>is
>>>>>>> strange that those hw-issues should suddenly start right after I
>>>>>>> updated
>>>>>>> the software on the machine...
>>>>>>>
>>>>>>> When I write this, the current snort process has been alive for 5
>>>>>>> hours.
>>>>>>> It's going to be interesting to see if the traffic tonight will
>>>>>>> cause it
>>>>>>> to crash many times again.
>>>>>>>
>>>>>>> /Elof
>>>>>>>
>>>>>>>
>>>>>>> 
>>>>>>>--------------------------------------------------------------------
>>>>>>>--
>>>>>>> --------
>>>>>>> _______________________________________________
>>>>>>> Snort-devel mailing list
>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>> Archive:
>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>>
>>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>---------------------------------------------------------------------
>>>>>>--
>>>>>> -------
>>>>>> _______________________________________________
>>>>>> Snort-devel mailing list
>>>>>> Snort-devel at lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>> Archive:
>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>
>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>
>>>>>
>>>>>
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>--
>>>>> ------
>>>>> _______________________________________________
>>>>> Snort-devel mailing list
>>>>> Snort-devel at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>> Archive:
>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>
>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>
>>>>
>>>>
>>>> 
>>>>-----------------------------------------------------------------------
>>>>--
>>>> -----
>>>> _______________________________________________
>>>> Snort-devel mailing list
>>>> Snort-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>> Archive:
>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>
>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>
>>





More information about the Snort-devel mailing list