[Snort-devel] Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

elof at ...969... elof at ...969...
Thu Jun 4 11:35:45 EDT 2015


Hi Hui.

That much I know. It is the debugging steps I'm curious about.

(I think you forgot one important first command: continue )


Is this a good start:

gdb /path/to/snort 1222
(gdb) set logging file gdb-snort.txt
(gdb) set logging on
(gdb) continue

<wait for it to crash>

(gdb) backtrace full
(gdb) info registers
(gdb) x/16i $pc
(gdb) thread apply all backtrace
(gdb) quit

Email the report.


Should I prepare more stuff before the 'continue'?
Like "handle SIG33 pass nostop noprint" or something?

/Elof


On Thu, 4 Jun 2015, Hui Cao (huica) wrote:

> Try
>
> Assume snort pid is 1222
>
> gdb /path/to/snort 1222
>
> Best,
> Hui.
> On 6/4/15, 10:37 AM, "elof at ...969..." <elof at ...969...> wrote:
>
>>
>> An update:
>>
>> On a sensor where snort crashed with signal 6 three times, I downgraded
>> daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem
>> to
>> be in 'snort' or 'daq'.
>>
>> With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.
>>
>>
>> This make me believe that there's something wrong in snort 2.9.7.3 and
>> not in daq 2.0.5.
>>
>>
>>
>> On this sensor I have now done the opposite, upgraded daq to 2.0.5 and
>> downgraded snort to 2.9.7.2 to see if I get any more signal 6.
>>
>> On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
>> 2.0.5
>> without chroot and uid/gid change, i.e. running as root, in order to
>> create a core file, if the problem happen again.
>> (if it doesn't happen on this sensor, I guess the problem lies somewhere
>> in the chrooting code in snort. I know it has been updated between
>> 2.9.7.2
>> and 2.9.7.3)
>>
>>
>>
>> Russ C also wrote:
>>> Elof - since this is happening frequently, you could try attaching the
>>> debugger to one of your Snort processes and wait wait for segfault.
>>
>> I know too little about debugging. :-/ Can you give me instructions or
>> point me to a guide that describes the steps I should take?
>>
>>
>>
>> /Elof
>>
>>
>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>
>>>
>>> Five different sensors have now had bus errors (signal 10), segmentation
>>> faults (signal 11) and even signal 6 (SIGABRT).
>>>
>>> My snort config uses both chroot and dropping user privileges, so even
>>> if
>>> I start out as root with ulimit unlimited, this doesn't seem to be in
>>> effect
>>> after the chroot/uid-change.
>>>
>>> So currently I have no core-file to debug. :-/
>>>
>>> Anyone know how to set the ulimits for a chrooted and uid/gid-changed
>>> process in FreeBSD?
>>>
>>> /Elof
>>>
>>>
>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>
>>>>
>>>> Hi Hui!
>>>>
>>>> Yes, the dynamic engine/preproc files are updated as well.
>>>>
>>>> Last night the problem reocurred, so this seem to be reproduceable.
>>>> Good.
>>>> Then there's a good chance this problem can be sorted out.
>>>>
>>>>
>>>> A few minutes ago a signal 10 happened on another sensor (running
>>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort
>>>> 2.9.7.3 and not in the hardware nor in FreeBSD.
>>>>
>>>>
>>>> I will compile a debug-snort and try to generate core files.
>>>> I'll let you know the outcome next week.
>>>>
>>>> /Elof
>>>>
>>>>
>>>> On Wed, 3 Jun 2015, Hui cao wrote:
>>>>
>>>>> Hi Elof,
>>>>>
>>>>> Are snort and snort dynamic preprocessors are in sync?
>>>>>
>>>>> If so, can you help us get a backtrace from the crush? You need
>>>>> 1)  build snort with ./configure --enable-debug
>>>>> 2)  allowing core dump (ulimit -c unlimited)
>>>>> 3) run the snort
>>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb command
>>>>> line
>>>>>
>>>>> Best,
>>>>> Hui.
>>>>>
>>>>>
>>>>> On 06/03/2015 05:51 AM, elof at ...969... wrote:
>>>>>> Hi all!
>>>>>>
>>>>>> This is just a report to inform that after I updated snort and DAQ
>>>>>> to the
>>>>>> latest versions, one of my sensors started throwing signal 10 (bus
>>>>>> error)
>>>>>> and signal 11 (segmentation fault).
>>>>>>
>>>>>> # uptime
>>>>>> 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
>>>>>> # dmesg | grep snort
>>>>>> pid 1183 (snort), uid 100: exited on signal 11
>>>>>> pid 16920 (snort), uid 100: exited on signal 11
>>>>>> pid 17502 (snort), uid 100: exited on signal 11
>>>>>> pid 18862 (snort), uid 100: exited on signal 11
>>>>>> pid 20223 (snort), uid 100: exited on signal 11
>>>>>> pid 20927 (snort), uid 100: exited on signal 11
>>>>>> pid 1193 (snort), uid 100: exited on signal 11
>>>>>> pid 2447 (snort), uid 100: exited on signal 11
>>>>>> pid 3811 (snort), uid 100: exited on signal 10
>>>>>> pid 7881 (snort), uid 100: exited on signal 11
>>>>>> pid 9252 (snort), uid 100: exited on signal 10
>>>>>> pid 25593 (snort), uid 100: exited on signal 11
>>>>>> pid 26627 (snort), uid 100: exited on signal 11
>>>>>> pid 56658 (snort), uid 100: exited on signal 11
>>>>>> pid 57237 (snort), uid 100: exited on signal 10
>>>>>> pid 58595 (snort), uid 100: exited on signal 11
>>>>>> pid 68639 (snort), uid 100: exited on signal 11
>>>>>> pid 70008 (snort), uid 100: exited on signal 11
>>>>>> pid 71361 (snort), uid 100: exited on signal 10
>>>>>> pid 72725 (snort), uid 100: exited on signal 11
>>>>>>
>>>>>> 20 crashes in a day...
>>>>>> A reboot didn't help.
>>>>>>
>>>>>> This sensor has never behaved like this during its lifetime (1 year).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> FreeBSD 9.3 amd64
>>>>>>
>>>>>>      ,,_     -*> Snort! <*-
>>>>>>     o"  )~   Version 2.9.7.3 (Build 217)
>>>>>>      ''''    By Martin Roesch & The Snort Team:
>>>>>> http://www.snort.org/contact#team
>>>>>>              Copyright (C) 2014-2015 Cisco and/or its affiliates.
>>>>>> All rights
>>>>>> reserved.
>>>>>>              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>>>              Using libpcap version 1.4.0
>>>>>>              Using PCRE version: 8.37 2015-04-28
>>>>>>              Using ZLIB version: 1.2.8
>>>>>>
>>>>>> daq-2.0.5
>>>>>>
>>>>>>
>>>>>>
>>>>>> Bus errors are quite unusual in general, so I'll keep looking at
>>>>>> this,
>>>>>> trying to see if it is e.g. paging errors.
>>>>>> It doesn't look like it though:
>>>>>> # swapinfo
>>>>>> Device          1K-blocks     Used    Avail Capacity
>>>>>> /dev/mirror/swap   4194300        0  4194300     0%
>>>>>>
>>>>>> The machine doesn't seem to be overheated either:
>>>>>> System Temp:	30 degrees C
>>>>>> Peripheral Temp: 40 degrees C
>>>>>> CPU Temp: Low
>>>>>>
>>>>>>
>>>>>> If you need me to do something special to debug this further, let me
>>>>>> know.
>>>>>>
>>>>>>
>>>>>> PS. It is only one sensor, out of 20, that behaves like this. So
>>>>>> perhaps
>>>>>> it is something in the mirrored traffic that make DAQ or snort point
>>>>>> at
>>>>>> illegal memory addresses and crash.
>>>>>> Or this particular machine is having hardware issues. However, it is
>>>>>> strange that those hw-issues should suddenly start right after I
>>>>>> updated
>>>>>> the software on the machine...
>>>>>>
>>>>>> When I write this, the current snort process has been alive for 5
>>>>>> hours.
>>>>>> It's going to be interesting to see if the traffic tonight will
>>>>>> cause it
>>>>>> to crash many times again.
>>>>>>
>>>>>> /Elof
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>> --------
>>>>>> _______________________________________________
>>>>>> Snort-devel mailing list
>>>>>> Snort-devel at lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>> Archive:
>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>
>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>
>>>>>
>>>>>
>>>>> -----------------------------------------------------------------------
>>>>> -------
>>>>> _______________________________________________
>>>>> Snort-devel mailing list
>>>>> Snort-devel at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>> Archive:
>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>
>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> ------
>>>> _______________________________________________
>>>> Snort-devel mailing list
>>>> Snort-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>> Archive:
>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>
>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>
>>>
>>>
>>> -------------------------------------------------------------------------
>>> -----
>>> _______________________________________________
>>> Snort-devel mailing list
>>> Snort-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>> Archive:
>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>
>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>
>




More information about the Snort-devel mailing list