[Snort-devel] Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

elof at ...969... elof at ...969...
Thu Jun 4 10:37:55 EDT 2015


An update:

On a sensor where snort crashed with signal 6 three times, I downgraded 
daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem to 
be in 'snort' or 'daq'.

With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.


This make me believe that there's something wrong in snort 2.9.7.3 and 
not in daq 2.0.5.



On this sensor I have now done the opposite, upgraded daq to 2.0.5 and 
downgraded snort to 2.9.7.2 to see if I get any more signal 6.

On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq 2.0.5 
without chroot and uid/gid change, i.e. running as root, in order to 
create a core file, if the problem happen again.
(if it doesn't happen on this sensor, I guess the problem lies somewhere 
in the chrooting code in snort. I know it has been updated between 2.9.7.2 
and 2.9.7.3)



Russ C also wrote:
> Elof - since this is happening frequently, you could try attaching the
> debugger to one of your Snort processes and wait wait for segfault.

I know too little about debugging. :-/ Can you give me instructions or 
point me to a guide that describes the steps I should take?



/Elof


On Thu, 4 Jun 2015, elof at ...969... wrote:

>
> Five different sensors have now had bus errors (signal 10), segmentation
> faults (signal 11) and even signal 6 (SIGABRT).
>
> My snort config uses both chroot and dropping user privileges, so even if
> I start out as root with ulimit unlimited, this doesn't seem to be in effect
> after the chroot/uid-change.
>
> So currently I have no core-file to debug. :-/
>
> Anyone know how to set the ulimits for a chrooted and uid/gid-changed
> process in FreeBSD?
>
> /Elof
>
>
> On Thu, 4 Jun 2015, elof at ...969... wrote:
>
>>
>> Hi Hui!
>>
>> Yes, the dynamic engine/preproc files are updated as well.
>>
>> Last night the problem reocurred, so this seem to be reproduceable. Good.
>> Then there's a good chance this problem can be sorted out.
>>
>>
>> A few minutes ago a signal 10 happened on another sensor (running
>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort
>> 2.9.7.3 and not in the hardware nor in FreeBSD.
>>
>>
>> I will compile a debug-snort and try to generate core files.
>> I'll let you know the outcome next week.
>>
>> /Elof
>>
>>
>> On Wed, 3 Jun 2015, Hui cao wrote:
>>
>>> Hi Elof,
>>>
>>> Are snort and snort dynamic preprocessors are in sync?
>>>
>>> If so, can you help us get a backtrace from the crush? You need
>>> 1)  build snort with ./configure --enable-debug
>>> 2)  allowing core dump (ulimit -c unlimited)
>>> 3) run the snort
>>> 4) use "gdb snort core_file " and them type "bt" in the gdb command line
>>>
>>> Best,
>>> Hui.
>>>
>>>
>>> On 06/03/2015 05:51 AM, elof at ...969... wrote:
>>>> Hi all!
>>>>
>>>> This is just a report to inform that after I updated snort and DAQ to the
>>>> latest versions, one of my sensors started throwing signal 10 (bus error)
>>>> and signal 11 (segmentation fault).
>>>>
>>>> # uptime
>>>> 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
>>>> # dmesg | grep snort
>>>> pid 1183 (snort), uid 100: exited on signal 11
>>>> pid 16920 (snort), uid 100: exited on signal 11
>>>> pid 17502 (snort), uid 100: exited on signal 11
>>>> pid 18862 (snort), uid 100: exited on signal 11
>>>> pid 20223 (snort), uid 100: exited on signal 11
>>>> pid 20927 (snort), uid 100: exited on signal 11
>>>> pid 1193 (snort), uid 100: exited on signal 11
>>>> pid 2447 (snort), uid 100: exited on signal 11
>>>> pid 3811 (snort), uid 100: exited on signal 10
>>>> pid 7881 (snort), uid 100: exited on signal 11
>>>> pid 9252 (snort), uid 100: exited on signal 10
>>>> pid 25593 (snort), uid 100: exited on signal 11
>>>> pid 26627 (snort), uid 100: exited on signal 11
>>>> pid 56658 (snort), uid 100: exited on signal 11
>>>> pid 57237 (snort), uid 100: exited on signal 10
>>>> pid 58595 (snort), uid 100: exited on signal 11
>>>> pid 68639 (snort), uid 100: exited on signal 11
>>>> pid 70008 (snort), uid 100: exited on signal 11
>>>> pid 71361 (snort), uid 100: exited on signal 10
>>>> pid 72725 (snort), uid 100: exited on signal 11
>>>>
>>>> 20 crashes in a day...
>>>> A reboot didn't help.
>>>>
>>>> This sensor has never behaved like this during its lifetime (1 year).
>>>>
>>>>
>>>>
>>>>
>>>> FreeBSD 9.3 amd64
>>>>
>>>>      ,,_     -*> Snort! <*-
>>>>     o"  )~   Version 2.9.7.3 (Build 217)
>>>>      ''''    By Martin Roesch & The Snort Team:
>>>> http://www.snort.org/contact#team
>>>>              Copyright (C) 2014-2015 Cisco and/or its affiliates. All rights
>>>> reserved.
>>>>              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>              Using libpcap version 1.4.0
>>>>              Using PCRE version: 8.37 2015-04-28
>>>>              Using ZLIB version: 1.2.8
>>>>
>>>> daq-2.0.5
>>>>
>>>>
>>>>
>>>> Bus errors are quite unusual in general, so I'll keep looking at this,
>>>> trying to see if it is e.g. paging errors.
>>>> It doesn't look like it though:
>>>> # swapinfo
>>>> Device          1K-blocks     Used    Avail Capacity
>>>> /dev/mirror/swap   4194300        0  4194300     0%
>>>>
>>>> The machine doesn't seem to be overheated either:
>>>> System Temp:	30 degrees C
>>>> Peripheral Temp: 40 degrees C
>>>> CPU Temp: Low
>>>>
>>>>
>>>> If you need me to do something special to debug this further, let me know.
>>>>
>>>>
>>>> PS. It is only one sensor, out of 20, that behaves like this. So perhaps
>>>> it is something in the mirrored traffic that make DAQ or snort point at
>>>> illegal memory addresses and crash.
>>>> Or this particular machine is having hardware issues. However, it is
>>>> strange that those hw-issues should suddenly start right after I updated
>>>> the software on the machine...
>>>>
>>>> When I write this, the current snort process has been alive for 5 hours.
>>>> It's going to be interesting to see if the traffic tonight will cause it
>>>> to crash many times again.
>>>>
>>>> /Elof
>>>>
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Snort-devel mailing list
>>>> Snort-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>> Archive:
>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>
>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Snort-devel mailing list
>>> Snort-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>> Archive:
>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>
>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Snort-devel mailing list
>> Snort-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>> Archive:
>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>
>> Please visit http://blog.snort.org for the latest news about Snort!
>>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Snort-devel mailing list
> Snort-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/snort-devel
> Archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>
> Please visit http://blog.snort.org for the latest news about Snort!
>




More information about the Snort-devel mailing list