[Snort-devel] Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

elof at ...969... elof at ...969...
Thu Jun 4 18:21:16 EDT 2015


Ok, this sensor is now running with these lines commented out:

#preprocessor rpc_decode: 111 32770 32771 32772 32773 32774 32775 32776 
32777 32778 32779 no_alert_multiple_requests no_alert_large_fragments 
no_alert_incomplete
#preprocessor dcerpc2: memcap 102400, events [co ]
#preprocessor dcerpc2_server: default, policy WinXP, \
#    detect [smb [139,445], tcp 135, udp 135, rpc-over-http-server 593], \
#    autodetect [tcp 1025:, udp 1025:, rpc-over-http-server 1025:], \
#    smb_max_chain 3, smb_invalid_shares ["C$", "D$", "ADMIN$"]

/Elof


On Thu, 4 Jun 2015, Hui cao wrote:

> Can you disable dce/rpc preprocessor in your configruation and restart snort?
>
> Best,
> Hui.
>
> On 06/04/2015 06:00 PM, elof at ...969... wrote:
>> 
>> So, my other sensor, on which I disabled chroot and uid/gid change in
>> snort.conf to keep snort running as root, I now got a signal 10 and a core
>> dumped.
>> 
>> pid 3744 (snort), uid 0: exited on signal 10 (core dumped)
>> 
>> 
>> # gdb /usr/local/bin/snort snort.core
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you 
>> are
>> welcome to change it and/or distribute copies of it under certain 
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>> Core was generated by `snort'.
>> Program terminated with signal 10, Bus error.
>> Reading symbols from /usr/local/lib/libdnet.so.1...done.
>> Loaded symbols for /usr/local/lib/libdnet.so.1
>> Reading symbols from /usr/local/lib/libpcre.so.1...done.
>> Loaded symbols for /usr/local/lib/libpcre.so.1
>> Reading symbols from /lib/libm.so.5...done.
>> Loaded symbols for /lib/libm.so.5
>> Reading symbols from /lib/libcrypto.so.7...done.
>> Loaded symbols for /lib/libcrypto.so.7
>> Reading symbols from /lib/libpcap.so.8...done.
>> Loaded symbols for /lib/libpcap.so.8
>> Reading symbols from /usr/local/lib/libsfbpf.so.0...done.
>> Loaded symbols for /usr/local/lib/libsfbpf.so.0
>> Reading symbols from /lib/libz.so.6...done.
>> Loaded symbols for /lib/libz.so.6
>> Reading symbols from /usr/lib/liblzma.so.5...done.
>> Loaded symbols for /usr/lib/liblzma.so.5
>> Reading symbols from /lib/libthr.so.3...done.
>> Loaded symbols for /lib/libthr.so.3
>> Reading symbols from /lib/libc.so.7...done.
>> Loaded symbols for /lib/libc.so.7
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicengine/libsf_engine.so...done.
>> Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so
>> Reading symbols from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
>> Loaded symbols for 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so
>> Reading symbols from /libexec/ld-elf.so.1...done.
>> Loaded symbols for /libexec/ld-elf.so.1
>> #0  0x0000000804c48193 in ?? () from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> [New Thread 815b9f800 (LWP 100714/snort)]
>> [New Thread 802806400 (LWP 100635/snort)]
>> (gdb)
>> (gdb) backtrace full
>> #0  0x0000000804c48193 in ?? () from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> No symbol table info available.
>> #1  0x0000000804c47e8f in ?? () from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> No symbol table info available.
>> #2  0x000000000052c967 in s5_paf_callback (ps=0x80ffbd350, ssn=0x8c2942bf0, 
>> data=0x8c7c36f48 "32142740|PONG\r\n", len=16, flags=128) at 
>> stream_paf.c:185
>>         bit = 128
>>         paf = PAF_ABORT
>>         mask = 128
>>         update = false
>>         i = 7
>> #3  0x000000000052bf7f in s5_paf_eval (pc=0x80374d000, ps=0x80ffbd350, 
>> ssn=0x8c2942bf0, port=5600, flags=128, fuzz=150, data=0x8c7c36f48 
>> "32142740|PONG\r\n", len=16, ft=0x7fffffffe050) at stream_paf.c:243
>> No locals.
>> #4  0x000000000052bbdd in s5_paf_check (pv=0x80374d000, ps=0x80ffbd350, 
>> ssn=0x8c2942bf0, data=0x8c7c36f48 "32142740|PONG\r\n", len=16, total=32, 
>> seq=3414157013, port=5600, flags=0x7fffffffe2b0, fuzz=150)
>>     at stream_paf.c:437
>>         ft = FT_NOP
>>         idx = 16
>>         shift = 28893
>>         cont = false
>>         pc = (PAF_Config *) 0x80374d000
>> #5  0x0000000000520766 in flush_pdu_ackd (config=0x80372c000, 
>> ssn=0x80ffbd1e0, trk=0x80ffbd338, pkt=0xf18ad0, flags=0x7fffffffe2b0) at 
>> snort_stream_tcp.c:9571
>>         flush_pt = 8
>>         size = 16
>>         end = 3414157029
>>         pos = 3414157013
>>         to_srv = true
>>         srv_port = 5600
>>         total = 32
>>         seg = (StreamSegment *) 0x8c7c36ec0
>>         snort_ticks_start = 34512853664
>>         snort_ticks_end = 4237555046520209951
>> #6  0x000000000051ff15 in CheckFlushPolicyOnAck (config=0x80372c000, 
>> tcpssn=0x80ffbd1e0, talker=0x80ffbd338, listener=0x80ffbd1e0, 
>> tdb=0x7fffffffe710, p=0xf18ad0) at snort_stream_tcp.c:9729
>>         flags = 128
>>         flush_amt = 8
>>         flushed = 0
>> #7  0x0000000000518e0f in ProcessTcp (scb=0x8c2942bf0, p=0xf18ad0, 
>> tdb=0x7fffffffe710, s5TcpPolicy=0x812c06000) at snort_stream_tcp.c:9260
>>         retcode = 0
>>         eventcode = 0
>>         ignore = 0 '\0'
>>         got_ts = 0
>>         new_ssn = 0
>>         ts_action = 0
>>         tcpssn = (TcpSession *) 0x80ffbd1e0
>>         talker = (StreamTracker *) 0x80ffbd338
>>         listener = (StreamTracker *) 0x80ffbd1e0
>>         require3Way = 0
>>         snort_ticks_start = 12884901889
>>         snort_ticks_end = 0
>> #8  0x0000000000514fd4 in StreamProcessTcp (p=0xf18ad0, scb=0x8c2942bf0, 
>> s5TcpPolicy=0x812c06000, skey=0x7fffffffe7c0) at snort_stream_tcp.c:5655
>>         tdb = {seq = 3659739341, ack = 3414157029, win = 5159, end_seq = 
>> 3659739341, ts = 0}
>>         rc = 0
>>         status = 15829712
>>         snort_ticks_start = 140737488348992
>>         snort_ticks_end = 5129361
>> #9  0x00000000004dc96b in StreamProcess (p=0xf18ad0, context=0x0) at 
>> spp_stream6.c:751
>>         key = {ip_l = {4294961120, 32767, 5559925, 0}, ip_h = {1045822549, 
>> 0, 364268896, 8}, port_l = 59392, port_h = 65535, vlan_tag = 32767, 
>> protocol = 0 '\0', pad = 0 '\0', mplsLabel = 4526162,
>>   addressSpaceId = 0, addressSpaceIdPad1 = 256}
>>         scb = (SessionControlBlock *) 0x8c2942bf0
>>         snort_ticks_start = 0
>>         snort_ticks_end = 18446744065119617024
>> #10 0x000000000044f542 in DispatchPreprocessors (p=0xf18ad0, policy_id=0, 
>> policy=0x802fb2000) at detect.c:136
>>         scb = (SessionControlBlock *) 0x8c2942bf0
>>         ppn = (PreprocEvalFuncNode *) 0x815b7aee0
>>         pps_enabled_foo = 3219496
>>         alerts_processed = true
>> #11 0x000000000044ef88 in Preprocess (p=0xf18ad0) at detect.c:234
>>         retval = 0
>>         policy_id = 0
>>         policy = (SnortPolicy *) 0x802fb2000
>>         pktcnt = 0
>>         snort_ticks_start = 34413820928
>>         snort_ticks_end = 2683929608
>> #12 0x000000000043e9e8 in ProcessPacket (p=0xf18ad0, pkthdr=0x7fffffffe9a0, 
>> pkt=0x821a3f77a "", ft=0x0) at snort.c:1873
>>         verdict = DAQ_VERDICT_PASS
>> #13 0x0000000000445608 in PacketCallback (user=0x0, pkthdr=0x7fffffffe9a0, 
>> pkt=0x821a3f77a "") at snort.c:1718
>>         inject = 0
>>         verdict = DAQ_VERDICT_PASS
>>         snort_ticks_start = 34896609280
>>         snort_ticks_end = 34896609306
>> #14 0x000000000056dc6a in pcap_process_loop ()
>> No symbol table info available.
>> #15 0x00000008014d0554 in pcap_platform_finddevs () from /lib/libpcap.so.8
>> No symbol table info available.
>> #16 0x000000000056d7d8 in pcap_daq_acquire ()
>> No symbol table info available.
>> #17 0x000000000046b66b in DAQ_Acquire (max=0, callback=0x445420 
>> <PacketCallback>, user=0x0) at sfdaq.c:541
>>         err = 32767
>> #18 0x000000000043e47c in PacketLoop () at snort.c:3268
>>         error = 0
>>         pkts_to_read = 0
>> #19 0x000000000043d3d9 in SnortMain (argc=6, argv=0x7fffffffec90) at 
>> snort.c:921
>>         tmp_ptr = 0x0
>>         intf = 0x8028527c8 "mon0"
>>         daqInit = 1
>> #20 0x000000000043d1f8 in main (argc=6, argv=0x7fffffffec90) at snort.c:817
>> No locals.
>> (gdb)
>> (gdb) info registers
>> rax            0x7669643c00000000       8532461177890930688
>> rbx            0x15e0   5600
>> rcx            0x1      1
>> rdx            0x804c47e80      34439724672
>> rsi            0x8a46f3b20      37118491424
>> rdi            0x33     51
>> rbp            0x7fffffffded0   0x7fffffffded0
>> rsp            0x7fffffffdea0   0x7fffffffdea0
>> r8             0x80     128
>> r9             0x80ffbd360      34627900256
>> r10            0x7fffffffe050   140737488347216
>> r11            0x8c7c36f48      37711212360
>> r12            0x821a3f77a      34924132218
>> r13            0x821a3f760      34924132192
>> r14            0x96     150
>> r15            0x3c     60
>> rip            0x804c48193      0x804c48193 <strchr at ...3580...+40023>
>> eflags         0x10206  66054
>> cs             0x43     67
>> ss             0x3b     59
>> ds             0x0      0
>> es             0x0      0
>> fs             0x0      0
>> gs             0x0      0
>> 
>> (gdb) x/16i $pc
>> 0x804c48193 <strchr at ...3580...+40023>: mov    (%rax),%cl
>> 0x804c48195 <strchr at ...3580...+40025>: mov    %cl,-0x11(%rbp)
>> 0x804c48198 <strchr at ...3580...+40028>: movsbl -0x11(%rbp),%edx
>> 0x804c4819c <strchr at ...3580...+40032>: cmp    $0x0,%edx
>> 0x804c481a2 <strchr at ...3580...+40038>: jne    0x804c48209 <strchr at ...3582...0...+40141>
>> 0x804c481a8 <strchr at ...3580...+40044>: movzbl -0x1(%rbp),%eax
>> 0x804c481ac <strchr at ...3580...+40048>: cmp    $0x3a,%eax
>> 0x804c481b1 <strchr at ...3580...+40053>: jne    0x804c481c6 <strchr at ...3582...0...+40074>
>> 0x804c481b7 <strchr at ...3580...+40059>: mov    -0x10(%rbp),%rax
>> 0x804c481bb <strchr at ...3580...+40063>: movl   $0x2,(%rax)
>> 0x804c481c1 <strchr at ...3580...+40069>: jmpq   0x804c48204 <strchr at ...3582...0...+40136>
>> 0x804c481c6 <strchr at ...3580...+40074>: mov    $0x20000,%rsi
>> 0x804c481d0 <strchr at ...3580...+40084>: movzbl -0x1(%rbp),%edi
>> 0x804c481d4 <strchr at ...3580...+40088>: callq  0x804c485c0 <strchr at ...3582...0...+41092>
>> 0x804c481d9 <strchr at ...3580...+40093>: cmp    $0x0,%eax
>> 0x804c481de <strchr at ...3580...+40098>: jne    0x804c481ff <strchr at ...3582...0...+40131>
>> (gdb)
>> (gdb) thread apply all backtrace
>> 
>> Thread 2 (Thread 802806400 (LWP 100635/snort)):
>> #0  0x0000000804c48193 in ?? () from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> #1  0x0000000804c47e8f in ?? () from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> #2  0x000000000052c967 in s5_paf_callback (ps=0x80ffbd350, ssn=0x8c2942bf0, 
>> data=0x8c7c36f48 "32142740|PONG\r\n", len=16, flags=128) at 
>> stream_paf.c:185
>> #3  0x000000000052bf7f in s5_paf_eval (pc=0x80374d000, ps=0x80ffbd350, 
>> ssn=0x8c2942bf0, port=5600, flags=128, fuzz=150, data=0x8c7c36f48 
>> "32142740|PONG\r\n", len=16, ft=0x7fffffffe050) at stream_paf.c:243
>> #4  0x000000000052bbdd in s5_paf_check (pv=0x80374d000, ps=0x80ffbd350, 
>> ssn=0x8c2942bf0, data=0x8c7c36f48 "32142740|PONG\r\n", len=16, total=32, 
>> seq=3414157013, port=5600, flags=0x7fffffffe2b0, fuzz=150)
>>     at stream_paf.c:437
>> #5  0x0000000000520766 in flush_pdu_ackd (config=0x80372c000, 
>> ssn=0x80ffbd1e0, trk=0x80ffbd338, pkt=0xf18ad0, flags=0x7fffffffe2b0) at 
>> snort_stream_tcp.c:9571
>> #6  0x000000000051ff15 in CheckFlushPolicyOnAck (config=0x80372c000, 
>> tcpssn=0x80ffbd1e0, talker=0x80ffbd338, listener=0x80ffbd1e0, 
>> tdb=0x7fffffffe710, p=0xf18ad0) at snort_stream_tcp.c:9729
>> #7  0x0000000000518e0f in ProcessTcp (scb=0x8c2942bf0, p=0xf18ad0, 
>> tdb=0x7fffffffe710, s5TcpPolicy=0x812c06000) at snort_stream_tcp.c:9260
>> #8  0x0000000000514fd4 in StreamProcessTcp (p=0xf18ad0, scb=0x8c2942bf0, 
>> s5TcpPolicy=0x812c06000, skey=0x7fffffffe7c0) at snort_stream_tcp.c:5655
>> #9  0x00000000004dc96b in StreamProcess (p=0xf18ad0, context=0x0) at 
>> spp_stream6.c:751
>> #10 0x000000000044f542 in DispatchPreprocessors (p=0xf18ad0, policy_id=0, 
>> policy=0x802fb2000) at detect.c:136
>> #11 0x000000000044ef88 in Preprocess (p=0xf18ad0) at detect.c:234
>> #12 0x000000000043e9e8 in ProcessPacket (p=0xf18ad0, pkthdr=0x7fffffffe9a0, 
>> pkt=0x821a3f77a "", ft=0x0) at snort.c:1873
>> #13 0x0000000000445608 in PacketCallback (user=0x0, pkthdr=0x7fffffffe9a0, 
>> pkt=0x821a3f77a "") at snort.c:1718
>> #14 0x000000000056dc6a in pcap_process_loop ()
>> #15 0x00000008014d0554 in pcap_platform_finddevs () from /lib/libpcap.so.8
>> #16 0x000000000056d7d8 in pcap_daq_acquire ()
>> #17 0x000000000046b66b in DAQ_Acquire (max=0, callback=0x445420 
>> <PacketCallback>, user=0x0) at sfdaq.c:541
>> #18 0x000000000043e47c in PacketLoop () at snort.c:3268
>> #19 0x000000000043d3d9 in SnortMain (argc=6, argv=0x7fffffffec90) at 
>> snort.c:921
>> #20 0x000000000043d1f8 in main (argc=6, argv=0x7fffffffec90) at snort.c:817
>> 
>> Thread 1 (Thread 815b9f800 (LWP 100714/snort)):
>> #0  0x000000080209a8ba in nanosleep () from /lib/libc.so.7
>> #1  0x0000000801fd72ea in sleep () from /lib/libc.so.7
>> #2  0x0000000801d5ec63 in sleep () from /lib/libthr.so.3
>> #3  0x0000000000446448 in ReloadConfigThread (data=0x0) at snort.c:5695
>> #4  0x0000000801d5c4f5 in pthread_create () from /lib/libthr.so.3
>> #5  0x0000000000000000 in ?? ()
>> #0  0x0000000804c48193 in ?? () from 
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> (gdb)
>> (gdb) quit
>> 
>> /Elof
>> 
>> 
>> On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>> 
>>> Thanks!
>>> 
>>> The issue happens on smtp preprocessor, but the so is not compiled with
>>> debug enabled. Can you recompile it with ―enable-debug ?
>>> 
>>> Best,
>>> Hui.
>>> 
>>> On 6/4/15, 12:10 PM, "elof at ...969..." <elof at ...969...> wrote:
>>> 
>>>> 
>>>> So I just had a signal 6...
>>>> 
>>>> I assume I can't attach files to the mailing list, so here it is,
>>>> directly
>>>> in the mailbody. :-)
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> gdb /usr/local/bin/snort 11057
>>>> 
>>>> GNU gdb 6.1.1 [FreeBSD]
>>>> Copyright 2004 Free Software Foundation, Inc.
>>>> GDB is free software, covered by the GNU General Public License, and you
>>>> are
>>>> welcome to change it and/or distribute copies of it under certain
>>>> conditions.
>>>> Type "show copying" to see the conditions.
>>>> There is absolutely no warranty for GDB.  Type "show warranty" for
>>>> details.
>>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>> Attaching to program: /usr/local/bin/snort, process 11057
>>>> Reading symbols from /usr/local/lib/libdnet.so.1...done.
>>>> Loaded symbols for /usr/local/lib/libdnet.so.1
>>>> Reading symbols from /usr/local/lib/libpcre.so.1...done.
>>>> Loaded symbols for /usr/local/lib/libpcre.so.1
>>>> Reading symbols from /lib/libm.so.5...done.
>>>> Loaded symbols for /lib/libm.so.5
>>>> Reading symbols from /lib/libcrypto.so.6...done.
>>>> Loaded symbols for /lib/libcrypto.so.6
>>>> Reading symbols from /lib/libpcap.so.8...done.
>>>> Loaded symbols for /lib/libpcap.so.8
>>>> Reading symbols from /usr/local/lib/libsfbpf.so.0...done.
>>>> Loaded symbols for /usr/local/lib/libsfbpf.so.0
>>>> Reading symbols from /lib/libz.so.6...done.
>>>> Loaded symbols for /lib/libz.so.6
>>>> Reading symbols from /usr/lib/liblzma.so.5...done.
>>>> Loaded symbols for /usr/lib/liblzma.so.5
>>>> Reading symbols from /lib/libthr.so.3...done.
>>>> [New Thread 815a59400 (LWP 100459/snort)]
>>>> [New Thread 802407400 (LWP 100375/snort)]
>>>> Loaded symbols for /lib/libthr.so.3
>>>> Reading symbols from /lib/libc.so.7...done.
>>>> Loaded symbols for /lib/libc.so.7
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicengine/libsf_engine.so...done.
>>>> Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...don 
>>>> e.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done. 
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...do 
>>>> ne.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so
>>>> Reading symbols from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done.
>>>> Loaded symbols for
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so
>>>> Reading symbols from /libexec/ld-elf.so.1...done.
>>>> Loaded symbols for /libexec/ld-elf.so.1
>>>> [Switching to Thread 815a59400 (LWP 100459/snort)]
>>>> 0x0000000801faa40c in nanosleep () from /lib/libc.so.7
>>>> (gdb) set logging file gdb-snort.txt
>>>> (gdb) set logging on
>>>> Copying output to gdb-snort.txt.
>>>> (gdb) continue
>>>> Continuing.
>>>> 
>>>> 
>>>> 
>>>> <...it has just been a few minutes when I receive a SIGABRT>
>>>> 
>>>> 
>>>> 
>>>> Program received signal SIGABRT, Aborted.
>>>> [Switching to Thread 802407400 (LWP 100375/snort)]
>>>> 0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>>>> 
>>>> (gdb) backtrace full
>>>> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>>>> No symbol table info available.
>>>> #1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
>>>> No symbol table info available.
>>>> #2  0x0000000801fab315 in __assert () from /lib/libc.so.7
>>>> No symbol table info available.
>>>> #3  0x0000000805068395 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> No symbol table info available.
>>>> #4  0x0000000805068781 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> No symbol table info available.
>>>> #5  0x000000080506afd0 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> No symbol table info available.
>>>> #6  0x000000080506b85b in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> No symbol table info available.
>>>> #7  0x000000080506c150 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> No symbol table info available.
>>>> #8  0x000000080506cb27 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> No symbol table info available.
>>>> #9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
>>>> policy_id=0, policy=0x802faa000) at detect.c:136
>>>>     scb = (SessionControlBlock *) 0x8a1aad2f0
>>>>     ppn = (PreprocEvalFuncNode *) 0x8033ff0a0
>>>>     pps_enabled_foo = 1123336
>>>>     alerts_processed = true
>>>> #10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
>>>>     retval = 0
>>>>     policy_id = 0
>>>>     policy = (SnortPolicy *) 0x802faa000
>>>>     pktcnt = 0
>>>>     snort_ticks_start = 34413886976
>>>>     snort_ticks_end = 34413888664
>>>> #11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
>>>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>>>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
>>>>     tmp_do_detect = 1
>>>>     tmp_do_detect_content = 1
>>>>     snort_ticks_start = 37073416192
>>>>     snort_ticks_end = 37069258752
>>>>     start_seq = 846966387
>>>>     stop_seq = 1940818286
>>>>     footprint = 3644
>>>>     bytes_processed = 3644
>>>>     flushed_bytes = 3644
>>>>     pkth = {ts = {tv_sec = 100375, tv_usec = 0}, caplen = 0, pktlen = 0,
>>>> ingress_index = -1, egress_index = -1, ingress_group = -1, egress_group =
>>>> -1, flags = 0, opaque = 8, priv_ptr = 0x8a1800000, flow_id = 535241216,
>>>> address_space_id = 0}
>>>>     enc_flags = 2147483648
>>>>     snort_ticks_start = 51544732022
>>>>     snort_ticks_end = 113187
>>>> #12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
>>>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>>>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
>>>> No locals.
>>>> #13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
>>>> p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
>>>> dir=128) at snort_stream_tcp.c:4559
>>>>     bytes = 3644
>>>> #14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
>>>> st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
>>>> sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
>>>>     fm = (FlushMgr *) 0x80811cfb4
>>>> #15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
>>>> tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
>>>>     p = (Packet *) 0x8033a4900
>>>>     flushed = 1926
>>>>     tmp_pcap_hdr = {ts = {tv_sec = 1433431165, tv_usec = 321125}, caplen 
>>>> =
>>>> 94, pktlen = 94, ingress_index = 5004089, egress_index = 0, ingress_group
>>>> = 38246208, egress_group = 8, flags = 4294960320, opaque = 32767,
>>>> priv_ptr = 0x4b4c81, flow_id = 0, address_space_id = 0}
>>>> #16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
>>>> freeApplicationData=1) at snort_stream_tcp.c:5115
>>>>     tcpssn = (TcpSession *) 0x80811ce50
>>>> #17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
>>>> (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
>>>> No locals.
>>>> #18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
>>>> s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
>>>>     sscc = {old_mem_in_use = 15788887, client_ip = {family = 2, bits = 
>>>> 32,
>>>> ip = {u6_addr8 = "\nm\027L", '\0' <repeats 11 times>, u6_addr16 = {27914,
>>>> 19479, 0, 0, 0, 0, 0, 0}, u6_addr32 = {1276603658, 0, 0, 0}}}, server_ip
>>>> = {family = 2, bits = 32, ip = {u6_addr8 = "\nm\026\024", '\0' <repeats
>>>> 11 times>,
>>>>       u6_addr16 = {27914, 5142, 0, 0, 0, 0, 0, 0}, u6_addr32 =
>>>> {337014026, 0, 0, 0}}}, client_port = 39946, server_port = 6400,
>>>> lw_session_state = 200, lw_session_flags = 4284679, app_proto_id = 0}
>>>>     tdb = {seq = 1940818286, ack = 2349672268, win = 64032, end_seq =
>>>> 1940818325, ts = 0}
>>>>     rc = 0
>>>>     status = 4512282
>>>>     snort_ticks_start = 34397587520
>>>>     snort_ticks_end = 140737488348864
>>>> #19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
>>>> spp_stream6.c:751
>>>>     key = {ip_l = {0, 0, 4216431, 0}, ip_h = {0, 2, 362856224, 8}, port_l 
>>>> =
>>>> 59136, port_h = 65535, vlan_tag = 32767, protocol = 0 '\0', pad = 0 '\0',
>>>> mplsLabel = 5328944, addressSpaceId = 0, addressSpaceIdPad1 = 0}
>>>>     scb = (SessionControlBlock *) 0x8a1aad2f0
>>>>     snort_ticks_start = 140737488348960
>>>>     snort_ticks_end = 34722594592
>>>> #20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
>>>> policy=0x802faa000) at detect.c:136
>>>>     scb = (SessionControlBlock *) 0x8a1aad2f0
>>>>     ppn = (PreprocEvalFuncNode *) 0x815b61340
>>>>     pps_enabled_foo = 1123336
>>>>     alerts_processed = true
>>>> #21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
>>>>     retval = 0
>>>>     policy_id = 0
>>>>     policy = (SnortPolicy *) 0x802faa000
>>>>     pktcnt = 0
>>>>     snort_ticks_start = 0
>>>>     snort_ticks_end = 6059431713369489410
>>>> #22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
>>>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at 
>>>> snort.c:1873
>>>>     verdict = DAQ_VERDICT_PASS
>>>>     __func__ = "ProcessPacket"
>>>> #23 0x0000000000434ccd in PacketCallback (user=0x0,
>>>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
>>>>     inject = 0
>>>>     verdict = DAQ_VERDICT_PASS
>>>>     snort_ticks_start = 34894979584
>>>>     snort_ticks_end = 34367935488
>>>> #24 0x000000000052fe34 in pcap_process_loop ()
>>>> No symbol table info available.
>>>> #25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
>>>> No symbol table info available.
>>>> #26 0x000000000053025f in pcap_daq_acquire ()
>>>> No symbol table info available.
>>>> #27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
>>>> <PacketCallback>, user=0x0) at sfdaq.c:541
>>>>     err = 0
>>>> #28 0x0000000000437616 in PacketLoop () at snort.c:3268
>>>>     error = 0
>>>>     pkts_to_read = 0
>>>> #29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
>>>> snort.c:921
>>>>     tmp_ptr = 0x0
>>>>     intf = 0x8024c4540 "mon0"
>>>>     daqInit = 1
>>>> #30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
>>>> snort.c:817
>>>> No locals.
>>>> rax            0x0    0
>>>> rbx            0x7fffffffddec    140737488346604
>>>> rcx            0x801fc8fbc    34393067452
>>>> rdx            0x0    0
>>>> rsi            0x6    6
>>>> rdi            0x18817    100375
>>>> rbp            0x7fffffffde60    0x7fffffffde60
>>>> rsp            0x7fffffffddd8    0x7fffffffddd8
>>>> r8             0x0    0
>>>> r9             0xfffffe0032ea54a8    -2198169037656
>>>> r10            0x59    89
>>>> r11            0x202    514
>>>> r12            0x80811ce50    34495123024
>>>> r13            0x8033a51b8    34413892024
>>>> r14            0x82251deaa    34935529130
>>>> r15            0x1ba24    113188
>>>> rip            0x801f2364c    0x801f2364c <thr_kill+12>
>>>> eflags         0x206    518
>>>> cs             0x43    67
>>>> ss             0x3b    59
>>>> ds             0x0    0
>>>> es             0x0    0
>>>> fs             0x0    0
>>>> gs             0x0    0
>>>> 0x801f2364c <thr_kill+12>:    jb     0x801f2364f <thr_kill+15>
>>>> 0x801f2364e <thr_kill+14>:    retq
>>>> 0x801f2364f <thr_kill+15>:    mov 0x2d6bea(%rip),%rcx        #
>>>> 0x8021fa240 <__nsdefaultsrc+5696>
>>>> 0x801f23656 <thr_kill+22>:    jmpq   *%rcx
>>>> 0x801f23658 <thr_kill+24>:    nop
>>>> 0x801f23659 <thr_kill+25>:    nop
>>>> 0x801f2365a <thr_kill+26>:    nop
>>>> 0x801f2365b <thr_kill+27>:    nop
>>>> 0x801f2365c <thr_kill+28>:    nop
>>>> 0x801f2365d <thr_kill+29>:    nop
>>>> 0x801f2365e <thr_kill+30>:    nop
>>>> 0x801f2365f <thr_kill+31>:    nop
>>>> 0x801f23660 <thr_self>:    mov    $0x1b0,%rax
>>>> 0x801f23667 <thr_self+7>:    mov    %rcx,%r10
>>>> 0x801f2366a <thr_self+10>:    syscall
>>>> 0x801f2366c <thr_self+12>:    jb     0x801f2366f <thr_self+15>
>>>> 
>>>> Thread 2 (Thread 802407400 (LWP 100375/snort)):
>>>> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>>>> #1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
>>>> #2  0x0000000801fab315 in __assert () from /lib/libc.so.7
>>>> #3  0x0000000805068395 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> #4  0x0000000805068781 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> #5  0x000000080506afd0 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> #6  0x000000080506b85b in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> #7  0x000000080506c150 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> #8  0x000000080506cb27 in ?? () from
>>>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>>>> #9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
>>>> policy_id=0, policy=0x802faa000) at detect.c:136
>>>> #10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
>>>> #11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
>>>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>>>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
>>>> #12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
>>>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>>>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
>>>> #13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
>>>> p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
>>>> dir=128) at snort_stream_tcp.c:4559
>>>> #14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
>>>> st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
>>>> sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
>>>> #15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
>>>> tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
>>>> #16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
>>>> freeApplicationData=1) at snort_stream_tcp.c:5115
>>>> #17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
>>>> (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
>>>> #18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
>>>> s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
>>>> #19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
>>>> spp_stream6.c:751
>>>> #20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
>>>> policy=0x802faa000) at detect.c:136
>>>> #21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
>>>> #22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
>>>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at 
>>>> snort.c:1873
>>>> #23 0x0000000000434ccd in PacketCallback (user=0x0,
>>>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
>>>> #24 0x000000000052fe34 in pcap_process_loop ()
>>>> #25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
>>>> #26 0x000000000053025f in pcap_daq_acquire ()
>>>> #27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
>>>> <PacketCallback>, user=0x0) at sfdaq.c:541
>>>> #28 0x0000000000437616 in PacketLoop () at snort.c:3268
>>>> #29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
>>>> snort.c:921
>>>> #30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
>>>> snort.c:817
>>>> 
>>>> Thread 1 (Thread 815a59400 (LWP 100459/snort)):
>>>> #0  0x0000000801faa40c in nanosleep () from /lib/libc.so.7
>>>> #1  0x0000000801f15a58 in sleep () from /lib/libc.so.7
>>>> #2  0x0000000801ca8078 in sleep () from /lib/libthr.so.3
>>>> #3  0x000000000043b215 in ReloadConfigThread (data=0x0) at snort.c:5695
>>>> #4  0x0000000801ca5dc4 in pthread_getprio () from /lib/libthr.so.3
>>>> #5  0x0000000000000000 in ?? ()
>>>> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>>>> The program is running.  Quit anyway (and detach it)? (y or n) Detaching
>>>> from program: /usr/local/bin/snort, process 11057
>>>> 
>>>> 
>>>> 
>>>> 
>>>> As gdb detached from snort, I got the signal 6 in my syslog:
>>>> 2015-06-04 17:51:53 +02:00 foobar kernel: pid 11057 (snort), uid 100:
>>>> exited on signal 6
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> So, this time we got a signal 6 but during this sensor's 14 hour uptime
>>>> we've seen:
>>>> pid 1199 (snort), uid 100: exited on signal 10
>>>> pid 4503 (snort), uid 100: exited on signal 10
>>>> pid 5908 (snort), uid 100: exited on signal 10
>>>> pid 11057 (snort), uid 100: exited on signal 6
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> I hope this gdb was helpful.
>>>> Let me know if it should be run again.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> This was all performed on a sensor running:
>>>>
>>>>    ,,_     -*> Snort! <*-
>>>>   o"  )~   Version 2.9.7.3 (Build 217)
>>>>    ''''    By Martin Roesch & The Snort Team:
>>>> http://www.snort.org/contact#team
>>>>            Copyright (C) 2014-2015 Cisco and/or its affiliates. All
>>>> rights reserved.
>>>>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>            Using libpcap version 1.4.0
>>>>            Using PCRE version: 8.37 2015-04-28
>>>>            Using ZLIB version: 1.2.8
>>>> 
>>>> 
>>>> daq-2.0.5
>>>> 
>>>> FreeBSD 9.3-RELEASE-p13
>>>> 
>>>> 
>>>> /Elof
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>>>> 
>>>>> That¹s cool. All looks good to me. No need to do more things...
>>>>> 
>>>>> Best,
>>>>> Hui
>>>>> 
>>>>> On 6/4/15, 11:35 AM, "elof at ...969..." <elof at ...969...> wrote:
>>>>> 
>>>>>> 
>>>>>> Hi Hui.
>>>>>> 
>>>>>> That much I know. It is the debugging steps I'm curious about.
>>>>>> 
>>>>>> (I think you forgot one important first command: continue )
>>>>>> 
>>>>>> 
>>>>>> Is this a good start:
>>>>>> 
>>>>>> gdb /path/to/snort 1222
>>>>>> (gdb) set logging file gdb-snort.txt
>>>>>> (gdb) set logging on
>>>>>> (gdb) continue
>>>>>> 
>>>>>> <wait for it to crash>
>>>>>> 
>>>>>> (gdb) backtrace full
>>>>>> (gdb) info registers
>>>>>> (gdb) x/16i $pc
>>>>>> (gdb) thread apply all backtrace
>>>>>> (gdb) quit
>>>>>> 
>>>>>> Email the report.
>>>>>> 
>>>>>> 
>>>>>> Should I prepare more stuff before the 'continue'?
>>>>>> Like "handle SIG33 pass nostop noprint" or something?
>>>>>> 
>>>>>> /Elof
>>>>>> 
>>>>>> 
>>>>>> On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>>>>>> 
>>>>>>> Try
>>>>>>> 
>>>>>>> Assume snort pid is 1222
>>>>>>> 
>>>>>>> gdb /path/to/snort 1222
>>>>>>> 
>>>>>>> Best,
>>>>>>> Hui.
>>>>>>> On 6/4/15, 10:37 AM, "elof at ...969..." <elof at ...969...> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> An update:
>>>>>>>> 
>>>>>>>> On a sensor where snort crashed with signal 6 three times, I
>>>>>>>> downgraded
>>>>>>>> daq to 2.0.4_1 and rebooted the machine to rule out if the problem
>>>>>>>> seem
>>>>>>>> to
>>>>>>>> be in 'snort' or 'daq'.
>>>>>>>> 
>>>>>>>> With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> This make me believe that there's something wrong in snort 2.9.7.3
>>>>>>>> and
>>>>>>>> not in daq 2.0.5.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On this sensor I have now done the opposite, upgraded daq to 2.0.5
>>>>>>>> and
>>>>>>>> downgraded snort to 2.9.7.2 to see if I get any more signal 6.
>>>>>>>> 
>>>>>>>> On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
>>>>>>>> 2.0.5
>>>>>>>> without chroot and uid/gid change, i.e. running as root, in order to
>>>>>>>> create a core file, if the problem happen again.
>>>>>>>> (if it doesn't happen on this sensor, I guess the problem lies
>>>>>>>> somewhere
>>>>>>>> in the chrooting code in snort. I know it has been updated between
>>>>>>>> 2.9.7.2
>>>>>>>> and 2.9.7.3)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Russ C also wrote:
>>>>>>>>> Elof - since this is happening frequently, you could try attaching
>>>>>>>>> the
>>>>>>>>> debugger to one of your Snort processes and wait wait for segfault.
>>>>>>>> 
>>>>>>>> I know too little about debugging. :-/ Can you give me instructions
>>>>>>>> or
>>>>>>>> point me to a guide that describes the steps I should take?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> /Elof
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Five different sensors have now had bus errors (signal 10),
>>>>>>>>> segmentation
>>>>>>>>> faults (signal 11) and even signal 6 (SIGABRT).
>>>>>>>>> 
>>>>>>>>> My snort config uses both chroot and dropping user privileges, so
>>>>>>>>> even
>>>>>>>>> if
>>>>>>>>> I start out as root with ulimit unlimited, this doesn't seem to be
>>>>>>>>> in
>>>>>>>>> effect
>>>>>>>>> after the chroot/uid-change.
>>>>>>>>> 
>>>>>>>>> So currently I have no core-file to debug. :-/
>>>>>>>>> 
>>>>>>>>> Anyone know how to set the ulimits for a chrooted and
>>>>>>>>> uid/gid-changed
>>>>>>>>> process in FreeBSD?
>>>>>>>>> 
>>>>>>>>> /Elof
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Hi Hui!
>>>>>>>>>> 
>>>>>>>>>> Yes, the dynamic engine/preproc files are updated as well.
>>>>>>>>>> 
>>>>>>>>>> Last night the problem reocurred, so this seem to be reproduceable.
>>>>>>>>>> Good.
>>>>>>>>>> Then there's a good chance this problem can be sorted out.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> A few minutes ago a signal 10 happened on another sensor (running
>>>>>>>>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in
>>>>>>>>>> Snort
>>>>>>>>>> 2.9.7.3 and not in the hardware nor in FreeBSD.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I will compile a debug-snort and try to generate core files.
>>>>>>>>>> I'll let you know the outcome next week.
>>>>>>>>>> 
>>>>>>>>>> /Elof
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Wed, 3 Jun 2015, Hui cao wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Elof,
>>>>>>>>>>> 
>>>>>>>>>>> Are snort and snort dynamic preprocessors are in sync?
>>>>>>>>>>> 
>>>>>>>>>>> If so, can you help us get a backtrace from the crush? You need
>>>>>>>>>>> 1)  build snort with ./configure --enable-debug
>>>>>>>>>>> 2)  allowing core dump (ulimit -c unlimited)
>>>>>>>>>>> 3) run the snort
>>>>>>>>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb
>>>>>>>>>>> command
>>>>>>>>>>> line
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Hui.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 06/03/2015 05:51 AM, elof at ...969... wrote:
>>>>>>>>>>>> Hi all!
>>>>>>>>>>>> 
>>>>>>>>>>>> This is just a report to inform that after I updated snort and
>>>>>>>>>>>> DAQ
>>>>>>>>>>>> to the
>>>>>>>>>>>> latest versions, one of my sensors started throwing signal 10
>>>>>>>>>>>> (bus
>>>>>>>>>>>> error)
>>>>>>>>>>>> and signal 11 (segmentation fault).
>>>>>>>>>>>> 
>>>>>>>>>>>> # uptime
>>>>>>>>>>>> 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
>>>>>>>>>>>> # dmesg | grep snort
>>>>>>>>>>>> pid 1183 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 16920 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 17502 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 18862 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 20223 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 20927 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 1193 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 2447 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 3811 (snort), uid 100: exited on signal 10
>>>>>>>>>>>> pid 7881 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 9252 (snort), uid 100: exited on signal 10
>>>>>>>>>>>> pid 25593 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 26627 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 56658 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 57237 (snort), uid 100: exited on signal 10
>>>>>>>>>>>> pid 58595 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 68639 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 70008 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> pid 71361 (snort), uid 100: exited on signal 10
>>>>>>>>>>>> pid 72725 (snort), uid 100: exited on signal 11
>>>>>>>>>>>> 
>>>>>>>>>>>> 20 crashes in a day...
>>>>>>>>>>>> A reboot didn't help.
>>>>>>>>>>>> 
>>>>>>>>>>>> This sensor has never behaved like this during its lifetime (1
>>>>>>>>>>>> year).
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> FreeBSD 9.3 amd64
>>>>>>>>>>>>
>>>>>>>>>>>>      ,,_     -*> Snort! <*-
>>>>>>>>>>>>     o"  )~   Version 2.9.7.3 (Build 217)
>>>>>>>>>>>>      ''''    By Martin Roesch & The Snort Team:
>>>>>>>>>>>> http://www.snort.org/contact#team
>>>>>>>>>>>>              Copyright (C) 2014-2015 Cisco and/or its affiliates.
>>>>>>>>>>>> All rights
>>>>>>>>>>>> reserved.
>>>>>>>>>>>>              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>>>>>>>>>              Using libpcap version 1.4.0
>>>>>>>>>>>>              Using PCRE version: 8.37 2015-04-28
>>>>>>>>>>>>              Using ZLIB version: 1.2.8
>>>>>>>>>>>> 
>>>>>>>>>>>> daq-2.0.5
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Bus errors are quite unusual in general, so I'll keep looking at
>>>>>>>>>>>> this,
>>>>>>>>>>>> trying to see if it is e.g. paging errors.
>>>>>>>>>>>> It doesn't look like it though:
>>>>>>>>>>>> # swapinfo
>>>>>>>>>>>> Device          1K-blocks     Used    Avail Capacity
>>>>>>>>>>>> /dev/mirror/swap   4194300        0 4194300     0%
>>>>>>>>>>>> 
>>>>>>>>>>>> The machine doesn't seem to be overheated either:
>>>>>>>>>>>> System Temp:    30 degrees C
>>>>>>>>>>>> Peripheral Temp: 40 degrees C
>>>>>>>>>>>> CPU Temp: Low
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> If you need me to do something special to debug this further, let
>>>>>>>>>>>> me
>>>>>>>>>>>> know.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> PS. It is only one sensor, out of 20, that behaves like this. So
>>>>>>>>>>>> perhaps
>>>>>>>>>>>> it is something in the mirrored traffic that make DAQ or snort
>>>>>>>>>>>> point
>>>>>>>>>>>> at
>>>>>>>>>>>> illegal memory addresses and crash.
>>>>>>>>>>>> Or this particular machine is having hardware issues. However, it
>>>>>>>>>>>> is
>>>>>>>>>>>> strange that those hw-issues should suddenly start right after I
>>>>>>>>>>>> updated
>>>>>>>>>>>> the software on the machine...
>>>>>>>>>>>> 
>>>>>>>>>>>> When I write this, the current snort process has been alive for 5
>>>>>>>>>>>> hours.
>>>>>>>>>>>> It's going to be interesting to see if the traffic tonight will
>>>>>>>>>>>> cause it
>>>>>>>>>>>> to crash many times again.
>>>>>>>>>>>> 
>>>>>>>>>>>> /Elof
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>
>>>>>>>>>>>> 
>>>>>>>>>>>> ------------------------------------------------------------------ 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> --------
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Snort-devel mailing list
>>>>>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>>>>>> Archive:
>>>>>>>>>>>> 
>>>>>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-deve 
>>>>>>>>>>>> l
>>>>>>>>>>>> 
>>>>>>>>>>>> Please visit http://blog.snort.org for the latest news about
>>>>>>>>>>>> Snort!
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>
>>>>>>>>>>> 
>>>>>>>>>>> ------------------------------------------------------------------- 
>>>>>>>>>>> -- 
>>>>>>>>>>> -- 
>>>>>>>>>>> -------
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Snort-devel mailing list
>>>>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>>>>> Archive:
>>>>>>>>>>> 
>>>>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel 
>>>>>>>>>>> 
>>>>>>>>>>> Please visit http://blog.snort.org for the latest news about
>>>>>>>>>>> Snort!
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>> -------------------------------------------------------------------- 
>>>>>>>>>> -- 
>>>>>>>>>> -- 
>>>>>>>>>> ------
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Snort-devel mailing list
>>>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>>>> Archive:
>>>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel 
>>>>>>>>>> 
>>>>>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>> --------------------------------------------------------------------- 
>>>>>>>>> -- 
>>>>>>>>> -- 
>>>>>>>>> -----
>>>>>>>>> _______________________________________________
>>>>>>>>> Snort-devel mailing list
>>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>>> Archive:
>>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel 
>>>>>>>>> 
>>>>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>>>
>>>>> 
>>>>> ------------------------------------------------------------------------- 
>>>>> -----
>>>>> _______________________________________________
>>>>> Snort-devel mailing list
>>>>> Snort-devel at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>> Archive:
>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>> 
>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>> 
>>> 
>


More information about the Snort-devel mailing list