[Snort-devel] Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

elof at ...969... elof at ...969...
Thu Jun 4 18:00:08 EDT 2015


So, my other sensor, on which I disabled chroot and uid/gid change in
snort.conf to keep snort running as root, I now got a signal 10 and a core
dumped.

pid 3744 (snort), uid 0: exited on signal 10 (core dumped)


# gdb /usr/local/bin/snort snort.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you 
are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for 
details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `snort'.
Program terminated with signal 10, Bus error.
Reading symbols from /usr/local/lib/libdnet.so.1...done.
Loaded symbols for /usr/local/lib/libdnet.so.1
Reading symbols from /usr/local/lib/libpcre.so.1...done.
Loaded symbols for /usr/local/lib/libpcre.so.1
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libcrypto.so.7...done.
Loaded symbols for /lib/libcrypto.so.7
Reading symbols from /lib/libpcap.so.8...done.
Loaded symbols for /lib/libpcap.so.8
Reading symbols from /usr/local/lib/libsfbpf.so.0...done.
Loaded symbols for /usr/local/lib/libsfbpf.so.0
Reading symbols from /lib/libz.so.6...done.
Loaded symbols for /lib/libz.so.6
Reading symbols from /usr/lib/liblzma.so.5...done.
Loaded symbols for /usr/lib/liblzma.so.5
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from 
/usr/local/lib/snort_dynamicengine/libsf_engine.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so
Reading symbols from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
Loaded symbols for 
/usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000804c48193 in ?? () from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
[New Thread 815b9f800 (LWP 100714/snort)]
[New Thread 802806400 (LWP 100635/snort)]
(gdb)
(gdb) backtrace full
#0  0x0000000804c48193 in ?? () from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
No symbol table info available.
#1  0x0000000804c47e8f in ?? () from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
No symbol table info available.
#2  0x000000000052c967 in s5_paf_callback (ps=0x80ffbd350, 
ssn=0x8c2942bf0, data=0x8c7c36f48 "32142740|PONG\r\n", len=16, flags=128) 
at stream_paf.c:185
         bit = 128
         paf = PAF_ABORT
         mask = 128
         update = false
         i = 7
#3  0x000000000052bf7f in s5_paf_eval (pc=0x80374d000, ps=0x80ffbd350, 
ssn=0x8c2942bf0, port=5600, flags=128, fuzz=150, data=0x8c7c36f48 
"32142740|PONG\r\n", len=16, ft=0x7fffffffe050) at stream_paf.c:243
No locals.
#4  0x000000000052bbdd in s5_paf_check (pv=0x80374d000, ps=0x80ffbd350, 
ssn=0x8c2942bf0, data=0x8c7c36f48 "32142740|PONG\r\n", len=16, total=32, 
seq=3414157013, port=5600, flags=0x7fffffffe2b0, fuzz=150)
     at stream_paf.c:437
         ft = FT_NOP
         idx = 16
         shift = 28893
         cont = false
         pc = (PAF_Config *) 0x80374d000
#5  0x0000000000520766 in flush_pdu_ackd (config=0x80372c000, 
ssn=0x80ffbd1e0, trk=0x80ffbd338, pkt=0xf18ad0, flags=0x7fffffffe2b0) at 
snort_stream_tcp.c:9571
         flush_pt = 8
         size = 16
         end = 3414157029
         pos = 3414157013
         to_srv = true
         srv_port = 5600
         total = 32
         seg = (StreamSegment *) 0x8c7c36ec0
         snort_ticks_start = 34512853664
         snort_ticks_end = 4237555046520209951
#6  0x000000000051ff15 in CheckFlushPolicyOnAck (config=0x80372c000, 
tcpssn=0x80ffbd1e0, talker=0x80ffbd338, listener=0x80ffbd1e0, 
tdb=0x7fffffffe710, p=0xf18ad0) at snort_stream_tcp.c:9729
         flags = 128
         flush_amt = 8
         flushed = 0
#7  0x0000000000518e0f in ProcessTcp (scb=0x8c2942bf0, p=0xf18ad0, 
tdb=0x7fffffffe710, s5TcpPolicy=0x812c06000) at snort_stream_tcp.c:9260
         retcode = 0
         eventcode = 0
         ignore = 0 '\0'
         got_ts = 0
         new_ssn = 0
         ts_action = 0
         tcpssn = (TcpSession *) 0x80ffbd1e0
         talker = (StreamTracker *) 0x80ffbd338
         listener = (StreamTracker *) 0x80ffbd1e0
         require3Way = 0
         snort_ticks_start = 12884901889
         snort_ticks_end = 0
#8  0x0000000000514fd4 in StreamProcessTcp (p=0xf18ad0, scb=0x8c2942bf0, 
s5TcpPolicy=0x812c06000, skey=0x7fffffffe7c0) at snort_stream_tcp.c:5655
         tdb = {seq = 3659739341, ack = 3414157029, win = 5159, end_seq = 
3659739341, ts = 0}
         rc = 0
         status = 15829712
         snort_ticks_start = 140737488348992
         snort_ticks_end = 5129361
#9  0x00000000004dc96b in StreamProcess (p=0xf18ad0, context=0x0) at 
spp_stream6.c:751
         key = {ip_l = {4294961120, 32767, 5559925, 0}, ip_h = {1045822549, 
0, 364268896, 8}, port_l = 59392, port_h = 65535, vlan_tag = 32767, 
protocol = 0 '\0', pad = 0 '\0', mplsLabel = 4526162,
   addressSpaceId = 0, addressSpaceIdPad1 = 256}
         scb = (SessionControlBlock *) 0x8c2942bf0
         snort_ticks_start = 0
         snort_ticks_end = 18446744065119617024
#10 0x000000000044f542 in DispatchPreprocessors (p=0xf18ad0, policy_id=0, 
policy=0x802fb2000) at detect.c:136
         scb = (SessionControlBlock *) 0x8c2942bf0
         ppn = (PreprocEvalFuncNode *) 0x815b7aee0
         pps_enabled_foo = 3219496
         alerts_processed = true
#11 0x000000000044ef88 in Preprocess (p=0xf18ad0) at detect.c:234
         retval = 0
         policy_id = 0
         policy = (SnortPolicy *) 0x802fb2000
         pktcnt = 0
         snort_ticks_start = 34413820928
         snort_ticks_end = 2683929608
#12 0x000000000043e9e8 in ProcessPacket (p=0xf18ad0, 
pkthdr=0x7fffffffe9a0, pkt=0x821a3f77a "", ft=0x0) at snort.c:1873
         verdict = DAQ_VERDICT_PASS
#13 0x0000000000445608 in PacketCallback (user=0x0, pkthdr=0x7fffffffe9a0, 
pkt=0x821a3f77a "") at snort.c:1718
         inject = 0
         verdict = DAQ_VERDICT_PASS
         snort_ticks_start = 34896609280
         snort_ticks_end = 34896609306
#14 0x000000000056dc6a in pcap_process_loop ()
No symbol table info available.
#15 0x00000008014d0554 in pcap_platform_finddevs () from /lib/libpcap.so.8
No symbol table info available.
#16 0x000000000056d7d8 in pcap_daq_acquire ()
No symbol table info available.
#17 0x000000000046b66b in DAQ_Acquire (max=0, callback=0x445420 
<PacketCallback>, user=0x0) at sfdaq.c:541
         err = 32767
#18 0x000000000043e47c in PacketLoop () at snort.c:3268
         error = 0
         pkts_to_read = 0
#19 0x000000000043d3d9 in SnortMain (argc=6, argv=0x7fffffffec90) at 
snort.c:921
         tmp_ptr = 0x0
         intf = 0x8028527c8 "mon0"
         daqInit = 1
#20 0x000000000043d1f8 in main (argc=6, argv=0x7fffffffec90) at 
snort.c:817
No locals.
(gdb)
(gdb) info registers
rax            0x7669643c00000000       8532461177890930688
rbx            0x15e0   5600
rcx            0x1      1
rdx            0x804c47e80      34439724672
rsi            0x8a46f3b20      37118491424
rdi            0x33     51
rbp            0x7fffffffded0   0x7fffffffded0
rsp            0x7fffffffdea0   0x7fffffffdea0
r8             0x80     128
r9             0x80ffbd360      34627900256
r10            0x7fffffffe050   140737488347216
r11            0x8c7c36f48      37711212360
r12            0x821a3f77a      34924132218
r13            0x821a3f760      34924132192
r14            0x96     150
r15            0x3c     60
rip            0x804c48193      0x804c48193 <strchr at ...3580...+40023>
eflags         0x10206  66054
cs             0x43     67
ss             0x3b     59
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

(gdb) x/16i $pc
0x804c48193 <strchr at ...3580...+40023>: mov    (%rax),%cl
0x804c48195 <strchr at ...3580...+40025>: mov    %cl,-0x11(%rbp)
0x804c48198 <strchr at ...3580...+40028>: movsbl -0x11(%rbp),%edx
0x804c4819c <strchr at ...3580...+40032>: cmp    $0x0,%edx
0x804c481a2 <strchr at ...3580...+40038>: jne    0x804c48209 <strchr at ...3581....+40141>
0x804c481a8 <strchr at ...3580...+40044>: movzbl -0x1(%rbp),%eax
0x804c481ac <strchr at ...3580...+40048>: cmp    $0x3a,%eax
0x804c481b1 <strchr at ...3580...+40053>: jne    0x804c481c6 <strchr at ...3581....+40074>
0x804c481b7 <strchr at ...3580...+40059>: mov    -0x10(%rbp),%rax
0x804c481bb <strchr at ...3580...+40063>: movl   $0x2,(%rax)
0x804c481c1 <strchr at ...3580...+40069>: jmpq   0x804c48204 <strchr at ...3581....+40136>
0x804c481c6 <strchr at ...3580...+40074>: mov    $0x20000,%rsi
0x804c481d0 <strchr at ...3580...+40084>: movzbl -0x1(%rbp),%edi
0x804c481d4 <strchr at ...3580...+40088>: callq  0x804c485c0 <strchr at ...3581....+41092>
0x804c481d9 <strchr at ...3580...+40093>: cmp    $0x0,%eax
0x804c481de <strchr at ...3580...+40098>: jne    0x804c481ff <strchr at ...3581....+40131>
(gdb)
(gdb) thread apply all backtrace

Thread 2 (Thread 802806400 (LWP 100635/snort)):
#0  0x0000000804c48193 in ?? () from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
#1  0x0000000804c47e8f in ?? () from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
#2  0x000000000052c967 in s5_paf_callback (ps=0x80ffbd350, 
ssn=0x8c2942bf0, data=0x8c7c36f48 "32142740|PONG\r\n", len=16, flags=128) 
at stream_paf.c:185
#3  0x000000000052bf7f in s5_paf_eval (pc=0x80374d000, ps=0x80ffbd350, 
ssn=0x8c2942bf0, port=5600, flags=128, fuzz=150, data=0x8c7c36f48 
"32142740|PONG\r\n", len=16, ft=0x7fffffffe050) at stream_paf.c:243
#4  0x000000000052bbdd in s5_paf_check (pv=0x80374d000, ps=0x80ffbd350, 
ssn=0x8c2942bf0, data=0x8c7c36f48 "32142740|PONG\r\n", len=16, total=32, 
seq=3414157013, port=5600, flags=0x7fffffffe2b0, fuzz=150)
     at stream_paf.c:437
#5  0x0000000000520766 in flush_pdu_ackd (config=0x80372c000, 
ssn=0x80ffbd1e0, trk=0x80ffbd338, pkt=0xf18ad0, flags=0x7fffffffe2b0) at 
snort_stream_tcp.c:9571
#6  0x000000000051ff15 in CheckFlushPolicyOnAck (config=0x80372c000, 
tcpssn=0x80ffbd1e0, talker=0x80ffbd338, listener=0x80ffbd1e0, 
tdb=0x7fffffffe710, p=0xf18ad0) at snort_stream_tcp.c:9729
#7  0x0000000000518e0f in ProcessTcp (scb=0x8c2942bf0, p=0xf18ad0, 
tdb=0x7fffffffe710, s5TcpPolicy=0x812c06000) at snort_stream_tcp.c:9260
#8  0x0000000000514fd4 in StreamProcessTcp (p=0xf18ad0, scb=0x8c2942bf0, 
s5TcpPolicy=0x812c06000, skey=0x7fffffffe7c0) at snort_stream_tcp.c:5655
#9  0x00000000004dc96b in StreamProcess (p=0xf18ad0, context=0x0) at 
spp_stream6.c:751
#10 0x000000000044f542 in DispatchPreprocessors (p=0xf18ad0, policy_id=0, 
policy=0x802fb2000) at detect.c:136
#11 0x000000000044ef88 in Preprocess (p=0xf18ad0) at detect.c:234
#12 0x000000000043e9e8 in ProcessPacket (p=0xf18ad0, 
pkthdr=0x7fffffffe9a0, pkt=0x821a3f77a "", ft=0x0) at snort.c:1873
#13 0x0000000000445608 in PacketCallback (user=0x0, pkthdr=0x7fffffffe9a0, 
pkt=0x821a3f77a "") at snort.c:1718
#14 0x000000000056dc6a in pcap_process_loop ()
#15 0x00000008014d0554 in pcap_platform_finddevs () from /lib/libpcap.so.8
#16 0x000000000056d7d8 in pcap_daq_acquire ()
#17 0x000000000046b66b in DAQ_Acquire (max=0, callback=0x445420 
<PacketCallback>, user=0x0) at sfdaq.c:541
#18 0x000000000043e47c in PacketLoop () at snort.c:3268
#19 0x000000000043d3d9 in SnortMain (argc=6, argv=0x7fffffffec90) at 
snort.c:921
#20 0x000000000043d1f8 in main (argc=6, argv=0x7fffffffec90) at 
snort.c:817

Thread 1 (Thread 815b9f800 (LWP 100714/snort)):
#0  0x000000080209a8ba in nanosleep () from /lib/libc.so.7
#1  0x0000000801fd72ea in sleep () from /lib/libc.so.7
#2  0x0000000801d5ec63 in sleep () from /lib/libthr.so.3
#3  0x0000000000446448 in ReloadConfigThread (data=0x0) at snort.c:5695
#4  0x0000000801d5c4f5 in pthread_create () from /lib/libthr.so.3
#5  0x0000000000000000 in ?? ()
#0  0x0000000804c48193 in ?? () from 
/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
(gdb)
(gdb) quit

/Elof


On Thu, 4 Jun 2015, Hui Cao (huica) wrote:

> Thanks!
>
> The issue happens on smtp preprocessor, but the so is not compiled with
> debug enabled. Can you recompile it with ―enable-debug ?
>
> Best,
> Hui.
>
> On 6/4/15, 12:10 PM, "elof at ...969..." <elof at ...969...> wrote:
>
>>
>> So I just had a signal 6...
>>
>> I assume I can't attach files to the mailing list, so here it is,
>> directly
>> in the mailbody. :-)
>>
>>
>>
>>
>>
>> gdb /usr/local/bin/snort 11057
>>
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you
>> are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for
>> details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>> Attaching to program: /usr/local/bin/snort, process 11057
>> Reading symbols from /usr/local/lib/libdnet.so.1...done.
>> Loaded symbols for /usr/local/lib/libdnet.so.1
>> Reading symbols from /usr/local/lib/libpcre.so.1...done.
>> Loaded symbols for /usr/local/lib/libpcre.so.1
>> Reading symbols from /lib/libm.so.5...done.
>> Loaded symbols for /lib/libm.so.5
>> Reading symbols from /lib/libcrypto.so.6...done.
>> Loaded symbols for /lib/libcrypto.so.6
>> Reading symbols from /lib/libpcap.so.8...done.
>> Loaded symbols for /lib/libpcap.so.8
>> Reading symbols from /usr/local/lib/libsfbpf.so.0...done.
>> Loaded symbols for /usr/local/lib/libsfbpf.so.0
>> Reading symbols from /lib/libz.so.6...done.
>> Loaded symbols for /lib/libz.so.6
>> Reading symbols from /usr/lib/liblzma.so.5...done.
>> Loaded symbols for /usr/lib/liblzma.so.5
>> Reading symbols from /lib/libthr.so.3...done.
>> [New Thread 815a59400 (LWP 100459/snort)]
>> [New Thread 802407400 (LWP 100375/snort)]
>> Loaded symbols for /lib/libthr.so.3
>> Reading symbols from /lib/libc.so.7...done.
>> Loaded symbols for /lib/libc.so.7
>> Reading symbols from
>> /usr/local/lib/snort_dynamicengine/libsf_engine.so...done.
>> Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...don
>> e.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...do
>> ne.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so
>> Reading symbols from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done.
>> Loaded symbols for
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so
>> Reading symbols from /libexec/ld-elf.so.1...done.
>> Loaded symbols for /libexec/ld-elf.so.1
>> [Switching to Thread 815a59400 (LWP 100459/snort)]
>> 0x0000000801faa40c in nanosleep () from /lib/libc.so.7
>> (gdb) set logging file gdb-snort.txt
>> (gdb) set logging on
>> Copying output to gdb-snort.txt.
>> (gdb) continue
>> Continuing.
>>
>>
>>
>> <...it has just been a few minutes when I receive a SIGABRT>
>>
>>
>>
>> Program received signal SIGABRT, Aborted.
>> [Switching to Thread 802407400 (LWP 100375/snort)]
>> 0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>>
>> (gdb) backtrace full
>> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>> No symbol table info available.
>> #1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
>> No symbol table info available.
>> #2  0x0000000801fab315 in __assert () from /lib/libc.so.7
>> No symbol table info available.
>> #3  0x0000000805068395 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> No symbol table info available.
>> #4  0x0000000805068781 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> No symbol table info available.
>> #5  0x000000080506afd0 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> No symbol table info available.
>> #6  0x000000080506b85b in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> No symbol table info available.
>> #7  0x000000080506c150 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> No symbol table info available.
>> #8  0x000000080506cb27 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> No symbol table info available.
>> #9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
>> policy_id=0, policy=0x802faa000) at detect.c:136
>> 	scb = (SessionControlBlock *) 0x8a1aad2f0
>> 	ppn = (PreprocEvalFuncNode *) 0x8033ff0a0
>> 	pps_enabled_foo = 1123336
>> 	alerts_processed = true
>> #10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
>> 	retval = 0
>> 	policy_id = 0
>> 	policy = (SnortPolicy *) 0x802faa000
>> 	pktcnt = 0
>> 	snort_ticks_start = 34413886976
>> 	snort_ticks_end = 34413888664
>> #11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
>> 	tmp_do_detect = 1
>> 	tmp_do_detect_content = 1
>> 	snort_ticks_start = 37073416192
>> 	snort_ticks_end = 37069258752
>> 	start_seq = 846966387
>> 	stop_seq = 1940818286
>> 	footprint = 3644
>> 	bytes_processed = 3644
>> 	flushed_bytes = 3644
>> 	pkth = {ts = {tv_sec = 100375, tv_usec = 0}, caplen = 0, pktlen = 0,
>> ingress_index = -1, egress_index = -1, ingress_group = -1, egress_group =
>> -1, flags = 0, opaque = 8, priv_ptr = 0x8a1800000, flow_id = 535241216,
>> address_space_id = 0}
>> 	enc_flags = 2147483648
>> 	snort_ticks_start = 51544732022
>> 	snort_ticks_end = 113187
>> #12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
>> No locals.
>> #13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
>> p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
>> dir=128) at snort_stream_tcp.c:4559
>> 	bytes = 3644
>> #14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
>> st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
>> sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
>> 	fm = (FlushMgr *) 0x80811cfb4
>> #15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
>> tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
>> 	p = (Packet *) 0x8033a4900
>> 	flushed = 1926
>> 	tmp_pcap_hdr = {ts = {tv_sec = 1433431165, tv_usec = 321125}, caplen =
>> 94, pktlen = 94, ingress_index = 5004089, egress_index = 0, ingress_group
>> = 38246208, egress_group = 8, flags = 4294960320, opaque = 32767,
>> priv_ptr = 0x4b4c81, flow_id = 0, address_space_id = 0}
>> #16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
>> freeApplicationData=1) at snort_stream_tcp.c:5115
>> 	tcpssn = (TcpSession *) 0x80811ce50
>> #17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
>> (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
>> No locals.
>> #18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
>> s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
>> 	sscc = {old_mem_in_use = 15788887, client_ip = {family = 2, bits = 32,
>> ip = {u6_addr8 = "\nm\027L", '\0' <repeats 11 times>, u6_addr16 = {27914,
>> 19479, 0, 0, 0, 0, 0, 0}, u6_addr32 = {1276603658, 0, 0, 0}}}, server_ip
>> = {family = 2, bits = 32, ip = {u6_addr8 = "\nm\026\024", '\0' <repeats
>> 11 times>,
>>       u6_addr16 = {27914, 5142, 0, 0, 0, 0, 0, 0}, u6_addr32 =
>> {337014026, 0, 0, 0}}}, client_port = 39946, server_port = 6400,
>> lw_session_state = 200, lw_session_flags = 4284679, app_proto_id = 0}
>> 	tdb = {seq = 1940818286, ack = 2349672268, win = 64032, end_seq =
>> 1940818325, ts = 0}
>> 	rc = 0
>> 	status = 4512282
>> 	snort_ticks_start = 34397587520
>> 	snort_ticks_end = 140737488348864
>> #19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
>> spp_stream6.c:751
>> 	key = {ip_l = {0, 0, 4216431, 0}, ip_h = {0, 2, 362856224, 8}, port_l =
>> 59136, port_h = 65535, vlan_tag = 32767, protocol = 0 '\0', pad = 0 '\0',
>> mplsLabel = 5328944, addressSpaceId = 0, addressSpaceIdPad1 = 0}
>> 	scb = (SessionControlBlock *) 0x8a1aad2f0
>> 	snort_ticks_start = 140737488348960
>> 	snort_ticks_end = 34722594592
>> #20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
>> policy=0x802faa000) at detect.c:136
>> 	scb = (SessionControlBlock *) 0x8a1aad2f0
>> 	ppn = (PreprocEvalFuncNode *) 0x815b61340
>> 	pps_enabled_foo = 1123336
>> 	alerts_processed = true
>> #21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
>> 	retval = 0
>> 	policy_id = 0
>> 	policy = (SnortPolicy *) 0x802faa000
>> 	pktcnt = 0
>> 	snort_ticks_start = 0
>> 	snort_ticks_end = 6059431713369489410
>> #22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873
>> 	verdict = DAQ_VERDICT_PASS
>> 	__func__ = "ProcessPacket"
>> #23 0x0000000000434ccd in PacketCallback (user=0x0,
>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
>> 	inject = 0
>> 	verdict = DAQ_VERDICT_PASS
>> 	snort_ticks_start = 34894979584
>> 	snort_ticks_end = 34367935488
>> #24 0x000000000052fe34 in pcap_process_loop ()
>> No symbol table info available.
>> #25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
>> No symbol table info available.
>> #26 0x000000000053025f in pcap_daq_acquire ()
>> No symbol table info available.
>> #27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
>> <PacketCallback>, user=0x0) at sfdaq.c:541
>> 	err = 0
>> #28 0x0000000000437616 in PacketLoop () at snort.c:3268
>> 	error = 0
>> 	pkts_to_read = 0
>> #29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
>> snort.c:921
>> 	tmp_ptr = 0x0
>> 	intf = 0x8024c4540 "mon0"
>> 	daqInit = 1
>> #30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
>> snort.c:817
>> No locals.
>> rax            0x0	0
>> rbx            0x7fffffffddec	140737488346604
>> rcx            0x801fc8fbc	34393067452
>> rdx            0x0	0
>> rsi            0x6	6
>> rdi            0x18817	100375
>> rbp            0x7fffffffde60	0x7fffffffde60
>> rsp            0x7fffffffddd8	0x7fffffffddd8
>> r8             0x0	0
>> r9             0xfffffe0032ea54a8	-2198169037656
>> r10            0x59	89
>> r11            0x202	514
>> r12            0x80811ce50	34495123024
>> r13            0x8033a51b8	34413892024
>> r14            0x82251deaa	34935529130
>> r15            0x1ba24	113188
>> rip            0x801f2364c	0x801f2364c <thr_kill+12>
>> eflags         0x206	518
>> cs             0x43	67
>> ss             0x3b	59
>> ds             0x0	0
>> es             0x0	0
>> fs             0x0	0
>> gs             0x0	0
>> 0x801f2364c <thr_kill+12>:	jb     0x801f2364f <thr_kill+15>
>> 0x801f2364e <thr_kill+14>:	retq
>> 0x801f2364f <thr_kill+15>:	mov    0x2d6bea(%rip),%rcx        #
>> 0x8021fa240 <__nsdefaultsrc+5696>
>> 0x801f23656 <thr_kill+22>:	jmpq   *%rcx
>> 0x801f23658 <thr_kill+24>:	nop
>> 0x801f23659 <thr_kill+25>:	nop
>> 0x801f2365a <thr_kill+26>:	nop
>> 0x801f2365b <thr_kill+27>:	nop
>> 0x801f2365c <thr_kill+28>:	nop
>> 0x801f2365d <thr_kill+29>:	nop
>> 0x801f2365e <thr_kill+30>:	nop
>> 0x801f2365f <thr_kill+31>:	nop
>> 0x801f23660 <thr_self>:	mov    $0x1b0,%rax
>> 0x801f23667 <thr_self+7>:	mov    %rcx,%r10
>> 0x801f2366a <thr_self+10>:	syscall
>> 0x801f2366c <thr_self+12>:	jb     0x801f2366f <thr_self+15>
>>
>> Thread 2 (Thread 802407400 (LWP 100375/snort)):
>> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>> #1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
>> #2  0x0000000801fab315 in __assert () from /lib/libc.so.7
>> #3  0x0000000805068395 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> #4  0x0000000805068781 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> #5  0x000000080506afd0 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> #6  0x000000080506b85b in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> #7  0x000000080506c150 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> #8  0x000000080506cb27 in ?? () from
>> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>> #9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
>> policy_id=0, policy=0x802faa000) at detect.c:136
>> #10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
>> #11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
>> #12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
>> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
>> #13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
>> p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
>> dir=128) at snort_stream_tcp.c:4559
>> #14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
>> st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
>> sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
>> #15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
>> tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
>> #16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
>> freeApplicationData=1) at snort_stream_tcp.c:5115
>> #17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
>> (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
>> #18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
>> s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
>> #19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
>> spp_stream6.c:751
>> #20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
>> policy=0x802faa000) at detect.c:136
>> #21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
>> #22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873
>> #23 0x0000000000434ccd in PacketCallback (user=0x0,
>> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
>> #24 0x000000000052fe34 in pcap_process_loop ()
>> #25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
>> #26 0x000000000053025f in pcap_daq_acquire ()
>> #27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
>> <PacketCallback>, user=0x0) at sfdaq.c:541
>> #28 0x0000000000437616 in PacketLoop () at snort.c:3268
>> #29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
>> snort.c:921
>> #30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
>> snort.c:817
>>
>> Thread 1 (Thread 815a59400 (LWP 100459/snort)):
>> #0  0x0000000801faa40c in nanosleep () from /lib/libc.so.7
>> #1  0x0000000801f15a58 in sleep () from /lib/libc.so.7
>> #2  0x0000000801ca8078 in sleep () from /lib/libthr.so.3
>> #3  0x000000000043b215 in ReloadConfigThread (data=0x0) at snort.c:5695
>> #4  0x0000000801ca5dc4 in pthread_getprio () from /lib/libthr.so.3
>> #5  0x0000000000000000 in ?? ()
>> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>> The program is running.  Quit anyway (and detach it)? (y or n) Detaching
>> from program: /usr/local/bin/snort, process 11057
>>
>>
>>
>>
>> As gdb detached from snort, I got the signal 6 in my syslog:
>> 2015-06-04 17:51:53 +02:00 foobar kernel: pid 11057 (snort), uid 100:
>> exited on signal 6
>>
>>
>>
>>
>>
>> So, this time we got a signal 6 but during this sensor's 14 hour uptime
>> we've seen:
>> pid 1199 (snort), uid 100: exited on signal 10
>> pid 4503 (snort), uid 100: exited on signal 10
>> pid 5908 (snort), uid 100: exited on signal 10
>> pid 11057 (snort), uid 100: exited on signal 6
>>
>>
>>
>>
>>
>> I hope this gdb was helpful.
>> Let me know if it should be run again.
>>
>>
>>
>>
>>
>> This was all performed on a sensor running:
>>
>>    ,,_     -*> Snort! <*-
>>   o"  )~   Version 2.9.7.3 (Build 217)
>>    ''''    By Martin Roesch & The Snort Team:
>> http://www.snort.org/contact#team
>>            Copyright (C) 2014-2015 Cisco and/or its affiliates. All
>> rights reserved.
>>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>            Using libpcap version 1.4.0
>>            Using PCRE version: 8.37 2015-04-28
>>            Using ZLIB version: 1.2.8
>>
>>
>> daq-2.0.5
>>
>> FreeBSD 9.3-RELEASE-p13
>>
>>
>> /Elof
>>
>>
>>
>>
>>
>> On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>>
>>> That¹s cool. All looks good to me. No need to do more things...
>>>
>>> Best,
>>> Hui
>>>
>>> On 6/4/15, 11:35 AM, "elof at ...969..." <elof at ...969...> wrote:
>>>
>>>>
>>>> Hi Hui.
>>>>
>>>> That much I know. It is the debugging steps I'm curious about.
>>>>
>>>> (I think you forgot one important first command: continue )
>>>>
>>>>
>>>> Is this a good start:
>>>>
>>>> gdb /path/to/snort 1222
>>>> (gdb) set logging file gdb-snort.txt
>>>> (gdb) set logging on
>>>> (gdb) continue
>>>>
>>>> <wait for it to crash>
>>>>
>>>> (gdb) backtrace full
>>>> (gdb) info registers
>>>> (gdb) x/16i $pc
>>>> (gdb) thread apply all backtrace
>>>> (gdb) quit
>>>>
>>>> Email the report.
>>>>
>>>>
>>>> Should I prepare more stuff before the 'continue'?
>>>> Like "handle SIG33 pass nostop noprint" or something?
>>>>
>>>> /Elof
>>>>
>>>>
>>>> On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>>>>
>>>>> Try
>>>>>
>>>>> Assume snort pid is 1222
>>>>>
>>>>> gdb /path/to/snort 1222
>>>>>
>>>>> Best,
>>>>> Hui.
>>>>> On 6/4/15, 10:37 AM, "elof at ...969..." <elof at ...969...> wrote:
>>>>>
>>>>>>
>>>>>> An update:
>>>>>>
>>>>>> On a sensor where snort crashed with signal 6 three times, I
>>>>>> downgraded
>>>>>> daq to 2.0.4_1 and rebooted the machine to rule out if the problem
>>>>>> seem
>>>>>> to
>>>>>> be in 'snort' or 'daq'.
>>>>>>
>>>>>> With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.
>>>>>>
>>>>>>
>>>>>> This make me believe that there's something wrong in snort 2.9.7.3
>>>>>> and
>>>>>> not in daq 2.0.5.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On this sensor I have now done the opposite, upgraded daq to 2.0.5
>>>>>> and
>>>>>> downgraded snort to 2.9.7.2 to see if I get any more signal 6.
>>>>>>
>>>>>> On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
>>>>>> 2.0.5
>>>>>> without chroot and uid/gid change, i.e. running as root, in order to
>>>>>> create a core file, if the problem happen again.
>>>>>> (if it doesn't happen on this sensor, I guess the problem lies
>>>>>> somewhere
>>>>>> in the chrooting code in snort. I know it has been updated between
>>>>>> 2.9.7.2
>>>>>> and 2.9.7.3)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Russ C also wrote:
>>>>>>> Elof - since this is happening frequently, you could try attaching
>>>>>>> the
>>>>>>> debugger to one of your Snort processes and wait wait for segfault.
>>>>>>
>>>>>> I know too little about debugging. :-/ Can you give me instructions
>>>>>> or
>>>>>> point me to a guide that describes the steps I should take?
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Elof
>>>>>>
>>>>>>
>>>>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>>>>
>>>>>>>
>>>>>>> Five different sensors have now had bus errors (signal 10),
>>>>>>> segmentation
>>>>>>> faults (signal 11) and even signal 6 (SIGABRT).
>>>>>>>
>>>>>>> My snort config uses both chroot and dropping user privileges, so
>>>>>>> even
>>>>>>> if
>>>>>>> I start out as root with ulimit unlimited, this doesn't seem to be
>>>>>>> in
>>>>>>> effect
>>>>>>> after the chroot/uid-change.
>>>>>>>
>>>>>>> So currently I have no core-file to debug. :-/
>>>>>>>
>>>>>>> Anyone know how to set the ulimits for a chrooted and
>>>>>>> uid/gid-changed
>>>>>>> process in FreeBSD?
>>>>>>>
>>>>>>> /Elof
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Hui!
>>>>>>>>
>>>>>>>> Yes, the dynamic engine/preproc files are updated as well.
>>>>>>>>
>>>>>>>> Last night the problem reocurred, so this seem to be reproduceable.
>>>>>>>> Good.
>>>>>>>> Then there's a good chance this problem can be sorted out.
>>>>>>>>
>>>>>>>>
>>>>>>>> A few minutes ago a signal 10 happened on another sensor (running
>>>>>>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in
>>>>>>>> Snort
>>>>>>>> 2.9.7.3 and not in the hardware nor in FreeBSD.
>>>>>>>>
>>>>>>>>
>>>>>>>> I will compile a debug-snort and try to generate core files.
>>>>>>>> I'll let you know the outcome next week.
>>>>>>>>
>>>>>>>> /Elof
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, 3 Jun 2015, Hui cao wrote:
>>>>>>>>
>>>>>>>>> Hi Elof,
>>>>>>>>>
>>>>>>>>> Are snort and snort dynamic preprocessors are in sync?
>>>>>>>>>
>>>>>>>>> If so, can you help us get a backtrace from the crush? You need
>>>>>>>>> 1)  build snort with ./configure --enable-debug
>>>>>>>>> 2)  allowing core dump (ulimit -c unlimited)
>>>>>>>>> 3) run the snort
>>>>>>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb
>>>>>>>>> command
>>>>>>>>> line
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Hui.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06/03/2015 05:51 AM, elof at ...969... wrote:
>>>>>>>>>> Hi all!
>>>>>>>>>>
>>>>>>>>>> This is just a report to inform that after I updated snort and
>>>>>>>>>> DAQ
>>>>>>>>>> to the
>>>>>>>>>> latest versions, one of my sensors started throwing signal 10
>>>>>>>>>> (bus
>>>>>>>>>> error)
>>>>>>>>>> and signal 11 (segmentation fault).
>>>>>>>>>>
>>>>>>>>>> # uptime
>>>>>>>>>> 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
>>>>>>>>>> # dmesg | grep snort
>>>>>>>>>> pid 1183 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 16920 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 17502 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 18862 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 20223 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 20927 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 1193 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 2447 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 3811 (snort), uid 100: exited on signal 10
>>>>>>>>>> pid 7881 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 9252 (snort), uid 100: exited on signal 10
>>>>>>>>>> pid 25593 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 26627 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 56658 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 57237 (snort), uid 100: exited on signal 10
>>>>>>>>>> pid 58595 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 68639 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 70008 (snort), uid 100: exited on signal 11
>>>>>>>>>> pid 71361 (snort), uid 100: exited on signal 10
>>>>>>>>>> pid 72725 (snort), uid 100: exited on signal 11
>>>>>>>>>>
>>>>>>>>>> 20 crashes in a day...
>>>>>>>>>> A reboot didn't help.
>>>>>>>>>>
>>>>>>>>>> This sensor has never behaved like this during its lifetime (1
>>>>>>>>>> year).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> FreeBSD 9.3 amd64
>>>>>>>>>>
>>>>>>>>>>      ,,_     -*> Snort! <*-
>>>>>>>>>>     o"  )~   Version 2.9.7.3 (Build 217)
>>>>>>>>>>      ''''    By Martin Roesch & The Snort Team:
>>>>>>>>>> http://www.snort.org/contact#team
>>>>>>>>>>              Copyright (C) 2014-2015 Cisco and/or its affiliates.
>>>>>>>>>> All rights
>>>>>>>>>> reserved.
>>>>>>>>>>              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>>>>>>>              Using libpcap version 1.4.0
>>>>>>>>>>              Using PCRE version: 8.37 2015-04-28
>>>>>>>>>>              Using ZLIB version: 1.2.8
>>>>>>>>>>
>>>>>>>>>> daq-2.0.5
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Bus errors are quite unusual in general, so I'll keep looking at
>>>>>>>>>> this,
>>>>>>>>>> trying to see if it is e.g. paging errors.
>>>>>>>>>> It doesn't look like it though:
>>>>>>>>>> # swapinfo
>>>>>>>>>> Device          1K-blocks     Used    Avail Capacity
>>>>>>>>>> /dev/mirror/swap   4194300        0  4194300     0%
>>>>>>>>>>
>>>>>>>>>> The machine doesn't seem to be overheated either:
>>>>>>>>>> System Temp:	30 degrees C
>>>>>>>>>> Peripheral Temp: 40 degrees C
>>>>>>>>>> CPU Temp: Low
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If you need me to do something special to debug this further, let
>>>>>>>>>> me
>>>>>>>>>> know.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> PS. It is only one sensor, out of 20, that behaves like this. So
>>>>>>>>>> perhaps
>>>>>>>>>> it is something in the mirrored traffic that make DAQ or snort
>>>>>>>>>> point
>>>>>>>>>> at
>>>>>>>>>> illegal memory addresses and crash.
>>>>>>>>>> Or this particular machine is having hardware issues. However, it
>>>>>>>>>> is
>>>>>>>>>> strange that those hw-issues should suddenly start right after I
>>>>>>>>>> updated
>>>>>>>>>> the software on the machine...
>>>>>>>>>>
>>>>>>>>>> When I write this, the current snort process has been alive for 5
>>>>>>>>>> hours.
>>>>>>>>>> It's going to be interesting to see if the traffic tonight will
>>>>>>>>>> cause it
>>>>>>>>>> to crash many times again.
>>>>>>>>>>
>>>>>>>>>> /Elof
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------------------------------------------
>>>>>>>>>> --
>>>>>>>>>> --
>>>>>>>>>> --------
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Snort-devel mailing list
>>>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>>>> Archive:
>>>>>>>>>>
>>>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-deve
>>>>>>>>>> l
>>>>>>>>>>
>>>>>>>>>> Please visit http://blog.snort.org for the latest news about
>>>>>>>>>> Snort!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -------------------------------------------------------------------
>>>>>>>>> --
>>>>>>>>> --
>>>>>>>>> -------
>>>>>>>>> _______________________________________________
>>>>>>>>> Snort-devel mailing list
>>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>>> Archive:
>>>>>>>>>
>>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>>>>
>>>>>>>>> Please visit http://blog.snort.org for the latest news about
>>>>>>>>> Snort!
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------
>>>>>>>> --
>>>>>>>> --
>>>>>>>> ------
>>>>>>>> _______________________________________________
>>>>>>>> Snort-devel mailing list
>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>> Archive:
>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>>>
>>>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> --
>>>>>>> --
>>>>>>> -----
>>>>>>> _______________________________________________
>>>>>>> Snort-devel mailing list
>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>> Archive:
>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>>
>>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>>
>>>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------------
>>> -----
>>> _______________________________________________
>>> Snort-devel mailing list
>>> Snort-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>> Archive:
>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>
>>> Please visit http://blog.snort.org for the latest news about Snort!
>
>


More information about the Snort-devel mailing list