[Snort-devel] Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

Hui Cao (huica) huica at ...3461...
Thu Jun 4 12:36:15 EDT 2015


Thanks!

The issue happens on smtp preprocessor, but the so is not compiled with
debug enabled. Can you recompile it with ―enable-debug ?

Best,
Hui.

On 6/4/15, 12:10 PM, "elof at ...969..." <elof at ...969...> wrote:

>
>So I just had a signal 6...
>
>I assume I can't attach files to the mailing list, so here it is,
>directly 
>in the mailbody. :-)
>
>
>
>
>
>gdb /usr/local/bin/snort 11057
>
>GNU gdb 6.1.1 [FreeBSD]
>Copyright 2004 Free Software Foundation, Inc.
>GDB is free software, covered by the GNU General Public License, and you
>are
>welcome to change it and/or distribute copies of it under certain
>conditions.
>Type "show copying" to see the conditions.
>There is absolutely no warranty for GDB.  Type "show warranty" for
>details.
>This GDB was configured as "amd64-marcel-freebsd"...
>Attaching to program: /usr/local/bin/snort, process 11057
>Reading symbols from /usr/local/lib/libdnet.so.1...done.
>Loaded symbols for /usr/local/lib/libdnet.so.1
>Reading symbols from /usr/local/lib/libpcre.so.1...done.
>Loaded symbols for /usr/local/lib/libpcre.so.1
>Reading symbols from /lib/libm.so.5...done.
>Loaded symbols for /lib/libm.so.5
>Reading symbols from /lib/libcrypto.so.6...done.
>Loaded symbols for /lib/libcrypto.so.6
>Reading symbols from /lib/libpcap.so.8...done.
>Loaded symbols for /lib/libpcap.so.8
>Reading symbols from /usr/local/lib/libsfbpf.so.0...done.
>Loaded symbols for /usr/local/lib/libsfbpf.so.0
>Reading symbols from /lib/libz.so.6...done.
>Loaded symbols for /lib/libz.so.6
>Reading symbols from /usr/lib/liblzma.so.5...done.
>Loaded symbols for /usr/lib/liblzma.so.5
>Reading symbols from /lib/libthr.so.3...done.
>[New Thread 815a59400 (LWP 100459/snort)]
>[New Thread 802407400 (LWP 100375/snort)]
>Loaded symbols for /lib/libthr.so.3
>Reading symbols from /lib/libc.so.7...done.
>Loaded symbols for /lib/libc.so.7
>Reading symbols from
>/usr/local/lib/snort_dynamicengine/libsf_engine.so...done.
>Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...don
>e.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...do
>ne.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so
>Reading symbols from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done.
>Loaded symbols for
>/usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so
>Reading symbols from /libexec/ld-elf.so.1...done.
>Loaded symbols for /libexec/ld-elf.so.1
>[Switching to Thread 815a59400 (LWP 100459/snort)]
>0x0000000801faa40c in nanosleep () from /lib/libc.so.7
>(gdb) set logging file gdb-snort.txt
>(gdb) set logging on
>Copying output to gdb-snort.txt.
>(gdb) continue
>Continuing.
>
>
>
><...it has just been a few minutes when I receive a SIGABRT>
>
>
>
>Program received signal SIGABRT, Aborted.
>[Switching to Thread 802407400 (LWP 100375/snort)]
>0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>
>(gdb) backtrace full
>#0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>No symbol table info available.
>#1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
>No symbol table info available.
>#2  0x0000000801fab315 in __assert () from /lib/libc.so.7
>No symbol table info available.
>#3  0x0000000805068395 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>No symbol table info available.
>#4  0x0000000805068781 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>No symbol table info available.
>#5  0x000000080506afd0 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>No symbol table info available.
>#6  0x000000080506b85b in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>No symbol table info available.
>#7  0x000000080506c150 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>No symbol table info available.
>#8  0x000000080506cb27 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>No symbol table info available.
>#9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
>policy_id=0, policy=0x802faa000) at detect.c:136
> 	scb = (SessionControlBlock *) 0x8a1aad2f0
> 	ppn = (PreprocEvalFuncNode *) 0x8033ff0a0
> 	pps_enabled_foo = 1123336
> 	alerts_processed = true
>#10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
> 	retval = 0
> 	policy_id = 0
> 	policy = (SnortPolicy *) 0x802faa000
> 	pktcnt = 0
> 	snort_ticks_start = 34413886976
> 	snort_ticks_end = 34413888664
>#11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
>st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
> 	tmp_do_detect = 1
> 	tmp_do_detect_content = 1
> 	snort_ticks_start = 37073416192
> 	snort_ticks_end = 37069258752
> 	start_seq = 846966387
> 	stop_seq = 1940818286
> 	footprint = 3644
> 	bytes_processed = 3644
> 	flushed_bytes = 3644
> 	pkth = {ts = {tv_sec = 100375, tv_usec = 0}, caplen = 0, pktlen = 0,
>ingress_index = -1, egress_index = -1, ingress_group = -1, egress_group =
>-1, flags = 0, opaque = 8, priv_ptr = 0x8a1800000, flow_id = 535241216,
>address_space_id = 0}
> 	enc_flags = 2147483648
> 	snort_ticks_start = 51544732022
> 	snort_ticks_end = 113187
>#12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
>st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
>No locals.
>#13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
>p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
>dir=128) at snort_stream_tcp.c:4559
> 	bytes = 3644
>#14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
>st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
>sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
> 	fm = (FlushMgr *) 0x80811cfb4
>#15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
>tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
> 	p = (Packet *) 0x8033a4900
> 	flushed = 1926
> 	tmp_pcap_hdr = {ts = {tv_sec = 1433431165, tv_usec = 321125}, caplen =
>94, pktlen = 94, ingress_index = 5004089, egress_index = 0, ingress_group
>= 38246208, egress_group = 8, flags = 4294960320, opaque = 32767,
>priv_ptr = 0x4b4c81, flow_id = 0, address_space_id = 0}
>#16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
>freeApplicationData=1) at snort_stream_tcp.c:5115
> 	tcpssn = (TcpSession *) 0x80811ce50
>#17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
>(scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
>No locals.
>#18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
>s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
> 	sscc = {old_mem_in_use = 15788887, client_ip = {family = 2, bits = 32,
>ip = {u6_addr8 = "\nm\027L", '\0' <repeats 11 times>, u6_addr16 = {27914,
>19479, 0, 0, 0, 0, 0, 0}, u6_addr32 = {1276603658, 0, 0, 0}}}, server_ip
>= {family = 2, bits = 32, ip = {u6_addr8 = "\nm\026\024", '\0' <repeats
>11 times>,
>       u6_addr16 = {27914, 5142, 0, 0, 0, 0, 0, 0}, u6_addr32 =
>{337014026, 0, 0, 0}}}, client_port = 39946, server_port = 6400,
>lw_session_state = 200, lw_session_flags = 4284679, app_proto_id = 0}
> 	tdb = {seq = 1940818286, ack = 2349672268, win = 64032, end_seq =
>1940818325, ts = 0}
> 	rc = 0
> 	status = 4512282
> 	snort_ticks_start = 34397587520
> 	snort_ticks_end = 140737488348864
>#19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
>spp_stream6.c:751
> 	key = {ip_l = {0, 0, 4216431, 0}, ip_h = {0, 2, 362856224, 8}, port_l =
>59136, port_h = 65535, vlan_tag = 32767, protocol = 0 '\0', pad = 0 '\0',
>mplsLabel = 5328944, addressSpaceId = 0, addressSpaceIdPad1 = 0}
> 	scb = (SessionControlBlock *) 0x8a1aad2f0
> 	snort_ticks_start = 140737488348960
> 	snort_ticks_end = 34722594592
>#20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
>policy=0x802faa000) at detect.c:136
> 	scb = (SessionControlBlock *) 0x8a1aad2f0
> 	ppn = (PreprocEvalFuncNode *) 0x815b61340
> 	pps_enabled_foo = 1123336
> 	alerts_processed = true
>#21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
> 	retval = 0
> 	policy_id = 0
> 	policy = (SnortPolicy *) 0x802faa000
> 	pktcnt = 0
> 	snort_ticks_start = 0
> 	snort_ticks_end = 6059431713369489410
>#22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
>pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873
> 	verdict = DAQ_VERDICT_PASS
> 	__func__ = "ProcessPacket"
>#23 0x0000000000434ccd in PacketCallback (user=0x0,
>pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
> 	inject = 0
> 	verdict = DAQ_VERDICT_PASS
> 	snort_ticks_start = 34894979584
> 	snort_ticks_end = 34367935488
>#24 0x000000000052fe34 in pcap_process_loop ()
>No symbol table info available.
>#25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
>No symbol table info available.
>#26 0x000000000053025f in pcap_daq_acquire ()
>No symbol table info available.
>#27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
><PacketCallback>, user=0x0) at sfdaq.c:541
> 	err = 0
>#28 0x0000000000437616 in PacketLoop () at snort.c:3268
> 	error = 0
> 	pkts_to_read = 0
>#29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
>snort.c:921
> 	tmp_ptr = 0x0
> 	intf = 0x8024c4540 "mon0"
> 	daqInit = 1
>#30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
>snort.c:817
>No locals.
>rax            0x0	0
>rbx            0x7fffffffddec	140737488346604
>rcx            0x801fc8fbc	34393067452
>rdx            0x0	0
>rsi            0x6	6
>rdi            0x18817	100375
>rbp            0x7fffffffde60	0x7fffffffde60
>rsp            0x7fffffffddd8	0x7fffffffddd8
>r8             0x0	0
>r9             0xfffffe0032ea54a8	-2198169037656
>r10            0x59	89
>r11            0x202	514
>r12            0x80811ce50	34495123024
>r13            0x8033a51b8	34413892024
>r14            0x82251deaa	34935529130
>r15            0x1ba24	113188
>rip            0x801f2364c	0x801f2364c <thr_kill+12>
>eflags         0x206	518
>cs             0x43	67
>ss             0x3b	59
>ds             0x0	0
>es             0x0	0
>fs             0x0	0
>gs             0x0	0
>0x801f2364c <thr_kill+12>:	jb     0x801f2364f <thr_kill+15>
>0x801f2364e <thr_kill+14>:	retq
>0x801f2364f <thr_kill+15>:	mov    0x2d6bea(%rip),%rcx        #
>0x8021fa240 <__nsdefaultsrc+5696>
>0x801f23656 <thr_kill+22>:	jmpq   *%rcx
>0x801f23658 <thr_kill+24>:	nop
>0x801f23659 <thr_kill+25>:	nop
>0x801f2365a <thr_kill+26>:	nop
>0x801f2365b <thr_kill+27>:	nop
>0x801f2365c <thr_kill+28>:	nop
>0x801f2365d <thr_kill+29>:	nop
>0x801f2365e <thr_kill+30>:	nop
>0x801f2365f <thr_kill+31>:	nop
>0x801f23660 <thr_self>:	mov    $0x1b0,%rax
>0x801f23667 <thr_self+7>:	mov    %rcx,%r10
>0x801f2366a <thr_self+10>:	syscall
>0x801f2366c <thr_self+12>:	jb     0x801f2366f <thr_self+15>
>
>Thread 2 (Thread 802407400 (LWP 100375/snort)):
>#0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>#1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
>#2  0x0000000801fab315 in __assert () from /lib/libc.so.7
>#3  0x0000000805068395 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>#4  0x0000000805068781 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>#5  0x000000080506afd0 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>#6  0x000000080506b85b in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>#7  0x000000080506c150 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>#8  0x000000080506cb27 in ?? () from
>/usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
>#9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
>policy_id=0, policy=0x802faa000) at detect.c:136
>#10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
>#11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
>st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
>#12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
>st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
>dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
>#13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
>p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
>dir=128) at snort_stream_tcp.c:4559
>#14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
>st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
>sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
>#15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
>tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
>#16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
>freeApplicationData=1) at snort_stream_tcp.c:5115
>#17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
>(scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
>#18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
>s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
>#19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
>spp_stream6.c:751
>#20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
>policy=0x802faa000) at detect.c:136
>#21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
>#22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
>pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873
>#23 0x0000000000434ccd in PacketCallback (user=0x0,
>pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
>#24 0x000000000052fe34 in pcap_process_loop ()
>#25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
>#26 0x000000000053025f in pcap_daq_acquire ()
>#27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
><PacketCallback>, user=0x0) at sfdaq.c:541
>#28 0x0000000000437616 in PacketLoop () at snort.c:3268
>#29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
>snort.c:921
>#30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
>snort.c:817
>
>Thread 1 (Thread 815a59400 (LWP 100459/snort)):
>#0  0x0000000801faa40c in nanosleep () from /lib/libc.so.7
>#1  0x0000000801f15a58 in sleep () from /lib/libc.so.7
>#2  0x0000000801ca8078 in sleep () from /lib/libthr.so.3
>#3  0x000000000043b215 in ReloadConfigThread (data=0x0) at snort.c:5695
>#4  0x0000000801ca5dc4 in pthread_getprio () from /lib/libthr.so.3
>#5  0x0000000000000000 in ?? ()
>#0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>The program is running.  Quit anyway (and detach it)? (y or n) Detaching
>from program: /usr/local/bin/snort, process 11057
>
>
>
>
>As gdb detached from snort, I got the signal 6 in my syslog:
>2015-06-04 17:51:53 +02:00 foobar kernel: pid 11057 (snort), uid 100:
>exited on signal 6
>
>
>
>
>
>So, this time we got a signal 6 but during this sensor's 14 hour uptime
>we've seen:
>pid 1199 (snort), uid 100: exited on signal 10
>pid 4503 (snort), uid 100: exited on signal 10
>pid 5908 (snort), uid 100: exited on signal 10
>pid 11057 (snort), uid 100: exited on signal 6
>
>
>
>
>
>I hope this gdb was helpful.
>Let me know if it should be run again.
>
>
>
>
>
>This was all performed on a sensor running:
>
>    ,,_     -*> Snort! <*-
>   o"  )~   Version 2.9.7.3 (Build 217)
>    ''''    By Martin Roesch & The Snort Team:
>http://www.snort.org/contact#team
>            Copyright (C) 2014-2015 Cisco and/or its affiliates. All
>rights reserved.
>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>            Using libpcap version 1.4.0
>            Using PCRE version: 8.37 2015-04-28
>            Using ZLIB version: 1.2.8
>
>
>daq-2.0.5
>
>FreeBSD 9.3-RELEASE-p13
>
>
>/Elof
>
>
>
>
>
>On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>
>> That¹s cool. All looks good to me. No need to do more things...
>>
>> Best,
>> Hui
>>
>> On 6/4/15, 11:35 AM, "elof at ...969..." <elof at ...969...> wrote:
>>
>>>
>>> Hi Hui.
>>>
>>> That much I know. It is the debugging steps I'm curious about.
>>>
>>> (I think you forgot one important first command: continue )
>>>
>>>
>>> Is this a good start:
>>>
>>> gdb /path/to/snort 1222
>>> (gdb) set logging file gdb-snort.txt
>>> (gdb) set logging on
>>> (gdb) continue
>>>
>>> <wait for it to crash>
>>>
>>> (gdb) backtrace full
>>> (gdb) info registers
>>> (gdb) x/16i $pc
>>> (gdb) thread apply all backtrace
>>> (gdb) quit
>>>
>>> Email the report.
>>>
>>>
>>> Should I prepare more stuff before the 'continue'?
>>> Like "handle SIG33 pass nostop noprint" or something?
>>>
>>> /Elof
>>>
>>>
>>> On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>>>
>>>> Try
>>>>
>>>> Assume snort pid is 1222
>>>>
>>>> gdb /path/to/snort 1222
>>>>
>>>> Best,
>>>> Hui.
>>>> On 6/4/15, 10:37 AM, "elof at ...969..." <elof at ...969...> wrote:
>>>>
>>>>>
>>>>> An update:
>>>>>
>>>>> On a sensor where snort crashed with signal 6 three times, I
>>>>>downgraded
>>>>> daq to 2.0.4_1 and rebooted the machine to rule out if the problem
>>>>>seem
>>>>> to
>>>>> be in 'snort' or 'daq'.
>>>>>
>>>>> With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.
>>>>>
>>>>>
>>>>> This make me believe that there's something wrong in snort 2.9.7.3
>>>>>and
>>>>> not in daq 2.0.5.
>>>>>
>>>>>
>>>>>
>>>>> On this sensor I have now done the opposite, upgraded daq to 2.0.5
>>>>>and
>>>>> downgraded snort to 2.9.7.2 to see if I get any more signal 6.
>>>>>
>>>>> On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
>>>>> 2.0.5
>>>>> without chroot and uid/gid change, i.e. running as root, in order to
>>>>> create a core file, if the problem happen again.
>>>>> (if it doesn't happen on this sensor, I guess the problem lies
>>>>> somewhere
>>>>> in the chrooting code in snort. I know it has been updated between
>>>>> 2.9.7.2
>>>>> and 2.9.7.3)
>>>>>
>>>>>
>>>>>
>>>>> Russ C also wrote:
>>>>>> Elof - since this is happening frequently, you could try attaching
>>>>>>the
>>>>>> debugger to one of your Snort processes and wait wait for segfault.
>>>>>
>>>>> I know too little about debugging. :-/ Can you give me instructions
>>>>>or
>>>>> point me to a guide that describes the steps I should take?
>>>>>
>>>>>
>>>>>
>>>>> /Elof
>>>>>
>>>>>
>>>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>>>
>>>>>>
>>>>>> Five different sensors have now had bus errors (signal 10),
>>>>>> segmentation
>>>>>> faults (signal 11) and even signal 6 (SIGABRT).
>>>>>>
>>>>>> My snort config uses both chroot and dropping user privileges, so
>>>>>>even
>>>>>> if
>>>>>> I start out as root with ulimit unlimited, this doesn't seem to be
>>>>>>in
>>>>>> effect
>>>>>> after the chroot/uid-change.
>>>>>>
>>>>>> So currently I have no core-file to debug. :-/
>>>>>>
>>>>>> Anyone know how to set the ulimits for a chrooted and
>>>>>>uid/gid-changed
>>>>>> process in FreeBSD?
>>>>>>
>>>>>> /Elof
>>>>>>
>>>>>>
>>>>>> On Thu, 4 Jun 2015, elof at ...969... wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi Hui!
>>>>>>>
>>>>>>> Yes, the dynamic engine/preproc files are updated as well.
>>>>>>>
>>>>>>> Last night the problem reocurred, so this seem to be reproduceable.
>>>>>>> Good.
>>>>>>> Then there's a good chance this problem can be sorted out.
>>>>>>>
>>>>>>>
>>>>>>> A few minutes ago a signal 10 happened on another sensor (running
>>>>>>> FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in
>>>>>>>Snort
>>>>>>> 2.9.7.3 and not in the hardware nor in FreeBSD.
>>>>>>>
>>>>>>>
>>>>>>> I will compile a debug-snort and try to generate core files.
>>>>>>> I'll let you know the outcome next week.
>>>>>>>
>>>>>>> /Elof
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 3 Jun 2015, Hui cao wrote:
>>>>>>>
>>>>>>>> Hi Elof,
>>>>>>>>
>>>>>>>> Are snort and snort dynamic preprocessors are in sync?
>>>>>>>>
>>>>>>>> If so, can you help us get a backtrace from the crush? You need
>>>>>>>> 1)  build snort with ./configure --enable-debug
>>>>>>>> 2)  allowing core dump (ulimit -c unlimited)
>>>>>>>> 3) run the snort
>>>>>>>> 4) use "gdb snort core_file " and them type "bt" in the gdb
>>>>>>>>command
>>>>>>>> line
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Hui.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/03/2015 05:51 AM, elof at ...969... wrote:
>>>>>>>>> Hi all!
>>>>>>>>>
>>>>>>>>> This is just a report to inform that after I updated snort and
>>>>>>>>>DAQ
>>>>>>>>> to the
>>>>>>>>> latest versions, one of my sensors started throwing signal 10
>>>>>>>>>(bus
>>>>>>>>> error)
>>>>>>>>> and signal 11 (segmentation fault).
>>>>>>>>>
>>>>>>>>> # uptime
>>>>>>>>> 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
>>>>>>>>> # dmesg | grep snort
>>>>>>>>> pid 1183 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 16920 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 17502 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 18862 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 20223 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 20927 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 1193 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 2447 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 3811 (snort), uid 100: exited on signal 10
>>>>>>>>> pid 7881 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 9252 (snort), uid 100: exited on signal 10
>>>>>>>>> pid 25593 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 26627 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 56658 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 57237 (snort), uid 100: exited on signal 10
>>>>>>>>> pid 58595 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 68639 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 70008 (snort), uid 100: exited on signal 11
>>>>>>>>> pid 71361 (snort), uid 100: exited on signal 10
>>>>>>>>> pid 72725 (snort), uid 100: exited on signal 11
>>>>>>>>>
>>>>>>>>> 20 crashes in a day...
>>>>>>>>> A reboot didn't help.
>>>>>>>>>
>>>>>>>>> This sensor has never behaved like this during its lifetime (1
>>>>>>>>> year).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> FreeBSD 9.3 amd64
>>>>>>>>>
>>>>>>>>>      ,,_     -*> Snort! <*-
>>>>>>>>>     o"  )~   Version 2.9.7.3 (Build 217)
>>>>>>>>>      ''''    By Martin Roesch & The Snort Team:
>>>>>>>>> http://www.snort.org/contact#team
>>>>>>>>>              Copyright (C) 2014-2015 Cisco and/or its affiliates.
>>>>>>>>> All rights
>>>>>>>>> reserved.
>>>>>>>>>              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>>>>>>>>              Using libpcap version 1.4.0
>>>>>>>>>              Using PCRE version: 8.37 2015-04-28
>>>>>>>>>              Using ZLIB version: 1.2.8
>>>>>>>>>
>>>>>>>>> daq-2.0.5
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Bus errors are quite unusual in general, so I'll keep looking at
>>>>>>>>> this,
>>>>>>>>> trying to see if it is e.g. paging errors.
>>>>>>>>> It doesn't look like it though:
>>>>>>>>> # swapinfo
>>>>>>>>> Device          1K-blocks     Used    Avail Capacity
>>>>>>>>> /dev/mirror/swap   4194300        0  4194300     0%
>>>>>>>>>
>>>>>>>>> The machine doesn't seem to be overheated either:
>>>>>>>>> System Temp:	30 degrees C
>>>>>>>>> Peripheral Temp: 40 degrees C
>>>>>>>>> CPU Temp: Low
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you need me to do something special to debug this further, let
>>>>>>>>> me
>>>>>>>>> know.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> PS. It is only one sensor, out of 20, that behaves like this. So
>>>>>>>>> perhaps
>>>>>>>>> it is something in the mirrored traffic that make DAQ or snort
>>>>>>>>> point
>>>>>>>>> at
>>>>>>>>> illegal memory addresses and crash.
>>>>>>>>> Or this particular machine is having hardware issues. However, it
>>>>>>>>> is
>>>>>>>>> strange that those hw-issues should suddenly start right after I
>>>>>>>>> updated
>>>>>>>>> the software on the machine...
>>>>>>>>>
>>>>>>>>> When I write this, the current snort process has been alive for 5
>>>>>>>>> hours.
>>>>>>>>> It's going to be interesting to see if the traffic tonight will
>>>>>>>>> cause it
>>>>>>>>> to crash many times again.
>>>>>>>>>
>>>>>>>>> /Elof
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>>------------------------------------------------------------------
>>>>>>>>>--
>>>>>>>>> --
>>>>>>>>> --------
>>>>>>>>> _______________________________________________
>>>>>>>>> Snort-devel mailing list
>>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>>> Archive:
>>>>>>>>> 
>>>>>>>>>http://sourceforge.net/mailarchive/forum.php?forum_name=snort-deve
>>>>>>>>>l
>>>>>>>>>
>>>>>>>>> Please visit http://blog.snort.org for the latest news about
>>>>>>>>>Snort!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 
>>>>>>>>-------------------------------------------------------------------
>>>>>>>>--
>>>>>>>> --
>>>>>>>> -------
>>>>>>>> _______________________________________________
>>>>>>>> Snort-devel mailing list
>>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>>> Archive:
>>>>>>>> 
>>>>>>>>http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>>>
>>>>>>>> Please visit http://blog.snort.org for the latest news about
>>>>>>>>Snort!
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 
>>>>>>>--------------------------------------------------------------------
>>>>>>>--
>>>>>>> --
>>>>>>> ------
>>>>>>> _______________________________________________
>>>>>>> Snort-devel mailing list
>>>>>>> Snort-devel at lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>>> Archive:
>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>>
>>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>---------------------------------------------------------------------
>>>>>>--
>>>>>> --
>>>>>> -----
>>>>>> _______________________________________________
>>>>>> Snort-devel mailing list
>>>>>> Snort-devel at lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>>> Archive:
>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>>>
>>>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>>>
>>>>
>>
>>
>> 
>>-------------------------------------------------------------------------
>>-----
>> _______________________________________________
>> Snort-devel mailing list
>> Snort-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>> Archive:
>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>
>> Please visit http://blog.snort.org for the latest news about Snort!



More information about the Snort-devel mailing list