[Snort-users] Poor performance with Snort 2.9.4.6 under OpenBSD 5.3

C. L. Martinez carlopmart at ...11827...
Wed Jun 12 03:08:16 EDT 2013


On Fri, Jun 7, 2013 at 10:36 AM, C. L. Martinez <carlopmart at ...11827...> wrote:
> On Thu, Jun 6, 2013 at 7:36 AM, C. L. Martinez <carlopmart at ...11827...> wrote:
>> On Wed, Jun 5, 2013 at 5:08 PM, Victor Roemer <vroemer at ...1935...> wrote:
>>> Martinez, as Joel already mentioned, we'll want to see your Snort
>>> configuration. Shutdown stats would also be useful, but perfmon data would
>>> be better; if those can be provided.
>>>
>>> You mentioned that OpenBSD configured the network sysctl parameters "on the
>>> fly"; could you direct us to some documentation about this?
>>>
>>> You also mentioned that Snort was listening on em3, however the startup
>>> information in your email indicates that Snort is listening on em4, could
>>> you elaborate on this setup?
>>>
>>>
>>> Regarding Suricata, I personally do not have any experience in deploying or
>>> configuring it. That said, are you using, relatively, the same
>>> configurations? (e.g., any rules enabled, acquiring packets via libpcap,
>>> etc..)
>>>
>>> Also, why are "tcp.reassembly_gap" and "tcp.invalid_checksum" relevant?
>>>
>>
>> Ok, first of all sorry for this long post. I will try to answer to all
>> questions. First, my environment:
>>
>> Host: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 8 GiB RAM
>> Snort: release 2.9.4.6 compiled against libpcap 1.3.0
>> Suricata: release 1.4.2 compiled against libpcap 1.3.0
>>
>> Snort is listening in em4 interface and suricata in em5 (there was a
>> typo error previously).
>>
>> Performance data using my actual snort.conf (with few rules and
>> without so_rules or IP based rules):
>>
>> Packet Performance Monitor Config:
>>   ticks per usec  : 2405 ticks
>>   max packet time : 10000 usecs
>>   packet action   : fastpath-expensive-packets
>>   packet logging  : log
>>   debug-pkts      : disabled
>>
>> Rule Performance Monitor Config:
>>   ticks per usec  : 2405 ticks
>>   max rule time   : 4096 usecs
>>   rule action     : suspend-expensive-rules
>>   rule threshold  : 5
>>   suspend timeout : 10 secs
>>   rule logging    : log
>> pcap DAQ configured to passive.
>> Acquiring network traffic from "em4".
>> Reload thread starting...
>> Reload thread started, thread 0x1ee8c530a500 (4050)
>> Decoding Ethernet
>>
>>         --== Initialization Complete ==--
>>
>>    ,,_     -*> Snort! <*-
>>   o"  )~   Version 2.9.4.6 GRE (Build 73)
>>    ''''    By Martin Roesch & The Snort Team:
>> http://www.snort.org/snort/snort-team
>>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>            Using libpcap version 1.3.0
>>            Using PCRE version: 8.31 2012-07-06
>>            Using ZLIB version: 1.2.3
>>
>>            Rules Engine: SF_SNORT_DETECTION_ENGINE  Version 1.17  <Build 18>
>>            Preprocessor Object: SF_DNP3  Version 1.1  <Build 1>
>>            Preprocessor Object: SF_MODBUS  Version 1.1  <Build 1>
>>            Preprocessor Object: SF_GTP  Version 1.1  <Build 1>
>>            Preprocessor Object: SF_REPUTATION  Version 1.1  <Build 1>
>>            Preprocessor Object: SF_SIP  Version 1.1  <Build 1>
>>            Preprocessor Object: SF_SDF  Version 1.1  <Build 1>
>>            Preprocessor Object: SF_DCERPC2  Version 1.0  <Build 3>
>>            Preprocessor Object: SF_SSLPP  Version 1.1  <Build 4>
>>            Preprocessor Object: SF_DNS  Version 1.1  <Build 4>
>>            Preprocessor Object: SF_SSH  Version 1.1  <Build 3>
>>            Preprocessor Object: SF_SMTP  Version 1.1  <Build 9>
>>            Preprocessor Object: SF_IMAP  Version 1.0  <Build 1>
>>            Preprocessor Object: SF_POP  Version 1.0  <Build 1>
>>            Preprocessor Object: SF_FTPTELNET  Version 1.2  <Build 13>
>> Commencing packet processing (pid=4050)
>> ===============================================================================
>> Run time for packet processing was 368.552024 seconds
>> Snort processed 643495 packets.
>> Snort ran for 0 days 0 hours 6 minutes 8 seconds
>>    Pkts/min:       107249
>>    Pkts/sec:         1748
>> ===============================================================================
>> Packet Performance Summary:
>>    max packet time       : 10000 usecs
>>    packet events         : 0
>>    avg pkt time          : 8.95128 usecs
>> Rule Performance Summary:
>>    max rule time         : 4096 usecs
>>    rule events           : 0
>>    avg rule time         : 1.96408 usecs
>> ===============================================================================
>> Packet I/O Totals:
>>    Received:      2121798
>>    Analyzed:       643495 ( 30.328%)
>>     Dropped:       618918 ( 22.582%)
>>    Filtered:            0 (  0.000%)
>> Outstanding:      1478303 ( 69.672%)
>>    Injected:            0
>> ===============================================================================
>>
>> And here they are some statistics outputs:
>>
>> top output:
>>
>> load averages:  0.18,  0.18,  0.13
>> 21 processes: 20 idle, 1 on processor
>> CPU0 states:  0.0% user,  0.0% nice,  0.0% system,  0.6% interrupt, 99.4% idle
>> CPU1 states:  1.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.0% idle
>> CPU2 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
>> CPU3 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
>> Memory: Real: 506M/2942M act/tot Free: 5018M Cache: 2319M Swap: 0K/6142M
>>
>>   PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU COMMAND
>>  4050 root       4    0  417M  151M sleep/0   bpf       0:06  2.83% snort
>> 31522 root      10    0  356M  328M sleep/0   nanosle  35:09  0.73% suricata
>>
>> Status interfaces (systat ifstat):
>>
>>     2 users    Load 0.17 0.17 0.13
>>
>> IFACE            STATE  DESC
>>                      IPKTS   IBYTES    IERRS    OPKTS   OBYTES
>> OERRS    COLLS
>> em0              up
>>                          1       72        0        0       71
>> 0        0
>> em1              up
>>                          0        0        0        0        0
>> 0        0
>> em2              up
>>                          0        0        0        0        0
>> 0        0
>> em3              up
>>                          1      334        0        2     1732
>> 0        0
>> em4              up
>>                       6773  4771755        0        0        0
>> 0        0
>> em5              up
>>                       6773  4771755        0        0        0
>> 0        0
>>
>> Interfaces configruation options:
>>
>> em4: flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
>> mtu 1514
>>         lladdr 00:50:56:17:fc:cd
>>         priority: 0
>>         media: Ethernet autoselect (1000baseT full-duplex,master)
>>         status: active
>>         inet6 fe80::250:56ff:fe17:fccd%em4 prefixlen 64 scopeid 0x5
>> em5: flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
>> mtu 1514
>>         lladdr 00:50:56:18:79:51
>>         priority: 0
>>         media: Ethernet autoselect (1000baseT full-duplex,master)
>>         status: active
>>         inet6 fe80::250:56ff:fe18:7951%em5 prefixlen 64 scopeid 0x6
>>
>> Checking mbuffers (systat mbufs):
>>
>>     2 users    Load 0.18 0.17 0.13
>>
>> IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
>> System                    0   256   361          59
>>                                2k   351         457
>> lo0
>> em0                            2k     7     4   256     7
>> em1                            2k     6     4   256     6
>> em2                            2k     4     4   256     4
>> em3                            2k     7     4   256     7
>> em4                            2k    69     4   256    69
>> em5                            2k    73     4   256    73
>>
>> sysctl.conf options enabled:
>> kern.bufcachepercent=80
>> net.bpf.bufsize=65536
>> net.bpf.maxbufsize=2097152
>>
>> Ok, now some Suricata stats:
>>
>> 5/6/2013 -- 14:29:09 - <Info> - stream "max-sessions": 262144
>> 5/6/2013 -- 14:29:09 - <Info> - stream "prealloc-sessions": 32768
>> 5/6/2013 -- 14:29:09 - <Info> - stream "memcap": 33554432
>> 5/6/2013 -- 14:29:09 - <Info> - stream "midstream" session pickups: disabled
>> 5/6/2013 -- 14:29:09 - <Info> - stream "async-oneside": disabled
>> 5/6/2013 -- 14:29:09 - <Info> - stream "checksum-validation": enabled
>> 5/6/2013 -- 14:29:09 - <Info> - stream."inline": disabled
>> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "memcap": 67108864
>> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "depth": 1048576
>> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "toserver-chunk-size": 2560
>> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "toclient-chunk-size": 2560
>> 5/6/2013 -- 14:29:09 - <Info> - all 7 packet processing threads, 3
>> management threads initialized, engine started.
>> 5/6/2013 -- 14:29:13 - <Info> - No packets with invalid checksum,
>> assuming checksum offloading is NOT used
>> 6/6/2013 -- 06:27:20 - <Info> - Signal Received.  Stopping engine.
>> 6/6/2013 -- 06:27:20 - <Info> - 0 new flows, 0 established flows were
>> timed out, 0 flows in closed state
>> 6/6/2013 -- 06:27:21 - <Info> - time elapsed 57492.090s
>> 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Packets 7317731, bytes 5500227933
>> 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Pcap Total:2974089715
>> Recv:2974089715 Drop:0 (0.0%).
>> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Total flow handler queues - 6
>> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 0  - pkts: 2754800
>> flows: 19921
>> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 1  - pkts: 2158501
>> flows: 15267
>> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 2  - pkts: 1276685
>> flows: 9932
>> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 3  - pkts: 678835
>> flows: 5843
>> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 4  - pkts: 358226
>> flows: 3109
>> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 5  - pkts: 127487
>> flows: 1488
>>
>> -------------------------------------------------------------------
>> Date: 6/6/2013 -- 06:27:21 (uptime: 0d, 15h 58m 44s)
>> -------------------------------------------------------------------
>> Counter                   | TM Name                   | Value
>> -------------------------------------------------------------------
>> capture.kernel_packets    | RxPcapem51                | 2974088271
>> capture.kernel_drops      | RxPcapem51                | 0
>> capture.kernel_ifdrops    | RxPcapem51                | 0
>> decoder.pkts              | RxPcapem51                | 7317731
>> decoder.bytes             | RxPcapem51                | 5500227933
>> decoder.ipv4              | RxPcapem51                | 7317731
>> decoder.ipv6              | RxPcapem51                | 0
>> decoder.ethernet          | RxPcapem51                | 7317731
>> decoder.raw               | RxPcapem51                | 0
>> decoder.sll               | RxPcapem51                | 0
>> decoder.tcp               | RxPcapem51                | 7317660
>> decoder.udp               | RxPcapem51                | 0
>> decoder.sctp              | RxPcapem51                | 0
>> decoder.icmpv4            | RxPcapem51                | 71
>> decoder.icmpv6            | RxPcapem51                | 0
>> decoder.ppp               | RxPcapem51                | 0
>> decoder.pppoe             | RxPcapem51                | 0
>> decoder.gre               | RxPcapem51                | 0
>> decoder.vlan              | RxPcapem51                | 0
>> decoder.teredo            | RxPcapem51                | 0
>> decoder.ipv4_in_ipv6      | RxPcapem51                | 0
>> decoder.ipv6_in_ipv6      | RxPcapem51                | 0
>> decoder.avg_pkt_size      | RxPcapem51                | 752
>> decoder.max_pkt_size      | RxPcapem51                | 1506
>> defrag.ipv4.fragments     | RxPcapem51                | 0
>> defrag.ipv4.reassembled   | RxPcapem51                | 0
>> defrag.ipv4.timeouts      | RxPcapem51                | 0
>> defrag.ipv6.fragments     | RxPcapem51                | 0
>> defrag.ipv6.reassembled   | RxPcapem51                | 0
>> defrag.ipv6.timeouts      | RxPcapem51                | 0
>> defrag.max_frag_hits      | RxPcapem51                | 0
>> tcp.sessions              | Detect                    | 41460
>> tcp.ssn_memcap_drop       | Detect                    | 0
>> tcp.pseudo                | Detect                    | 5705
>> tcp.invalid_checksum      | Detect                    | 12
>> tcp.no_flow               | Detect                    | 0
>> tcp.reused_ssn            | Detect                    | 0
>> tcp.memuse                | Detect                    | 36175872
>> tcp.syn                   | Detect                    | 80827
>> tcp.synack                | Detect                    | 80035
>> tcp.rst                   | Detect                    | 33845
>> tcp.segment_memcap_drop   | Detect                    | 0
>> tcp.stream_depth_reached  | Detect                    | 148
>> tcp.reassembly_memuse     | Detect                    | 67770240
>> tcp.reassembly_gap        | Detect                    | 2222
>> detect.alert              | Detect                    | 29
>> flow_mgr.closed_pruned    | FlowManagerThread         | 49711
>> flow_mgr.new_pruned       | FlowManagerThread         | 3558
>> flow_mgr.est_pruned       | FlowManagerThread         | 1974
>> flow.memuse               | FlowManagerThread         | 3884096
>> flow.spare                | FlowManagerThread         | 10001
>> flow.emerg_mode_entered   | FlowManagerThread         | 0
>> flow.emerg_mode_over      | FlowManagerThread         | 0
>>
>> As you can see, packets dropped counter shows 0% (well, in fact this
>> is not always the case. Suricata may lost between 1-2% of packets).
>> And this suricata sensor is configured to use IP based rules, when
>> snort not.
>> But there are two counters that can shows if some problems exists:
>> tcp.reassembly_gap and tcp.invalid_checksum. But as you can see, are
>> minimal according to Suricata docs:
>>
>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Statistics
>>
>> When I say: "OpenBSD tunes them "on the fly" according to network
>> load", I would say that OpenBSD 5.1 and later will dynamically adjust
>> the TCP send and receive window sizes. Please, refer to:
>>
>> http://marc.info/?l=openbsd-misc&m=130804020930481&w=2
>> http://marc.info/?l=openbsd-misc&m=128904951710762&w=2
>> http://www.openbsd.org/faq/faq6.html#Tuning
>> http://quigon.bsws.de/papers/2013/AsiaBSDcon/cksums.pdf
>>
>> Do you need to add more info or perform more tests?
>>
>> And many thanks for your help.
>
>
> Ok, more info. I have disabled SMP kernel version and changed by
> uni-processor kernel (bsd) and now performance has grown: packets
> dropped are between 5-20% now ...
>
> Maybe all my problems are with system interrupts??

One question: it could be possible to use libpcap provided by OpenBSD
instead of install from www.tcpdump.org's site??

For example, I am trying to build daq against OpenBSD's libpcap and:

checking libnetfilter_queue/libnetfilter_queue.h usability... no
checking libnetfilter_queue/libnetfilter_queue.h presence... no
checking for libnetfilter_queue/libnetfilter_queue.h... no
checking for linux/netfilter.h... (cached) no
checking for pcap.h... (cached) yes
checking for pcap_lib_version... checking for pcap_lib_version in
-lpcap... (cached) yes
checking for libpcap version >= "1.0.0"... no

    ERROR!  Libpcap library version >= 1.0.0  not found.
    Get it from http://www.tcpdump.org

Suricata builds without problems using libpcap provided by OpenBSD ...

I have attached config.log.

Thanks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: application/octet-stream
Size: 41866 bytes
Desc: not available
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20130612/b501354a/attachment.obj>


More information about the Snort-users mailing list