[Snort-users] Poor performance with Snort 2.9.4.6 under OpenBSD 5.3

C. L. Martinez carlopmart at ...11827...
Fri Jun 7 06:36:17 EDT 2013


On Thu, Jun 6, 2013 at 7:36 AM, C. L. Martinez <carlopmart at ...11827...> wrote:
> On Wed, Jun 5, 2013 at 5:08 PM, Victor Roemer <vroemer at ...1935...> wrote:
>> Martinez, as Joel already mentioned, we'll want to see your Snort
>> configuration. Shutdown stats would also be useful, but perfmon data would
>> be better; if those can be provided.
>>
>> You mentioned that OpenBSD configured the network sysctl parameters "on the
>> fly"; could you direct us to some documentation about this?
>>
>> You also mentioned that Snort was listening on em3, however the startup
>> information in your email indicates that Snort is listening on em4, could
>> you elaborate on this setup?
>>
>>
>> Regarding Suricata, I personally do not have any experience in deploying or
>> configuring it. That said, are you using, relatively, the same
>> configurations? (e.g., any rules enabled, acquiring packets via libpcap,
>> etc..)
>>
>> Also, why are "tcp.reassembly_gap" and "tcp.invalid_checksum" relevant?
>>
>
> Ok, first of all sorry for this long post. I will try to answer to all
> questions. First, my environment:
>
> Host: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 8 GiB RAM
> Snort: release 2.9.4.6 compiled against libpcap 1.3.0
> Suricata: release 1.4.2 compiled against libpcap 1.3.0
>
> Snort is listening in em4 interface and suricata in em5 (there was a
> typo error previously).
>
> Performance data using my actual snort.conf (with few rules and
> without so_rules or IP based rules):
>
> Packet Performance Monitor Config:
>   ticks per usec  : 2405 ticks
>   max packet time : 10000 usecs
>   packet action   : fastpath-expensive-packets
>   packet logging  : log
>   debug-pkts      : disabled
>
> Rule Performance Monitor Config:
>   ticks per usec  : 2405 ticks
>   max rule time   : 4096 usecs
>   rule action     : suspend-expensive-rules
>   rule threshold  : 5
>   suspend timeout : 10 secs
>   rule logging    : log
> pcap DAQ configured to passive.
> Acquiring network traffic from "em4".
> Reload thread starting...
> Reload thread started, thread 0x1ee8c530a500 (4050)
> Decoding Ethernet
>
>         --== Initialization Complete ==--
>
>    ,,_     -*> Snort! <*-
>   o"  )~   Version 2.9.4.6 GRE (Build 73)
>    ''''    By Martin Roesch & The Snort Team:
> http://www.snort.org/snort/snort-team
>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>            Using libpcap version 1.3.0
>            Using PCRE version: 8.31 2012-07-06
>            Using ZLIB version: 1.2.3
>
>            Rules Engine: SF_SNORT_DETECTION_ENGINE  Version 1.17  <Build 18>
>            Preprocessor Object: SF_DNP3  Version 1.1  <Build 1>
>            Preprocessor Object: SF_MODBUS  Version 1.1  <Build 1>
>            Preprocessor Object: SF_GTP  Version 1.1  <Build 1>
>            Preprocessor Object: SF_REPUTATION  Version 1.1  <Build 1>
>            Preprocessor Object: SF_SIP  Version 1.1  <Build 1>
>            Preprocessor Object: SF_SDF  Version 1.1  <Build 1>
>            Preprocessor Object: SF_DCERPC2  Version 1.0  <Build 3>
>            Preprocessor Object: SF_SSLPP  Version 1.1  <Build 4>
>            Preprocessor Object: SF_DNS  Version 1.1  <Build 4>
>            Preprocessor Object: SF_SSH  Version 1.1  <Build 3>
>            Preprocessor Object: SF_SMTP  Version 1.1  <Build 9>
>            Preprocessor Object: SF_IMAP  Version 1.0  <Build 1>
>            Preprocessor Object: SF_POP  Version 1.0  <Build 1>
>            Preprocessor Object: SF_FTPTELNET  Version 1.2  <Build 13>
> Commencing packet processing (pid=4050)
> ===============================================================================
> Run time for packet processing was 368.552024 seconds
> Snort processed 643495 packets.
> Snort ran for 0 days 0 hours 6 minutes 8 seconds
>    Pkts/min:       107249
>    Pkts/sec:         1748
> ===============================================================================
> Packet Performance Summary:
>    max packet time       : 10000 usecs
>    packet events         : 0
>    avg pkt time          : 8.95128 usecs
> Rule Performance Summary:
>    max rule time         : 4096 usecs
>    rule events           : 0
>    avg rule time         : 1.96408 usecs
> ===============================================================================
> Packet I/O Totals:
>    Received:      2121798
>    Analyzed:       643495 ( 30.328%)
>     Dropped:       618918 ( 22.582%)
>    Filtered:            0 (  0.000%)
> Outstanding:      1478303 ( 69.672%)
>    Injected:            0
> ===============================================================================
>
> And here they are some statistics outputs:
>
> top output:
>
> load averages:  0.18,  0.18,  0.13
> 21 processes: 20 idle, 1 on processor
> CPU0 states:  0.0% user,  0.0% nice,  0.0% system,  0.6% interrupt, 99.4% idle
> CPU1 states:  1.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.0% idle
> CPU2 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> CPU3 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> Memory: Real: 506M/2942M act/tot Free: 5018M Cache: 2319M Swap: 0K/6142M
>
>   PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU COMMAND
>  4050 root       4    0  417M  151M sleep/0   bpf       0:06  2.83% snort
> 31522 root      10    0  356M  328M sleep/0   nanosle  35:09  0.73% suricata
>
> Status interfaces (systat ifstat):
>
>     2 users    Load 0.17 0.17 0.13
>
> IFACE            STATE  DESC
>                      IPKTS   IBYTES    IERRS    OPKTS   OBYTES
> OERRS    COLLS
> em0              up
>                          1       72        0        0       71
> 0        0
> em1              up
>                          0        0        0        0        0
> 0        0
> em2              up
>                          0        0        0        0        0
> 0        0
> em3              up
>                          1      334        0        2     1732
> 0        0
> em4              up
>                       6773  4771755        0        0        0
> 0        0
> em5              up
>                       6773  4771755        0        0        0
> 0        0
>
> Interfaces configruation options:
>
> em4: flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> mtu 1514
>         lladdr 00:50:56:17:fc:cd
>         priority: 0
>         media: Ethernet autoselect (1000baseT full-duplex,master)
>         status: active
>         inet6 fe80::250:56ff:fe17:fccd%em4 prefixlen 64 scopeid 0x5
> em5: flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> mtu 1514
>         lladdr 00:50:56:18:79:51
>         priority: 0
>         media: Ethernet autoselect (1000baseT full-duplex,master)
>         status: active
>         inet6 fe80::250:56ff:fe18:7951%em5 prefixlen 64 scopeid 0x6
>
> Checking mbuffers (systat mbufs):
>
>     2 users    Load 0.18 0.17 0.13
>
> IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> System                    0   256   361          59
>                                2k   351         457
> lo0
> em0                            2k     7     4   256     7
> em1                            2k     6     4   256     6
> em2                            2k     4     4   256     4
> em3                            2k     7     4   256     7
> em4                            2k    69     4   256    69
> em5                            2k    73     4   256    73
>
> sysctl.conf options enabled:
> kern.bufcachepercent=80
> net.bpf.bufsize=65536
> net.bpf.maxbufsize=2097152
>
> Ok, now some Suricata stats:
>
> 5/6/2013 -- 14:29:09 - <Info> - stream "max-sessions": 262144
> 5/6/2013 -- 14:29:09 - <Info> - stream "prealloc-sessions": 32768
> 5/6/2013 -- 14:29:09 - <Info> - stream "memcap": 33554432
> 5/6/2013 -- 14:29:09 - <Info> - stream "midstream" session pickups: disabled
> 5/6/2013 -- 14:29:09 - <Info> - stream "async-oneside": disabled
> 5/6/2013 -- 14:29:09 - <Info> - stream "checksum-validation": enabled
> 5/6/2013 -- 14:29:09 - <Info> - stream."inline": disabled
> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "memcap": 67108864
> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "depth": 1048576
> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "toserver-chunk-size": 2560
> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "toclient-chunk-size": 2560
> 5/6/2013 -- 14:29:09 - <Info> - all 7 packet processing threads, 3
> management threads initialized, engine started.
> 5/6/2013 -- 14:29:13 - <Info> - No packets with invalid checksum,
> assuming checksum offloading is NOT used
> 6/6/2013 -- 06:27:20 - <Info> - Signal Received.  Stopping engine.
> 6/6/2013 -- 06:27:20 - <Info> - 0 new flows, 0 established flows were
> timed out, 0 flows in closed state
> 6/6/2013 -- 06:27:21 - <Info> - time elapsed 57492.090s
> 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Packets 7317731, bytes 5500227933
> 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Pcap Total:2974089715
> Recv:2974089715 Drop:0 (0.0%).
> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Total flow handler queues - 6
> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 0  - pkts: 2754800
> flows: 19921
> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 1  - pkts: 2158501
> flows: 15267
> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 2  - pkts: 1276685
> flows: 9932
> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 3  - pkts: 678835
> flows: 5843
> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 4  - pkts: 358226
> flows: 3109
> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 5  - pkts: 127487
> flows: 1488
>
> -------------------------------------------------------------------
> Date: 6/6/2013 -- 06:27:21 (uptime: 0d, 15h 58m 44s)
> -------------------------------------------------------------------
> Counter                   | TM Name                   | Value
> -------------------------------------------------------------------
> capture.kernel_packets    | RxPcapem51                | 2974088271
> capture.kernel_drops      | RxPcapem51                | 0
> capture.kernel_ifdrops    | RxPcapem51                | 0
> decoder.pkts              | RxPcapem51                | 7317731
> decoder.bytes             | RxPcapem51                | 5500227933
> decoder.ipv4              | RxPcapem51                | 7317731
> decoder.ipv6              | RxPcapem51                | 0
> decoder.ethernet          | RxPcapem51                | 7317731
> decoder.raw               | RxPcapem51                | 0
> decoder.sll               | RxPcapem51                | 0
> decoder.tcp               | RxPcapem51                | 7317660
> decoder.udp               | RxPcapem51                | 0
> decoder.sctp              | RxPcapem51                | 0
> decoder.icmpv4            | RxPcapem51                | 71
> decoder.icmpv6            | RxPcapem51                | 0
> decoder.ppp               | RxPcapem51                | 0
> decoder.pppoe             | RxPcapem51                | 0
> decoder.gre               | RxPcapem51                | 0
> decoder.vlan              | RxPcapem51                | 0
> decoder.teredo            | RxPcapem51                | 0
> decoder.ipv4_in_ipv6      | RxPcapem51                | 0
> decoder.ipv6_in_ipv6      | RxPcapem51                | 0
> decoder.avg_pkt_size      | RxPcapem51                | 752
> decoder.max_pkt_size      | RxPcapem51                | 1506
> defrag.ipv4.fragments     | RxPcapem51                | 0
> defrag.ipv4.reassembled   | RxPcapem51                | 0
> defrag.ipv4.timeouts      | RxPcapem51                | 0
> defrag.ipv6.fragments     | RxPcapem51                | 0
> defrag.ipv6.reassembled   | RxPcapem51                | 0
> defrag.ipv6.timeouts      | RxPcapem51                | 0
> defrag.max_frag_hits      | RxPcapem51                | 0
> tcp.sessions              | Detect                    | 41460
> tcp.ssn_memcap_drop       | Detect                    | 0
> tcp.pseudo                | Detect                    | 5705
> tcp.invalid_checksum      | Detect                    | 12
> tcp.no_flow               | Detect                    | 0
> tcp.reused_ssn            | Detect                    | 0
> tcp.memuse                | Detect                    | 36175872
> tcp.syn                   | Detect                    | 80827
> tcp.synack                | Detect                    | 80035
> tcp.rst                   | Detect                    | 33845
> tcp.segment_memcap_drop   | Detect                    | 0
> tcp.stream_depth_reached  | Detect                    | 148
> tcp.reassembly_memuse     | Detect                    | 67770240
> tcp.reassembly_gap        | Detect                    | 2222
> detect.alert              | Detect                    | 29
> flow_mgr.closed_pruned    | FlowManagerThread         | 49711
> flow_mgr.new_pruned       | FlowManagerThread         | 3558
> flow_mgr.est_pruned       | FlowManagerThread         | 1974
> flow.memuse               | FlowManagerThread         | 3884096
> flow.spare                | FlowManagerThread         | 10001
> flow.emerg_mode_entered   | FlowManagerThread         | 0
> flow.emerg_mode_over      | FlowManagerThread         | 0
>
> As you can see, packets dropped counter shows 0% (well, in fact this
> is not always the case. Suricata may lost between 1-2% of packets).
> And this suricata sensor is configured to use IP based rules, when
> snort not.
> But there are two counters that can shows if some problems exists:
> tcp.reassembly_gap and tcp.invalid_checksum. But as you can see, are
> minimal according to Suricata docs:
>
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Statistics
>
> When I say: "OpenBSD tunes them "on the fly" according to network
> load", I would say that OpenBSD 5.1 and later will dynamically adjust
> the TCP send and receive window sizes. Please, refer to:
>
> http://marc.info/?l=openbsd-misc&m=130804020930481&w=2
> http://marc.info/?l=openbsd-misc&m=128904951710762&w=2
> http://www.openbsd.org/faq/faq6.html#Tuning
> http://quigon.bsws.de/papers/2013/AsiaBSDcon/cksums.pdf
>
> Do you need to add more info or perform more tests?
>
> And many thanks for your help.


Ok, more info. I have disabled SMP kernel version and changed by
uni-processor kernel (bsd) and now performance has grown: packets
dropped are between 5-20% now ...

Maybe all my problems are with system interrupts??




More information about the Snort-users mailing list