[Snort-users] Poor performance with Snort 2.9.4.6 under OpenBSD 5.3

Victor Roemer vroemer at ...1935...
Wed Jun 12 12:15:42 EDT 2013


First, I would recommend you tune your policy down. I believe for most
environments that this is not a
quick process; something to keep in mind.

Things to think about

preprocessor pop ...
preprocessor imap ...
preprocessor smtp ...

are each configured with

    b64_decode_depth 0
    qp_decode_depth 0
    bitenc_decode_depth 0
    uu_decode_depth 0

which permits unlimited decoding of the specified encoding. If your seeing
alot of email traffic,
you may do better to assign a fixed decoding depth.

Choosing an appropriate value that provides sufficient detection and
performance is where tuning gets tricky.

Similarly, if you don't need these preproc's you could also consider
disabling them.
(e.g., little traffic ever seen; or maybe you're rule-set doesn't benefit
with them enabled).


-------

preprocessor ssh ...

Without the appropriate preproc rules enabled, this preprocessor is not
likely to be doing anything for you.

You're also using the "autodetect" keyword in this preprocessor, which is a
performance
consideration as it will cause the preproc to look at traffic on all tcp
ports.


--------


preprocessor http_inspect_server:
...
    server_flow_depth 0

    client_flow_depth 0

Saving the best for last, HTTP Inspect will normally analyze the most
traffic of all the
application-level preprocs. The included options are easily the best place
to start tuning here.


--------

Also curious why you have 2 output-plugins configured, this is not helping
performance either.
Its recommended to use unified2, especially if you want to be fast.





Another thing to consider, and this should directly relate to Suricata
getting better numbers,
is load-balancing across multiple Snort processes.

Though I may be mistaken, Suricata should be doing across multiple threads,
rather than processes. (Anyone care to chime in?)


On Wed, Jun 12, 2013 at 3:08 AM, C. L. Martinez <carlopmart at ...11827...>wrote:

> On Fri, Jun 7, 2013 at 10:36 AM, C. L. Martinez <carlopmart at ...11827...>
> wrote:
> > On Thu, Jun 6, 2013 at 7:36 AM, C. L. Martinez <carlopmart at ...11827...>
> wrote:
> >> On Wed, Jun 5, 2013 at 5:08 PM, Victor Roemer <vroemer at ...1935...>
> wrote:
> >>> Martinez, as Joel already mentioned, we'll want to see your Snort
> >>> configuration. Shutdown stats would also be useful, but perfmon data
> would
> >>> be better; if those can be provided.
> >>>
> >>> You mentioned that OpenBSD configured the network sysctl parameters
> "on the
> >>> fly"; could you direct us to some documentation about this?
> >>>
> >>> You also mentioned that Snort was listening on em3, however the startup
> >>> information in your email indicates that Snort is listening on em4,
> could
> >>> you elaborate on this setup?
> >>>
> >>>
> >>> Regarding Suricata, I personally do not have any experience in
> deploying or
> >>> configuring it. That said, are you using, relatively, the same
> >>> configurations? (e.g., any rules enabled, acquiring packets via
> libpcap,
> >>> etc..)
> >>>
> >>> Also, why are "tcp.reassembly_gap" and "tcp.invalid_checksum" relevant?
> >>>
> >>
> >> Ok, first of all sorry for this long post. I will try to answer to all
> >> questions. First, my environment:
> >>
> >> Host: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 8 GiB RAM
> >> Snort: release 2.9.4.6 compiled against libpcap 1.3.0
> >> Suricata: release 1.4.2 compiled against libpcap 1.3.0
> >>
> >> Snort is listening in em4 interface and suricata in em5 (there was a
> >> typo error previously).
> >>
> >> Performance data using my actual snort.conf (with few rules and
> >> without so_rules or IP based rules):
> >>
> >> Packet Performance Monitor Config:
> >>   ticks per usec  : 2405 ticks
> >>   max packet time : 10000 usecs
> >>   packet action   : fastpath-expensive-packets
> >>   packet logging  : log
> >>   debug-pkts      : disabled
> >>
> >> Rule Performance Monitor Config:
> >>   ticks per usec  : 2405 ticks
> >>   max rule time   : 4096 usecs
> >>   rule action     : suspend-expensive-rules
> >>   rule threshold  : 5
> >>   suspend timeout : 10 secs
> >>   rule logging    : log
> >> pcap DAQ configured to passive.
> >> Acquiring network traffic from "em4".
> >> Reload thread starting...
> >> Reload thread started, thread 0x1ee8c530a500 (4050)
> >> Decoding Ethernet
> >>
> >>         --== Initialization Complete ==--
> >>
> >>    ,,_     -*> Snort! <*-
> >>   o"  )~   Version 2.9.4.6 GRE (Build 73)
> >>    ''''    By Martin Roesch & The Snort Team:
> >> http://www.snort.org/snort/snort-team
> >>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
> >>            Using libpcap version 1.3.0
> >>            Using PCRE version: 8.31 2012-07-06
> >>            Using ZLIB version: 1.2.3
> >>
> >>            Rules Engine: SF_SNORT_DETECTION_ENGINE  Version 1.17
>  <Build 18>
> >>            Preprocessor Object: SF_DNP3  Version 1.1  <Build 1>
> >>            Preprocessor Object: SF_MODBUS  Version 1.1  <Build 1>
> >>            Preprocessor Object: SF_GTP  Version 1.1  <Build 1>
> >>            Preprocessor Object: SF_REPUTATION  Version 1.1  <Build 1>
> >>            Preprocessor Object: SF_SIP  Version 1.1  <Build 1>
> >>            Preprocessor Object: SF_SDF  Version 1.1  <Build 1>
> >>            Preprocessor Object: SF_DCERPC2  Version 1.0  <Build 3>
> >>            Preprocessor Object: SF_SSLPP  Version 1.1  <Build 4>
> >>            Preprocessor Object: SF_DNS  Version 1.1  <Build 4>
> >>            Preprocessor Object: SF_SSH  Version 1.1  <Build 3>
> >>            Preprocessor Object: SF_SMTP  Version 1.1  <Build 9>
> >>            Preprocessor Object: SF_IMAP  Version 1.0  <Build 1>
> >>            Preprocessor Object: SF_POP  Version 1.0  <Build 1>
> >>            Preprocessor Object: SF_FTPTELNET  Version 1.2  <Build 13>
> >> Commencing packet processing (pid=4050)
> >>
> ===============================================================================
> >> Run time for packet processing was 368.552024 seconds
> >> Snort processed 643495 packets.
> >> Snort ran for 0 days 0 hours 6 minutes 8 seconds
> >>    Pkts/min:       107249
> >>    Pkts/sec:         1748
> >>
> ===============================================================================
> >> Packet Performance Summary:
> >>    max packet time       : 10000 usecs
> >>    packet events         : 0
> >>    avg pkt time          : 8.95128 usecs
> >> Rule Performance Summary:
> >>    max rule time         : 4096 usecs
> >>    rule events           : 0
> >>    avg rule time         : 1.96408 usecs
> >>
> ===============================================================================
> >> Packet I/O Totals:
> >>    Received:      2121798
> >>    Analyzed:       643495 ( 30.328%)
> >>     Dropped:       618918 ( 22.582%)
> >>    Filtered:            0 (  0.000%)
> >> Outstanding:      1478303 ( 69.672%)
> >>    Injected:            0
> >>
> ===============================================================================
> >>
> >> And here they are some statistics outputs:
> >>
> >> top output:
> >>
> >> load averages:  0.18,  0.18,  0.13
> >> 21 processes: 20 idle, 1 on processor
> >> CPU0 states:  0.0% user,  0.0% nice,  0.0% system,  0.6% interrupt,
> 99.4% idle
> >> CPU1 states:  1.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,
> 99.0% idle
> >> CPU2 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,
>  100% idle
> >> CPU3 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,
>  100% idle
> >> Memory: Real: 506M/2942M act/tot Free: 5018M Cache: 2319M Swap: 0K/6142M
> >>
> >>   PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU
> COMMAND
> >>  4050 root       4    0  417M  151M sleep/0   bpf       0:06  2.83%
> snort
> >> 31522 root      10    0  356M  328M sleep/0   nanosle  35:09  0.73%
> suricata
> >>
> >> Status interfaces (systat ifstat):
> >>
> >>     2 users    Load 0.17 0.17 0.13
> >>
> >> IFACE            STATE  DESC
> >>                      IPKTS   IBYTES    IERRS    OPKTS   OBYTES
> >> OERRS    COLLS
> >> em0              up
> >>                          1       72        0        0       71
> >> 0        0
> >> em1              up
> >>                          0        0        0        0        0
> >> 0        0
> >> em2              up
> >>                          0        0        0        0        0
> >> 0        0
> >> em3              up
> >>                          1      334        0        2     1732
> >> 0        0
> >> em4              up
> >>                       6773  4771755        0        0        0
> >> 0        0
> >> em5              up
> >>                       6773  4771755        0        0        0
> >> 0        0
> >>
> >> Interfaces configruation options:
> >>
> >> em4:
> flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> >> mtu 1514
> >>         lladdr 00:50:56:17:fc:cd
> >>         priority: 0
> >>         media: Ethernet autoselect (1000baseT full-duplex,master)
> >>         status: active
> >>         inet6 fe80::250:56ff:fe17:fccd%em4 prefixlen 64 scopeid 0x5
> >> em5:
> flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> >> mtu 1514
> >>         lladdr 00:50:56:18:79:51
> >>         priority: 0
> >>         media: Ethernet autoselect (1000baseT full-duplex,master)
> >>         status: active
> >>         inet6 fe80::250:56ff:fe18:7951%em5 prefixlen 64 scopeid 0x6
> >>
> >> Checking mbuffers (systat mbufs):
> >>
> >>     2 users    Load 0.18 0.17 0.13
> >>
> >> IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> >> System                    0   256   361          59
> >>                                2k   351         457
> >> lo0
> >> em0                            2k     7     4   256     7
> >> em1                            2k     6     4   256     6
> >> em2                            2k     4     4   256     4
> >> em3                            2k     7     4   256     7
> >> em4                            2k    69     4   256    69
> >> em5                            2k    73     4   256    73
> >>
> >> sysctl.conf options enabled:
> >> kern.bufcachepercent=80
> >> net.bpf.bufsize=65536
> >> net.bpf.maxbufsize=2097152
> >>
> >> Ok, now some Suricata stats:
> >>
> >> 5/6/2013 -- 14:29:09 - <Info> - stream "max-sessions": 262144
> >> 5/6/2013 -- 14:29:09 - <Info> - stream "prealloc-sessions": 32768
> >> 5/6/2013 -- 14:29:09 - <Info> - stream "memcap": 33554432
> >> 5/6/2013 -- 14:29:09 - <Info> - stream "midstream" session pickups:
> disabled
> >> 5/6/2013 -- 14:29:09 - <Info> - stream "async-oneside": disabled
> >> 5/6/2013 -- 14:29:09 - <Info> - stream "checksum-validation": enabled
> >> 5/6/2013 -- 14:29:09 - <Info> - stream."inline": disabled
> >> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "memcap": 67108864
> >> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "depth": 1048576
> >> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly
> "toserver-chunk-size": 2560
> >> 5/6/2013 -- 14:29:09 - <Info> - stream.reassembly
> "toclient-chunk-size": 2560
> >> 5/6/2013 -- 14:29:09 - <Info> - all 7 packet processing threads, 3
> >> management threads initialized, engine started.
> >> 5/6/2013 -- 14:29:13 - <Info> - No packets with invalid checksum,
> >> assuming checksum offloading is NOT used
> >> 6/6/2013 -- 06:27:20 - <Info> - Signal Received.  Stopping engine.
> >> 6/6/2013 -- 06:27:20 - <Info> - 0 new flows, 0 established flows were
> >> timed out, 0 flows in closed state
> >> 6/6/2013 -- 06:27:21 - <Info> - time elapsed 57492.090s
> >> 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Packets 7317731, bytes
> 5500227933
> >> 6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Pcap Total:2974089715
> >> Recv:2974089715 Drop:0 (0.0%).
> >> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Total flow handler queues - 6
> >> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 0  - pkts: 2754800
> >> flows: 19921
> >> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 1  - pkts: 2158501
> >> flows: 15267
> >> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 2  - pkts: 1276685
> >> flows: 9932
> >> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 3  - pkts: 678835
> >> flows: 5843
> >> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 4  - pkts: 358226
> >> flows: 3109
> >> 6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 5  - pkts: 127487
> >> flows: 1488
> >>
> >> -------------------------------------------------------------------
> >> Date: 6/6/2013 -- 06:27:21 (uptime: 0d, 15h 58m 44s)
> >> -------------------------------------------------------------------
> >> Counter                   | TM Name                   | Value
> >> -------------------------------------------------------------------
> >> capture.kernel_packets    | RxPcapem51                | 2974088271
> >> capture.kernel_drops      | RxPcapem51                | 0
> >> capture.kernel_ifdrops    | RxPcapem51                | 0
> >> decoder.pkts              | RxPcapem51                | 7317731
> >> decoder.bytes             | RxPcapem51                | 5500227933
> >> decoder.ipv4              | RxPcapem51                | 7317731
> >> decoder.ipv6              | RxPcapem51                | 0
> >> decoder.ethernet          | RxPcapem51                | 7317731
> >> decoder.raw               | RxPcapem51                | 0
> >> decoder.sll               | RxPcapem51                | 0
> >> decoder.tcp               | RxPcapem51                | 7317660
> >> decoder.udp               | RxPcapem51                | 0
> >> decoder.sctp              | RxPcapem51                | 0
> >> decoder.icmpv4            | RxPcapem51                | 71
> >> decoder.icmpv6            | RxPcapem51                | 0
> >> decoder.ppp               | RxPcapem51                | 0
> >> decoder.pppoe             | RxPcapem51                | 0
> >> decoder.gre               | RxPcapem51                | 0
> >> decoder.vlan              | RxPcapem51                | 0
> >> decoder.teredo            | RxPcapem51                | 0
> >> decoder.ipv4_in_ipv6      | RxPcapem51                | 0
> >> decoder.ipv6_in_ipv6      | RxPcapem51                | 0
> >> decoder.avg_pkt_size      | RxPcapem51                | 752
> >> decoder.max_pkt_size      | RxPcapem51                | 1506
> >> defrag.ipv4.fragments     | RxPcapem51                | 0
> >> defrag.ipv4.reassembled   | RxPcapem51                | 0
> >> defrag.ipv4.timeouts      | RxPcapem51                | 0
> >> defrag.ipv6.fragments     | RxPcapem51                | 0
> >> defrag.ipv6.reassembled   | RxPcapem51                | 0
> >> defrag.ipv6.timeouts      | RxPcapem51                | 0
> >> defrag.max_frag_hits      | RxPcapem51                | 0
> >> tcp.sessions              | Detect                    | 41460
> >> tcp.ssn_memcap_drop       | Detect                    | 0
> >> tcp.pseudo                | Detect                    | 5705
> >> tcp.invalid_checksum      | Detect                    | 12
> >> tcp.no_flow               | Detect                    | 0
> >> tcp.reused_ssn            | Detect                    | 0
> >> tcp.memuse                | Detect                    | 36175872
> >> tcp.syn                   | Detect                    | 80827
> >> tcp.synack                | Detect                    | 80035
> >> tcp.rst                   | Detect                    | 33845
> >> tcp.segment_memcap_drop   | Detect                    | 0
> >> tcp.stream_depth_reached  | Detect                    | 148
> >> tcp.reassembly_memuse     | Detect                    | 67770240
> >> tcp.reassembly_gap        | Detect                    | 2222
> >> detect.alert              | Detect                    | 29
> >> flow_mgr.closed_pruned    | FlowManagerThread         | 49711
> >> flow_mgr.new_pruned       | FlowManagerThread         | 3558
> >> flow_mgr.est_pruned       | FlowManagerThread         | 1974
> >> flow.memuse               | FlowManagerThread         | 3884096
> >> flow.spare                | FlowManagerThread         | 10001
> >> flow.emerg_mode_entered   | FlowManagerThread         | 0
> >> flow.emerg_mode_over      | FlowManagerThread         | 0
> >>
> >> As you can see, packets dropped counter shows 0% (well, in fact this
> >> is not always the case. Suricata may lost between 1-2% of packets).
> >> And this suricata sensor is configured to use IP based rules, when
> >> snort not.
> >> But there are two counters that can shows if some problems exists:
> >> tcp.reassembly_gap and tcp.invalid_checksum. But as you can see, are
> >> minimal according to Suricata docs:
>
>>
> >>
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Statistics
> >>
>
>> When I say: "OpenBSD tunes them "on the fly" according to network
> >> load", I would say that OpenBSD 5.1 and later will dynamically adjust
> >> the TCP send and receive window sizes. Please, refer to:
> >>
> >> http://marc.info/?l=openbsd-misc&m=130804020930481&w=2
> >> http://marc.info/?l=openbsd-misc&m=128904951710762&w=2
> >> http://www.openbsd.org/faq/faq6.html#Tuning
> >> http://quigon.bsws.de/papers/2013/AsiaBSDcon/cksums.pdf
> >>
> >> Do you need to add more info or perform more tests?
> >>
> >> And many thanks for your help.
> >
> >
> > Ok, more info. I have disabled SMP kernel version and changed by
> > uni-processor kernel (bsd) and now performance has grown: packets
> > dropped are between 5-20% now ...
> >
> > Maybe all my problems are with system interrupts??
>

Interesting that this would be a problem, While I'm not well read on SMP, I
thought these kinds of
issues were only prevalent on single core hardware.



>
>
One question: it could be possible to use libpcap provided by OpenBSD
> instead of install from www.tcpdump.org's site??
>
> For example, I am trying to build daq against OpenBSD's libpcap and:
>
> checking libnetfilter_queue/libnetfilter_queue.h usability... no
> checking libnetfilter_queue/libnetfilter_queue.h presence... no
> checking for libnetfilter_queue/libnetfilter_queue.h... no
> checking for linux/netfilter.h... (cached) no
> checking for pcap.h... (cached) yes
> checking for pcap_lib_version... checking for pcap_lib_version in
> -lpcap... (cached) yes
> checking for libpcap version >= "1.0.0"... no
>
>     ERROR!  Libpcap library version >= 1.0.0  not found.
>     Get it from http://www.tcpdump.org
>
> Suricata builds without problems using libpcap provided by OpenBSD ...
>
> I have attached config.log.
>

This is a problem that we are already familiar with; the OpenBSD guys
maintain they're own customized
libpcap which is actually API compatible, but does not report as being
1.0.0 or greater.

Its certainly possible that the libpcap provided by them is faster, but I
really don't know.

>
> Thanks.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20130612/0bff9acb/attachment.html>


More information about the Snort-users mailing list