[Snort-devel] Critical: Memory leak in snort 2.9 and FreeBSD >= 10.4

James sjamek at gmail.com
Fri May 25 02:05:34 EDT 2018


Please unsubscribe

On Thu, 24 May 2018 at 17:12 <elof at sentor.se> wrote:

>
> Hi list (and Zi from FreeBSD ports)!
>
> I sent the below question to snort-users in March but got no response.
>
> Now I have upgraded a couple of more systems. This time from FreeBSD
> 10.3 to 10.4 (not from 10.3 to 11.1 as before) and snort start leaking
> memory on all of them, just as it did in FreeBSD 11.1!
>
>
> FreeBSD 10.3: snort --version
>     ,,_     -*> Snort! <*-
>    o"  )~   Version 2.9.11.1 (Build 268)
>     ''''    By Martin Roesch & The Snort Team:
> http://www.snort.org/contact#team
>             Copyright (C) 2014-2017 Cisco and/or its affiliates. All
> rights
> reserved.
>             Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>             Using libpcap version 1.8.1
>             Using PCRE version: 8.40 2017-01-11
>             Using ZLIB version: 1.2.8
> Snort is working fine. No memory leak. :-)
>
>
> FreeBSD 10.4: snort --version
>     ,,_     -*> Snort! <*-
>    o"  )~   Version 2.9.11.1 (Build 268)
>     ''''    By Martin Roesch & The Snort Team:
> http://www.snort.org/contact#team
>             Copyright (C) 2014-2017 Cisco and/or its affiliates. All
> rights
> reserved.
>             Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>             Using libpcap version 1.8.1
>             Using PCRE version: 8.42 2018-03-20
>             Using ZLIB version: 1.2.11
> Snort has a memory leak. :-(
>
>
> FreeBSD 11.1: snort --version
>    ,,_     -*> Snort! <*-
>   o"  )~   Version 2.9.11.1 (Build 268)
>    ''''    By Martin Roesch & The Snort Team:
> http://www.snort.org/contact#team
>            Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights
> reserved.
>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>            Using libpcap version 1.8.1
>            Using PCRE version: 8.40 2017-01-11
>            Using ZLIB version: 1.2.11
> Snort has a memory leak. :-(
>
>
> The snort version+build is exactly the same in all three OS versions.
> Libpcap is the same.
> PCRE is the same between 10.3 and the test on 11.1.
> ZLIB has changed from v1.2.8 to v1.2.11 in both cases.
>
> So in 10.3 everything is working fine.
> In 10.4 and 11.1, with ZLIB 1.2.11, there is a memory leak.
>
> I suspect there's a problem in snort together with ZLIB 1.2.11.
>
>
>
> On a sensor which see lots of traffic, all of its 16 GB RAM is consumed by
> snort in roughly 30 minutes. (swap get full and things crash)
>
> This problem is reproduceable all the time, on all upgraded boxes.
>
>
>
> Can you developers please take a look at this?
>
> Let me know if you need more information/testing from me.
> I'm no programmer, but I can compile snort with debugging symbols and I
> can run gdb commands if you provide them.
>
>
>
> Some more info:
>
> Arch: amd64
> 12 CPUs: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
> RAM: 16 GB
>
> I'm building snort from FreeBSD ports, using poudriere.
>
> Build options:
> ---Begin OPTIONS List---
> ===> The following configuration options are available for
> snort-2.9.11.1_1:
>       APPID=off: Build with application id support (EXPERIMENTAL)
>       DOCS=on: Build and/or install documentation
>       FILEINSPECT=off: Build with extended file inspection features
> (EXPERIMENTAL)
>       GRE=off: GRE support
>       HA=off: Enable high-availability state sharing (EXPERIMENTAL)
>       IPV6=off: IPv6 in snort.conf
>       LRGPCAP=off: Pcaps larger than 2GB
>       NONETHER=off: Non-Ethernet Decoders
>       NORMALIZER=on: Normalizer
>       PERFPROFILE=on: Performance profiling
>       SOURCEFIRE=on: Sourcefire recommended build options
> ====> Depend on 3rd party addons
>       BARNYARD=off: Depend on barnyard2 (supports also snortsam)
>       PULLEDPORK=off: Depend on pulledpork
> ====> Developer options
>       DBGSNORT=off: Enable debugging symbols+core dumps
> ===> Use 'make config' to modify these settings
> ---End OPTIONS List---
>
> Snort daq is running in pcap mode.
> Snort is running in passive mode.
> Snort is using search-method ac-split.
> preprocessor http_inspect: global iis_unicode_map
> /usr/local/etc/snort/unicode.map 1252 compress_depth 65535
> decompress_depth 65535 max_gzip_mem 1000000 memcap 603979776
> http_inspect: unlimited_decompress is enabled
> http_inspect: inspect_gzip is enabled
>
>
> PS:
> libz is part of the FreeBSD base system, it is not a port, so I can't hold
> it back when upgrading the OS.
>
> /Elof
>
>
>
> ---------- Forwarded message ----------
> From: elof at sentor.se
> To: snort-users mailinglist <snort-users at lists.sourceforge.net>
> Date: Fri, 2 Mar 2018 16:35:24 +0100 (CET)
> Subject: Memory leak in snort 2.9 and FreeBSD 11?
>
>
> Critical issue.
>
>
> After I upgraded a few FreeBSD 10.3 machines to 11.1, snort has begun
> eating
> memory until it crashes.
> This seem to be happening on all upgraded machines, all the time.
>
>
> I suspect there's a memory leak somewhere.
>
>
> Example of 40 minutes after I start snort. I run:
>
> while true
> do
>    ps faxuw | egrep "^USER|/[s]nort "
>    echo "---"
>    top | grep -B3 ^Swap
>    echo "---"
>    sleep 120
> done
>
> Here you see it start to consume RAM:
>                      ####
> USER      PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED      TIME COMMAND
> snort    7337  98.4  1.4  356096 232376  -  Rs   14:40     0:01.35 snort
> ---
> Mem: 550M Active, 174M Inact, 1585M Wired, 13G Free
> ARC: 711M Total, 153M MFU, 545M MRU, 1600K Anon, 4623K Header, 7465K Other
>       593M Compressed, 1647M Uncompressed, 2.78:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED      TIME COMMAND
> snort    7337  54.3  7.2 3002112 1199900  -  Rs   14:40     1:04.85 snort
> ---
> Mem: 1499M Active, 191M Inact, 1670M Wired, 12G Free
> ARC: 763M Total, 178M MFU, 572M MRU, 1308K Anon, 4860K Header, 7441K Other
>       646M Compressed, 1779M Uncompressed, 2.75:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED      TIME COMMAND
> snort    7337  54.3 12.9 5644032 2155388  -  Ss   14:40     2:07.16 snort
> ---
> Mem: 2427M Active, 191M Inact, 1682M Wired, 11G Free
> ARC: 777M Total, 178M MFU, 585M MRU, 1344K Anon, 4935K Header, 7513K Other
>       661M Compressed, 1815M Uncompressed, 2.75:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED      TIME COMMAND
> snort    7337  54.5 18.7 8275712 3114844  -  Rs   14:40     3:09.86 snort
> ---
> Mem: 3357M Active, 192M Inact, 1768M Wired, 10G Free
> ARC: 821M Total, 194M MFU, 614M MRU, 556K Anon, 5195K Header, 7513K Other
>       711M Compressed, 1942M Uncompressed, 2.73:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    7337  54.2 24.3 10862336 4053456  -  Rs   14:40     4:11.28 snort
> ---
> Mem: 4270M Active, 194M Inact, 1778M Wired, 9646M Free
> ARC: 890M Total, 299M MFU, 578M MRU, 400K Anon, 5243K Header, 7442K Other
>       726M Compressed, 1978M Uncompressed, 2.73:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    7337  56.2 29.9 13461248 4998904  -  Ss   14:40     5:13.96 snort
> ---
> Mem: 5188M Active, 195M Inact, 1798M Wired, 8708M Free
> ARC: 826M Total, 261M MFU, 551M MRU, 528K Anon, 5300K Header, 7410K Other
>       741M Compressed, 2015M Uncompressed, 2.72:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    7337  53.1 35.5 16033536 5929068  -  Rs   14:40     6:15.56 snort
> ---
> Mem: 6091M Active, 195M Inact, 1823M Wired, 7779M Free
> ARC: 870M Total, 255M MFU, 602M MRU, 276K Anon, 5391K Header, 7521K Other
>       755M Compressed, 2051M Uncompressed, 2.72:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    7337  51.2 41.1 18605824 6867124  -  Ss   14:40     7:16.95 snort
> ---
> Mem: 7002M Active, 195M Inact, 1848M Wired, 6843M Free
> ARC: 885M Total, 221M MFU, 651M MRU, 288K Anon, 5454K Header, 7515K Other
>       769M Compressed, 2087M Uncompressed, 2.71:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER      PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    7337  54.0 46.8 21212928 7810464  -  Rs   14:40     8:19.24 snort
> ---
> Mem: 7924M Active, 195M Inact, 1943M Wired, 5826M Free
> ARC: 936M Total, 218M MFU, 703M MRU, 952K Anon, 5766K Header, 7829K Other
>       824M Compressed, 2222M Uncompressed, 2.70:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER       PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  53.0 52.5 23834368 8762624  -  Ss   14:40     9:21.14 snort
> ---
> Mem: 8849M Active, 195M Inact, 1954M Wired, 4891M Free
> ARC: 951M Total, 218M MFU, 718M MRU, 920K Anon, 5827K Header, 7814K Other
>       838M Compressed, 2259M Uncompressed, 2.69:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER       PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  50.9 58.2 26472192 9721948  -  Rs   14:40    10:23.33 snort
> ---
> Mem: 9782M Active, 195M Inact, 1971M Wired, 3941M Free
> ARC: 965M Total, 220M MFU, 731M MRU, 920K Anon, 5882K Header, 7822K Other
>       853M Compressed, 2295M Uncompressed, 2.69:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  50.3 63.9 29105920 10676928  -  Rs   14:40    11:26.69
> snort
> ---
> Mem: 10G Active, 204M Inact, 2116M Wired, 2852M Free
> ARC: 1062M Total, 274M MFU, 774M MRU, 1052K Anon, 6252K Header, 7728K Other
>       939M Compressed, 2510M Uncompressed, 2.67:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  55.5 69.6 31735552 11617828  -  Rs   14:40    12:28.81
> snort
> ---
> Mem: 11G Active, 224M Inact, 2132M Wired, 1921M Free
> ARC: 1092M Total, 300M MFU, 777M MRU, 1072K Anon, 6285K Header, 7591K Other
>       954M Compressed, 2547M Uncompressed, 2.67:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  55.1 75.1 34324224 12545340  -  Rs   14:40    13:30.78
> snort
> ---
> Mem: 12G Active, 230M Inact, 2222M Wired, 931M Free
> ARC: 1117M Total, 325M MFU, 777M MRU, 1204K Anon, 6542K Header, 7580K Other
>       1010M Compressed, 2686M Uncompressed, 2.66:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  55.2 80.4 36824832 13427160  -  Ss   14:40    14:32.93
> snort
> ---
> Mem: 12G Active, 263M Inact, 1428M Laundry, 1929M Wired, 367M Free
> ARC: 1156M Total, 325M MFU, 816M MRU, 1068K Anon, 6614K Header, 7613K Other
>       1025M Compressed, 2722M Uncompressed, 2.66:1 Ratio
> Swap: 4096M Total, 4096M Free
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  54.6 82.6 39397120 13800500  -  Rs   14:40    15:35.14
> snort
> ---
> Mem: 12G Active, 110M Inact, 1784M Laundry, 1937M Wired, 191M Free
> ARC: 1159M Total, 323M MFU, 822M MRU, 936K Anon, 6648K Header, 7488K Other
>       1039M Compressed, 2758M Uncompressed, 2.65:1 Ratio
> Swap: 4096M Total, 685M Used, 3411M Free, 16% Inuse
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  53.8 82.6 41801472 13800476  -  Ss   14:40    16:35.38
> snort
> ---
> Mem: 11G Active, 107M Inact, 1846M Laundry, 1990M Wired, 182M Free
> ARC: 1208M Total, 338M MFU, 854M MRU, 1552K Anon, 6866K Header, 7564K Other
>       1087M Compressed, 2878M Uncompressed, 2.65:1 Ratio
> Swap: 4096M Total, 1508M Used, 2587M Free, 36% Inuse
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  54.5 82.2 44414720 13735076  -  Rs   14:40    17:38.62
> snort
> ---
> Mem: 12G Active, 34M Inact, 1766M Laundry, 2014M Wired, 292M Free
> ARC: 1221M Total, 337M MFU, 868M MRU, 1436K Anon, 6944K Header, 7616K Other
>       1102M Compressed, 2914M Uncompressed, 2.64:1 Ratio
> Swap: 4096M Total, 2551M Used, 1544M Free, 62% Inuse
> ---
> USER       PID  %CPU %MEM      VSZ      RSS TT  STAT STARTED      TIME
> COMMAND
> snort     7337  54.6 82.1 46968576 13714884  -  Ss   14:40    18:42.19
> snort
> ---
> Mem: 11G Active, 64M Inact, 1884M Laundry, 2058M Wired, 259M Free
> ARC: 1252M Total, 352M MFU, 885M MRU, 672K Anon, 7150K Header, 7525K Other
>       1153M Compressed, 3043M Uncompressed, 2.64:1 Ratio
> Swap: 4096M Total, 3457M Used, 639M Free, 84% Inuse
> ---
> USER       PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED      TIME COMMAND
> snort     7337   2.2  0.0       0     16  -  R<Es 14:40    19:49.65 snort
> ---
> Mem: 12G Active, 5252K Inact, 786M Laundry, 1937M Wired, 461M Free
> ARC: 1319M Total, 384M MFU, 920M MRU, 967K Anon, 7275K Header, 7534K Other
>       1181M Compressed, 3112M Uncompressed, 2.63:1 Ratio
> Swap: 4096M Total, 4096M Used, K Free, 100% Inuse
> ---
>
>
>
> After maxing out at 82.2% or RAM for a copuple of minutes, the process is
> automatically killed by the system:
>
> Mar  2 15:17:48 chobetsu-10 kernel: swap_pager: out of swap space
> Mar  2 15:17:48 chobetsu-10 kernel: swap_pager_getswapspace(11): failed
> Mar  2 15:20:18 chobetsu-10 kernel: pid 7337 (snort), uid 100, was killed:
> out
> of swap space
>
>
>
>
>
> ...the while-loop continues...
> USER       PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED      TIME COMMAND
> <no snort process started>
> ---
> Mem: 272M Active, 12M Inact, 2079M Wired, 13G Free
> ARC: 1376M Total, 516M MFU, 844M MRU, 1552K Anon, 7347K Header, 7524K Other
>       1197M Compressed, 3150M Uncompressed, 2.63:1 Ratio
> Swap: 4096M Total, 225M Used, 3871M Free, 5% Inuse
> ---
>
>
>
>
> I startup snort again.
> ...the while-loop continues...
>
> USER       PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED      TIME COMMAND
> snort    14277  57.3  5.1 2055936 856740  -  Rs   15:23     0:43.21 snort
> ---
> Mem: 1090M Active, 56M Inact, 2136M Wired, 12G Free
> ARC: 1330M Total, 435M MFU, 879M MRU, 1432K Anon, 7487K Header, 7824K Other
>       1213M Compressed, 3193M Uncompressed, 2.63:1 Ratio
> Swap: 4096M Total, 190M Used, 3906M Free, 4% Inuse
> ---
> USER       PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    14277  57.9 11.0 4714240 1829656  -  Ss   15:23     1:49.20 snort
> ---
> Mem: 2046M Active, 97M Inact, 2366M Wired, 11G Free
> ARC: 1401M Total, 438M MFU, 945M MRU, 1296K Anon, 8041K Header, 8700K Other
>       1314M Compressed, 3438M Uncompressed, 2.62:1 Ratio
> Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
> ---
> USER       PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    14277  54.3 16.8 7362304 2798096  -  Ss   15:23     2:54.43 snort
> ---
> Mem: 2985M Active, 98M Inact, 2378M Wired, 10G Free
> ARC: 1450M Total, 438M MFU, 994M MRU, 1468K Anon, 8120K Header, 8754K Other
>       1329M Compressed, 3475M Uncompressed, 2.62:1 Ratio
> Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
> ---
> USER       PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME
> COMMAND
> snort    14277  60.4 22.6 10053376 3776916  -  Ss   15:23     4:00.53 snort
>
> ...and so on until pid 14277 gets out of swap.
>
>
>
> I'm running:
> FreeBSD 11.1-RELEASE-p4 amd64
> Snort      Version 2.9.11.1 (Build 268)
>             Using libpcap version 1.8.1
>             Using PCRE version: 8.40 2017-01-11
>             Using ZLIB version: 1.2.11
>
>
>
> Has anyone else observed this?
>
> Any tips on how I can help debug this further?
>
> A SIGHUP don't reveal anything about what subsystem is eating memory.
>
>
>
>
> I've tried setting two memcaps to a really low value, to see if the
> process
> stop increasing in size:
>    preprocessor stream5_global: ......... memcap 128257751
>    preprocessor http_inspect: global .... memcap 85505167
> No luck. The snort process grows to >80% of system RAM and then dies.
>
>
>
> Some info about the snort conf:
>
> Running in IDS mode
> Detection: Search-Method = AC-Full-Q
>             Split Any/Any group = enabled
>             Search-Method-Optimizations = enabled
>             Maximum pattern length = 20
>
> +-----------------------[detection-filter-config]------------------------------
>             memory-cap : 1048576 bytes
>
> +-----------------------[rate-filter-config]-----------------------------------
>             memory-cap : 1048576 bytes
>
> +-----------------------[event-filter-config]----------------------------------
>             memory-cap : 1048576 bytes
> Rule application order:
> pass->activation->dynamic->drop->alert->log->sdrop->reject
> pcap DAQ configured to passive.
> chroot
> Set gid to 100
> Set uid to 100
>
>
>
> The same snort version was running just fine on FreeBSD 10.3 before the
> upgrade.
>
>
> /Elof
> _______________________________________________
> Snort-devel mailing list
> Snort-devel at lists.snort.org
> https://lists.snort.org/mailman/listinfo/snort-devel
>
> Please visit http://blog.snort.org for the latest news about Snort!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20180525/ead14d24/attachment-0001.html>


More information about the Snort-devel mailing list