[Snort-devel] Segfault building Snort3 b245 with --enable-tcmalloc flag

Michael Altizer mialtize at cisco.com
Mon Jun 18 16:43:21 EDT 2018


Good to hear that switching to a newer version of gperftools solved the 
problem, although that's still a little troubling.  I don't have exact 
numbers, but at least in one of the more contrived tests we were doing, 
it looked like a 7-12% reduction in processing time.

On 06/15/2018 06:19 AM, Noah Dietrich wrote:
> Thank you Michael for the deep-dive analysis of this issue. It looks 
> that installing gperftools 2.7 from github fixes the problem, although 
> I haven't done more than just run snort with the -V flag. The steps it 
> took for this are below.
> Has anyone done an analysis of the performance benefits of using 
> tcmalloc?  I'd assume that you're getting more speed for the trade-off 
> of increased memory usage, but it would be interesting if someone has 
> some results to look at.
>
> Thanks,
> Noah
>
>
> *# remove gperftools 2.5 from system:*
> sudo apt-get remove libgoogle-perftools-dev
> sudo apt-get autoremove
>
> *$ install gperftools 2.7 from github from source:*
> cd ~
> wget 
> https://github.com/gperftools/gperftools/releases/download/gperftools-2.7/gperftools-2.7.tar.gz
> tar xzvf gperftools-2.7.tar.gz
> cd gperftools-2.7/
> ./configure
> make
> sudo make install
>
> *# get, build, and install Snort3*
> cd ~/snort_src/
> git clone https://github.com/snort3/snort3.git
> cd snort3/
> ./configure_cmake.sh --prefix=/opt/snort3a4 --enable-tcmalloc
> cd build/
> make
> sudo make install
>
> *# Verify libraries linked and that snort3 runs:*
> noah at snort18x:~$ ldd /opt/snort3a4/bin/snort
> ...
> libtcmalloc.so.4 => /usr/local/lib/libtcmalloc.so.4 (0x00007fc7ce334000)
> ...
>
> noah at snort18x~$ /opt/snort3a4/bin/snort -V
>
>    ,,_     -*> Snort++ <*-
>   o"  )~   Version 3.0.0 (Build 245) from 2.9.11
>    ''''    By Martin Roesch & The Snort Team
> http://snort.org/contact#team
>            Copyright (C) 2014-2018 Cisco and/or its affiliates. All 
> rights reserved.
>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>            Using DAQ version 2.2.2
>            Using LuaJIT version 2.1.0-beta3
>            Using OpenSSL 1.1.0g  2 Nov 2017
>            Using libpcap version 1.8.1
>            Using PCRE version 8.39 2016-06-14
>            Using ZLIB version 1.2.11
>            Using FlatBuffers 1.9.0
>            Using Hyperscan version 4.7.0 2018-06-13
>            Using LZMA version 5.2.2
>
>
> On Thu, Jun 14, 2018 at 6:16 AM, Michael Altizer via Snort-devel 
> <snort-devel at lists.snort.org <mailto:snort-devel at lists.snort.org>> wrote:
>
>     I can reproduce it on my Ubuntu 18.04 system, but it doesn't look
>     like a bug in Snort.  From combing through the backtrace and
>     stepping through the code, it looks like a messy bug in the
>     interaction between tcmalloc, libhwloc, and libltdl.  libhwloc
>     uses libltdl to iterate over and load all of its plugins. 
>     Somewhere in the middle of "canonicalizing" the paths of the
>     plugins it is evaluating, libltdl's loop gets horribly confused
>     and starts writing over itself forever until get gets a SEGV.  In
>     stepping through it in a debugger, it appears that the malloc it
>     calls to create the new character array to write into returns an
>     address 16 bytes into the original path.  I didn't dig into the
>     assembly, but it might actually be a compiler issue where it
>     optimized libltdl around assumptions using builtin malloc and then
>     it gets hosed by tcmalloc doing the actual allocation.
>     Alternatively, something in what it's doing is buggily stomping on
>     tcmalloc's internal metadata, allowing it to give out the "path"
>     memory again even while it's being used and read.  Or, who knows,
>     maybe there's a bug in the gperftools 2.5 that Ubuntu is packing
>     (we tested with 2.7)?  Not that it's a great solution, but I can
>     make the crash go away by uninstalling the hwloc-plugins package.
>
>     On 06/13/2018 12:50 AM, Noah Dietrich wrote:
>>     Hello,
>>
>>     thanks for tracking that down.  I have verified that the segfault
>>     issue also occurs when building on Ubuntu 16 x64.
>>
>>     Changing the command to:
>>     *      ./configure_cmake.sh --prefix=/opt/snort --enable-tcmalloc
>>     --disable-docs*
>>     does not generate any errors and allows for successful build and
>>     installation.
>>
>>     however, running snort3 when*-enable-tcmalloc *is included as
>>     above generates a segfault;
>>        noah at snort18x:~/snort_src/snort3/build$*/opt/snort/bin/snort -V*
>>        Segmentation fault (core dumped)
>>
>>     Please let me know if any further info is needed.
>>     thanks
>>     Noah
>>
>>
>>
>>
>>     On Tue, Jun 12, 2018 at 8:32 PM, Y M via Snort-devel
>>     <snort-devel at lists.snort.org
>>     <mailto:snort-devel at lists.snort.org>> wrote:
>>
>>         After looking at the errors, which I received too, the build
>>         fails at generating the text documentation at the target
>>         prefix. Using the --disable-text-docs alone did not stop the
>>         documentation generation. However, disabling all types of
>>         documentation generation resulted in a successful build.
>>
>>         # ./configure_cmake.sh --prefix=/usr/local/snort
>>         --enable-tcmalloc --disable-text-docs --disable-html-docs
>>         --disable-pdf-docs --disable-docs
>>
>>         ...
>>         [ 99%] Built target preprocessor_states
>>         Scanning dependencies of target snort2lua
>>         [100%] Building CXX object
>>         tools/snort2lua/CMakeFiles/snort2lua.dir/init_state.cc.o
>>         [100%] Building CXX object
>>         tools/snort2lua/CMakeFiles/snort2lua.dir/snort2lua.cc.o
>>         [100%] Linking CXX executable snort
>>         [100%] Linking CXX executable snort2lua
>>         [100%] Built target snort2lua
>>         [100%] Built target snort
>>
>>         This doesn't particularly answers your question, but
>>         hopefully can help you move along.
>>
>>         Thanks.
>>         YM
>>
>>         ------------------------------------------------------------------------
>>         *From:* Snort-devel <snort-devel-bounces at lists.snort.org
>>         <mailto:snort-devel-bounces at lists.snort.org>> on behalf of Y
>>         M via Snort-devel <snort-devel at lists.snort.org
>>         <mailto:snort-devel at lists.snort.org>>
>>         *Sent:* Tuesday, June 12, 2018 8:25 PM
>>         *To:* snort-devel at lists.snort.org
>>         <mailto:snort-devel at lists.snort.org>
>>         *Subject:* Re: [Snort-devel] Segfault building Snort3 b245
>>         with --enable-tcmalloc flag
>>         Interesting. Snort build 245 with tcmalloc over CetnOS 7 and
>>         FreeBSD 11 was successful. There have been recent changes in
>>         Ubuntu 18 that are making several apps too unhappy. I will do
>>         some testing on my side and report back if I come up with
>>         anything useful.
>>
>>         YM
>>
>>         ------------------------------------------------------------------------
>>         *From:* Snort-devel <snort-devel-bounces at lists.snort.org
>>         <mailto:snort-devel-bounces at lists.snort.org>> on behalf of
>>         Noah Dietrich <noah_dietrich at 86penny.org
>>         <mailto:noah_dietrich at 86penny.org>>
>>         *Sent:* Tuesday, June 12, 2018 5:26 PM
>>         *To:* snort-devel at lists.snort.org
>>         <mailto:snort-devel at lists.snort.org>
>>         *Subject:* [Snort-devel] Segfault building Snort3 b245 with
>>         --enable-tcmalloc flag
>>         Hello,
>>
>>         when building Snort 3 245 with the --enable-tcmalloc flag,
>>         snort throws a segfault.
>>
>>         *Environment: *git clone of the latest Snort3 (b245) on
>>         Ubuntu 18 x64 in a VMware virtual Machine.
>>
>>         *Commands:*
>>         cd ~/snort_src
>>         git clone https://github.com/snort3/snort3.git
>>         <https://github.com/snort3/snort3.git>
>>         cd ./snort3
>>         ./configure_cmake.sh --prefix=/usr/local --enable-tcmalloc
>>         cd build
>>         make
>>
>>         *Error:*
>>         ...
>>         Scanning dependencies of target all_built_sources
>>         [ 97%] Documents: building commands.txt with
>>         Segmentation fault (core dumped)
>>         doc/CMakeFiles/all_built_sources.dir/build.make:80: recipe
>>         for target 'doc/commands.txt' failed
>>         make[2]: *** [doc/commands.txt] Error 139
>>         make[2]: *** Deleting file 'doc/commands.txt'
>>         CMakeFiles/Makefile2:6091: recipe for target
>>         'doc/CMakeFiles/all_built_sources.dir/all' failed
>>         make[1]: *** [doc/CMakeFiles/all_built_sources.dir/all] Error 2
>>         Makefile:151: recipe for target 'all' failed
>>         make: *** [all] Error 2
>>
>>
>>
>>         When you remove the --enable-tcmalloc flag, snort compiles
>>         and runs without error:
>>
>>            ,,_     -*> Snort++ <*-
>>           o"  )~   Version 3.0.0 (Build 245) from 2.9.11
>>            ''''    By Martin Roesch & The Snort Team
>>         http://snort.org/contact#team
>>                    Copyright (C) 2014-2018 Cisco and/or its
>>         affiliates. All rights reserved.
>>                    Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>                    Using DAQ version 2.2.2
>>                    Using LuaJIT version 2.1.0-beta3
>>                    Using OpenSSL 1.1.0g  2 Nov 2017
>>                    Using libpcap version 1.8.1
>>                    Using PCRE version 8.39 2016-06-14
>>                    Using ZLIB version 1.2.11
>>                    Using FlatBuffers 1.9.0
>>                    Using Hyperscan version 4.7.0 2018-06-12
>>                    Using LZMA version 5.2.2
>>
>>
>>         please let me know if you need any other information.
>>         thank you,
>>         Noah
>>
>>
>>
>>         _______________________________________________
>>         Snort-devel mailing list
>>         Snort-devel at lists.snort.org <mailto:Snort-devel at lists.snort.org>
>>         https://lists.snort.org/mailman/listinfo/snort-devel
>>         <https://lists.snort.org/mailman/listinfo/snort-devel>
>>
>>         Please visit http://blog.snort.org for the latest news about
>>         Snort!
>>
>>
>>
>>
>>     _______________________________________________
>>     Snort-devel mailing list
>>     Snort-devel at lists.snort.org <mailto:Snort-devel at lists.snort.org>
>>     https://lists.snort.org/mailman/listinfo/snort-devel
>>     <https://lists.snort.org/mailman/listinfo/snort-devel>
>>
>>     Please visithttp://blog.snort.org  for the latest news about Snort!
>
>
>
>     _______________________________________________
>     Snort-devel mailing list
>     Snort-devel at lists.snort.org <mailto:Snort-devel at lists.snort.org>
>     https://lists.snort.org/mailman/listinfo/snort-devel
>     <https://lists.snort.org/mailman/listinfo/snort-devel>
>
>     Please visit http://blog.snort.org for the latest news about Snort!
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20180618/08f548cf/attachment-0001.html>


More information about the Snort-devel mailing list