[Snort-devel] Segfault building Snort3 b245 with --enable-tcmalloc flag

Noah Dietrich noah_dietrich at 86penny.org
Fri Jun 15 06:19:14 EDT 2018


Thank you Michael for the deep-dive analysis of this issue. It looks that
installing gperftools 2.7 from github fixes the problem, although I haven't
done more than just run snort with the -V flag. The steps it took for this
are below.
Has anyone done an analysis of the performance benefits of using tcmalloc?
I'd assume that you're getting more speed for the trade-off of increased
memory usage, but it would be interesting if someone has some results to
look at.

Thanks,
Noah


*# remove gperftools 2.5 from system:*
sudo apt-get remove libgoogle-perftools-dev
sudo apt-get autoremove

*$ install gperftools 2.7 from github from source:*
cd ~
wget
https://github.com/gperftools/gperftools/releases/download/gperftools-2.7/gperftools-2.7.tar.gz
tar xzvf gperftools-2.7.tar.gz
cd gperftools-2.7/
./configure
make
sudo make install

*# get, build, and install Snort3*
cd ~/snort_src/
git clone https://github.com/snort3/snort3.git
cd snort3/
./configure_cmake.sh --prefix=/opt/snort3a4 --enable-tcmalloc
cd build/
make
sudo make install

*# Verify libraries linked and that snort3 runs:*
noah at snort18x:~$ ldd /opt/snort3a4/bin/snort
...
libtcmalloc.so.4 => /usr/local/lib/libtcmalloc.so.4 (0x00007fc7ce334000)
...

noah at snort18x~$ /opt/snort3a4/bin/snort -V

   ,,_     -*> Snort++ <*-
  o"  )~   Version 3.0.0 (Build 245) from 2.9.11
   ''''    By Martin Roesch & The Snort Team
           http://snort.org/contact#team
           Copyright (C) 2014-2018 Cisco and/or its affiliates. All rights
reserved.
           Copyright (C) 1998-2013 Sourcefire, Inc., et al.
           Using DAQ version 2.2.2
           Using LuaJIT version 2.1.0-beta3
           Using OpenSSL 1.1.0g  2 Nov 2017
           Using libpcap version 1.8.1
           Using PCRE version 8.39 2016-06-14
           Using ZLIB version 1.2.11
           Using FlatBuffers 1.9.0
           Using Hyperscan version 4.7.0 2018-06-13
           Using LZMA version 5.2.2


On Thu, Jun 14, 2018 at 6:16 AM, Michael Altizer via Snort-devel <
snort-devel at lists.snort.org> wrote:

> I can reproduce it on my Ubuntu 18.04 system, but it doesn't look like a
> bug in Snort.  From combing through the backtrace and stepping through the
> code, it looks like a messy bug in the interaction between tcmalloc,
> libhwloc, and libltdl.  libhwloc uses libltdl to iterate over and load all
> of its plugins.  Somewhere in the middle of "canonicalizing" the paths of
> the plugins it is evaluating, libltdl's loop gets horribly confused and
> starts writing over itself forever until get gets a SEGV.  In stepping
> through it in a debugger, it appears that the malloc it calls to create the
> new character array to write into returns an address 16 bytes into the
> original path.  I didn't dig into the assembly, but it might actually be a
> compiler issue where it optimized libltdl around assumptions using builtin
> malloc and then it gets hosed by tcmalloc doing the actual allocation.
> Alternatively, something in what it's doing is buggily stomping on
> tcmalloc's internal metadata, allowing it to give out the "path" memory
> again even while it's being used and read.  Or, who knows, maybe there's a
> bug in the gperftools 2.5 that Ubuntu is packing (we tested with 2.7)?  Not
> that it's a great solution, but I can make the crash go away by
> uninstalling the hwloc-plugins package.
>
> On 06/13/2018 12:50 AM, Noah Dietrich wrote:
>
> Hello,
>
> thanks for tracking that down.  I have verified that the segfault issue
> also occurs when building on Ubuntu 16 x64.
>
> Changing the command to:
> *      ./configure_cmake.sh --prefix=/opt/snort --enable-tcmalloc
> --disable-docs*
> does not generate any errors and allows for successful build and
> installation.
>
> however, running snort3 when* -enable-tcmalloc *is included as above
> generates a segfault;
>    noah at snort18x:~/snort_src/snort3/build$* /opt/snort/bin/snort -V*
>    Segmentation fault (core dumped)
>
> Please let me know if any further info is needed.
> thanks
> Noah
>
>
>
>
> On Tue, Jun 12, 2018 at 8:32 PM, Y M via Snort-devel <
> snort-devel at lists.snort.org> wrote:
>
>> After looking at the errors, which I received too, the build fails at
>> generating the text documentation at the target prefix. Using the
>> --disable-text-docs alone did not stop the documentation generation.
>> However, disabling all types of documentation generation resulted in a
>> successful build.
>>
>> # ./configure_cmake.sh --prefix=/usr/local/snort --enable-tcmalloc
>> --disable-text-docs --disable-html-docs --disable-pdf-docs --disable-docs
>>
>> ...
>> [ 99%] Built target preprocessor_states
>> Scanning dependencies of target snort2lua
>> [100%] Building CXX object tools/snort2lua/CMakeFiles/sno
>> rt2lua.dir/init_state.cc.o
>> [100%] Building CXX object tools/snort2lua/CMakeFiles/sno
>> rt2lua.dir/snort2lua.cc.o
>> [100%] Linking CXX executable snort
>> [100%] Linking CXX executable snort2lua
>> [100%] Built target snort2lua
>> [100%] Built target snort
>>
>> This doesn't particularly answers your question, but hopefully can help
>> you move along.
>>
>> Thanks.
>> YM
>>
>> ------------------------------
>> *From:* Snort-devel <snort-devel-bounces at lists.snort.org> on behalf of Y
>> M via Snort-devel <snort-devel at lists.snort.org>
>> *Sent:* Tuesday, June 12, 2018 8:25 PM
>> *To:* snort-devel at lists.snort.org
>> *Subject:* Re: [Snort-devel] Segfault building Snort3 b245 with
>> --enable-tcmalloc flag
>>
>> Interesting. Snort build 245 with tcmalloc over CetnOS 7 and FreeBSD 11
>> was successful. There have been recent changes in Ubuntu 18 that are making
>> several apps too unhappy. I will do some testing on my side and report back
>> if I come up with anything useful.
>>
>> YM
>>
>> ------------------------------
>> *From:* Snort-devel <snort-devel-bounces at lists.snort.org> on behalf of
>> Noah Dietrich <noah_dietrich at 86penny.org>
>> *Sent:* Tuesday, June 12, 2018 5:26 PM
>> *To:* snort-devel at lists.snort.org
>> *Subject:* [Snort-devel] Segfault building Snort3 b245 with
>> --enable-tcmalloc flag
>>
>> Hello,
>>
>> when building Snort 3 245 with the --enable-tcmalloc flag, snort throws a
>> segfault.
>>
>> *Environment: *git clone of the latest Snort3 (b245) on Ubuntu 18 x64 in
>> a VMware virtual Machine.
>>
>> *Commands:*
>> cd ~/snort_src
>> git clone https://github.com/snort3/snort3.git
>> cd ./snort3
>> ./configure_cmake.sh --prefix=/usr/local  --enable-tcmalloc
>> cd build
>> make
>>
>> *Error:*
>> ...
>> Scanning dependencies of target all_built_sources
>> [ 97%] Documents: building commands.txt with
>> Segmentation fault (core dumped)
>> doc/CMakeFiles/all_built_sources.dir/build.make:80: recipe for target
>> 'doc/commands.txt' failed
>> make[2]: *** [doc/commands.txt] Error 139
>> make[2]: *** Deleting file 'doc/commands.txt'
>> CMakeFiles/Makefile2:6091: recipe for target
>> 'doc/CMakeFiles/all_built_sources.dir/all' failed
>> make[1]: *** [doc/CMakeFiles/all_built_sources.dir/all] Error 2
>> Makefile:151: recipe for target 'all' failed
>> make: *** [all] Error 2
>>
>>
>>
>> When you remove the --enable-tcmalloc flag, snort compiles and runs
>> without error:
>>
>>    ,,_     -*> Snort++ <*-
>>   o"  )~   Version 3.0.0 (Build 245) from 2.9.11
>>    ''''    By Martin Roesch & The Snort Team
>>            http://snort.org/contact#team
>>            Copyright (C) 2014-2018 Cisco and/or its affiliates. All
>> rights reserved.
>>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>>            Using DAQ version 2.2.2
>>            Using LuaJIT version 2.1.0-beta3
>>            Using OpenSSL 1.1.0g  2 Nov 2017
>>            Using libpcap version 1.8.1
>>            Using PCRE version 8.39 2016-06-14
>>            Using ZLIB version 1.2.11
>>            Using FlatBuffers 1.9.0
>>            Using Hyperscan version 4.7.0 2018-06-12
>>            Using LZMA version 5.2.2
>>
>>
>> please let me know if you need any other information.
>> thank you,
>> Noah
>>
>>
>>
>> _______________________________________________
>> Snort-devel mailing list
>> Snort-devel at lists.snort.org
>> https://lists.snort.org/mailman/listinfo/snort-devel
>>
>> Please visit http://blog.snort.org for the latest news about Snort!
>>
>>
>
>
> _______________________________________________
> Snort-devel mailing listSnort-devel at lists.snort.orghttps://lists.snort.org/mailman/listinfo/snort-devel
>
> Please visit http://blog.snort.org for the latest news about Snort!
>
>
>
> _______________________________________________
> Snort-devel mailing list
> Snort-devel at lists.snort.org
> https://lists.snort.org/mailman/listinfo/snort-devel
>
> Please visit http://blog.snort.org for the latest news about Snort!
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20180615/ee6b0434/attachment-0001.html>


More information about the Snort-devel mailing list