[Snort-users] File extraction during http/ftp transaction

Rishabh Shah rishabh420 at ...11827...
Wed Mar 11 13:21:39 EDT 2015


Hi Hui,

I missed creating the directory(assumed that snort would create one). It is
working now. Thanks a ton Hui.

One minor query regarding the new files:
-rw------- 1 root root  7091 Mar 11 22:48
9D29C44863C6A27D45F8621E6A636DF0746245C5F436DB9CA488252A7FF76579
-rw------- 1 root root 22016 Mar 11 22:49
67792ACE824606664CE51973800D6B952CA4733CAF6F03CCF5F636768EFB39B1

Can it not retain the name/extension of the file?

Thanks,
Rishabh.

On Wed, Mar 11, 2015 at 10:12 PM, Hui cao <huica at ...589...> wrote:

>  Sorry. Don't change the conf, but check whether you have permission
> "write" on the folder
>
>
>
> */home/file_capture/tmp/ Best, Hui. *
>
> On 03/11/2015 12:37 PM, Rishabh Shah wrote:
>
> Hi Hui,
>
>  I removed signature and transferred two pcap files, but no luck:
>
>  File Preprocessor Statistics
>   Total file type callbacks:            2
>   Total file signature callbacks:       2
>   Total files would saved to disk:      2
>   Total files saved to disk:            0
>   Total file data saved to disk:        0         bytes
>   Total files duplicated:               0
>   Total files reserving failed:         0
>   Total file capture min:               0
>   Total file capture max:               0
>   Total file capture memcap:            0
>   Total files reading failed:           0
>   Total file agent memcap failures:     0
>   Total files sent:                     0
>   Total file data sent:                 0
>   Total file transfer failures:         0
>
> ===============================================================================
> File type stats:
>          Type              Download   (Bytes)      Upload     (Bytes)
>         PCAP(145)          2          3870         0          0
>             Total          2          3870         0          0
>
>  File signature stats:
>          Type              Download   Upload
>         PCAP(145)          2          0
>             Total          2          0
>
>  File type verdicts:
>         UNKNOWN:           2
>             LOG:           0
>            STOP:           0
>           BLOCK:           0
>          REJECT:           0
>         PENDING:           0
>    STOP CAPTURE:           0
>           Total:           2
>
>  File signature verdicts:
>         UNKNOWN:           2
>             LOG:           0
>            STOP:           0
>           BLOCK:           0
>          REJECT:           0
>         PENDING:           0
>    STOP CAPTURE:           0
>           Total:           2
>
>  Total files processed:             2
> Total files data processed:        3870      bytes
> Total files buffered:              2
> Total files released:              2
> Total files freed:                 0
> Total files captured:              2
> Total files within one packet:     2
> Total buffers allocated:           2
> Total buffers freed:               0
> Total buffers released:            2
> Maximum file buffers used:         1
> Total buffers free errors:         0
> Total buffers release errors:      0
> Total memcap failures:             0
> Total memcap failures at reserve:  0
> Total reserve failures:            0
> Total file capture size min:       0
> Total file capture size max:       0
> Total capture max before reserve:  0
> Total file signature max:          0
> Maximum buffers can allocate:      3196
> Number of buffers in use:          0
> Number of buffers in free list:    3194
> Number of buffers in release list: 2
>
>
>
> On Wed, Mar 11, 2015 at 10:02 PM, Hui cao <huica at ...589...> wrote:
>
>>  Can you remove signature? If this is enabled, it only captures file that
>> matches to a signature list.
>>
>>  *preprocessor file_inspect: type_id, capture_disk
>> /home/file_capture/tmp/, capture_queue_size 5000*
>>
>> Best,
>> Hui.
>>
>>
>> On 03/11/2015 12:24 PM, Rishabh Shah wrote:
>>
>> Hi Hui,
>>
>>  I included file_magic.conf in my snort configuration file. After
>> starting the snort process, I transferred 3 files and this is the output
>> after stopping snort:
>>
>>  File Preprocessor Statistics
>>   Total file type callbacks:            1
>>   Total file signature callbacks:       1
>>   Total files would saved to disk:      1
>>   Total files saved to disk:            0
>>   Total file data saved to disk:        0         bytes
>>   Total files duplicated:               0
>>   Total files reserving failed:         0
>>   Total file capture min:               0
>>   Total file capture max:               0
>>   Total file capture memcap:            0
>>   Total files reading failed:           0
>>   Total file agent memcap failures:     0
>>   Total files sent:                     0
>>   Total file data sent:                 0
>>   Total file transfer failures:         0
>>
>> ===============================================================================
>> File type stats:
>>          Type              Download   (Bytes)      Upload     (Bytes)
>>         PCAP(145)          1          1935         0          0
>>             Total          1          1935         0          0
>>
>>  File signature stats:
>>          Type              Download   Upload
>>         PCAP(145)          1          0
>>             Total          1          0
>>
>>  File type verdicts:
>>         UNKNOWN:           1
>>             LOG:           0
>>            STOP:           0
>>           BLOCK:           0
>>          REJECT:           0
>>         PENDING:           0
>>    STOP CAPTURE:           0
>>           Total:           1
>>
>>  File signature verdicts:
>>         UNKNOWN:           1
>>             LOG:           0
>>            STOP:           0
>>           BLOCK:           0
>>          REJECT:           0
>>         PENDING:           0
>>    STOP CAPTURE:           0
>>           Total:           1
>>
>>  *Total files processed:             3*
>> Total files data processed:        8124      bytes
>> Total files buffered:              1
>> Total files released:              1
>> Total files freed:                 0
>> Total files captured:              1
>> Total files within one packet:     1
>> Total buffers allocated:           1
>> Total buffers freed:               0
>> Total buffers released:            1
>> Maximum file buffers used:         1
>> Total buffers free errors:         0
>> Total buffers release errors:      0
>> Total memcap failures:             0
>> Total memcap failures at reserve:  0
>> Total reserve failures:            0
>> Total file capture size min:       0
>> Total file capture size max:       0
>> Total capture max before reserve:  0
>> Total file signature max:          0
>> Maximum buffers can allocate:      3196
>> Number of buffers in use:          0
>> Number of buffers in free list:    3195
>> Number of buffers in release list: 1
>>
>> ===============================================================================
>>
>>
>> On Wed, Mar 11, 2015 at 9:34 PM, Hui cao <huica at ...589...> wrote:
>>
>>>  In READMe.file:
>>>
>>> Pre-packaged file magic rules:
>>>
>>> A set of file magic rules is packaged with Snort. They can be located at
>>> "etc/file_magic.conf". To use this feature, it is recommended that the
>>> these pre-packaged rules are used; doing so requires that you include
>>> the file in your Snort configuration as such:
>>>
>>>   include etc/filemagic.conf
>>>
>>> On 03/11/2015 12:01 PM, Hui cao wrote:
>>>
>>> Have you added file magic into your configuration. What's the snort
>>> output?
>>>
>>> Best,
>>> Hui.
>>>
>>> On 03/11/2015 11:56 AM, Rishabh Shah wrote:
>>>
>>> Thanks Hui. That worked for me!
>>> Now I started snort after adding file_inspect preprocessor.
>>>  *preprocessor file_inspect: type_id, signature, capture_disk
>>> /home/file_capture/tmp/, capture_queue_size 5000*
>>>
>>>  (Got the following console logs to confirm that file_inspect has
>>> started)
>>>
>>>  File config:
>>>     file type: ENABLED
>>>     file signature: ENABLED
>>>     file capture: ENABLED
>>> *    file capture directory: /home/file_capture/tmp/*
>>>     file capture disk size: 300 (Default) megabytes
>>>     file sent to host: DISABLED (Default), port number: 0
>>>
>>>  File service: file type enabled.
>>> File service: file signature enabled.
>>> File service: file capture enabled.
>>> File capture thread started tid=0x7f0aaa783700 (pid=19354)
>>>
>>>
>>>  I initiated file transfer via HTTP/FTP as shown below:
>>>
>>>  rishab%ftp 192.168.2.200
>>> Connected to 192.168.2.200:21.
>>> 220 (vsFTPd 2.0.5)
>>> Name (192.168.2.200:21:fwdevtest1): fwuser
>>> 331 Please specify the password.
>>> Password:
>>> 230 Login successful.
>>> Remote system type is UNIX.
>>> Using binary mode to transfer files.
>>> *ftp> get new.pcap*
>>> *200 PORT command successful. Consider using PASV.*
>>> *150 Opening BINARY mode data connection for new.pcap (1555 bytes).*
>>> *226 File send OK.*
>>> *1555 bytes received in 0.4 seconds (3887 bytes/s)*
>>> ftp>
>>> ftp> quit
>>> 221 Goodbye.
>>> *rishab%wget 192.168.2.200/dns.pcap <http://192.168.2.200/dns.pcap>*
>>> *--2015-03-11 21:23:16--  http://192.168.2.200/dns.pcap
>>> <http://192.168.2.200/dns.pcap>*
>>> *Connecting to 192.168.2.200:80... connected.*
>>> *HTTP request sent, awaiting response... 200 OK*
>>> *Length: 1935 (1.9K) [text/plain]*
>>> *Saving to: ?dns.pcap?*
>>>
>>>  *100%[======================================================================================================================================================================================>]
>>> 1,935       9.39KB/s   in 0.2s*
>>>
>>>  *2015-03-11 21:23:19 (9.39 KB/s) - ?dns.pcap? saved [1935/1935]*
>>>
>>>
>>>  After killing the snort process, I do not see any file created in that
>>> location:
>>>
>>>  root at ...17114...:/home# ls
>>> fwuser
>>>
>>>  Am I missing anything?
>>>
>>>
>>> On Wed, Mar 11, 2015 at 9:09 PM, Hui cao <huica at ...589...> wrote:
>>>
>>>>  Have you done make clean before you do a make?
>>>>
>>>> Best,
>>>> Hui.
>>>>
>>>>
>>>> On 03/11/2015 11:38 AM, Rishabh Shah wrote:
>>>>
>>>> Hi Hui,
>>>>
>>>>  I am hitting the same issue while executing make. These are the
>>>> commands that I issued:
>>>>  root at ...17114...:~/snort_src/snort-2.9.7.0# ./configure
>>>> --enable-file-inspect --enable-open-appid --enable-sourcefire
>>>>
>>>>  root at ...17114...:~/snort_src/snort-2.9.7.0# make
>>>>
>>>>
>>>>  /root/snort_src/snort-2.9.7.0/src/plugbase.c:216: undefined reference
>>>> to `SetupAppId'
>>>> detection-plugins/libspd.a(detection_options.o): In function
>>>> `detection_hash_free_func':
>>>> /root/snort_src/snort-2.9.7.0/src/detection-plugins/detection_options.c:553:
>>>> undefined reference to `optionAppIdFree'
>>>> detection-plugins/libspd.a(detection_options.o): In function
>>>> `detection_option_hash_func':
>>>> /root/snort_src/snort-2.9.7.0/src/detection-plugins/detection_options.c:252:
>>>> undefined reference to `optionAppIdHash'
>>>> detection-plugins/libspd.a(detection_options.o): In function
>>>> `detection_option_key_compare_func':
>>>> /root/snort_src/snort-2.9.7.0/src/detection-plugins/detection_options.c:409:
>>>> undefined reference to `optionAppIdCompare'
>>>> collect2: error: ld returned 1 exit status
>>>> make[3]: *** [snort] Error 1
>>>> make[3]: Leaving directory `/root/snort_src/snort-2.9.7.0/src'
>>>> make[2]: *** [all-recursive] Error 1
>>>> make[2]: Leaving directory `/root/snort_src/snort-2.9.7.0/src'
>>>> make[1]: *** [all-recursive] Error 1
>>>> make[1]: Leaving directory `/root/snort_src/snort-2.9.7.0'
>>>> make: *** [all] Error 2
>>>>
>>>>
>>>> On Wed, Mar 11, 2015 at 8:40 PM, Hui cao <huica at ...589...> wrote:
>>>>
>>>>>  Hi Rishabh,
>>>>>
>>>>> You need to add —enable-open-appid to you ./configure.
>>>>>
>>>>> ./configure --enable-file-inspect —enable-open-appid
>>>>>
>>>>> Best,
>>>>> Hui.
>>>>>
>>>>> On 03/11/2015 10:33 AM, Rishabh Shah wrote:
>>>>>
>>>>>  Hi Joel,
>>>>>
>>>>>  Thanks for your prompt reply. I did a ./configure
>>>>> --enable-file-inspect and while executing make, I saw the following error
>>>>> messages:
>>>>>
>>>>>  */root/snort_src/snort-2.9.7.0/src/plugbase.c:216: undefined
>>>>> reference to `SetupAppId'*
>>>>> *detection-plugins/libspd.a(detection_options.o): In function
>>>>> `detection_hash_free_func':*
>>>>> */root/snort_src/snort-2.9.7.0/src/detection-plugins/detection_options.c:553:
>>>>> undefined reference to `optionAppIdFree'*
>>>>> *detection-plugins/libspd.a(detection_options.o): In function
>>>>> `detection_option_hash_func':*
>>>>> */root/snort_src/snort-2.9.7.0/src/detection-plugins/detection_options.c:252:
>>>>> undefined reference to `optionAppIdHash'*
>>>>> *detection-plugins/libspd.a(detection_options.o): In function
>>>>> `detection_option_key_compare_func':*
>>>>> */root/snort_src/snort-2.9.7.0/src/detection-plugins/detection_options.c:409:
>>>>> undefined reference to `optionAppIdCompare'*
>>>>> *collect2: error: ld returned 1 exit status*
>>>>> make[3]: *** [snort] Error 1
>>>>> make[3]: Leaving directory `/root/snort_src/snort-2.9.7.0/src'
>>>>> make[2]: *** [all-recursive] Error 1
>>>>> make[2]: Leaving directory `/root/snort_src/snort-2.9.7.0/src'
>>>>> make[1]: *** [all-recursive] Error 1
>>>>> make[1]: Leaving directory `/root/snort_src/snort-2.9.7.0'
>>>>> make: *** [all] Error 2
>>>>>
>>>>>  I am not sure why am I seeing those messages as I see a reference to
>>>>> the above errors:
>>>>>
>>>>>  root at ...17114...:~/snort_src/snort-2.9.7.0/src# grep -r
>>>>> "optionAppIdFree" .
>>>>> Binary file ./detection-plugins/detection_options.o matches
>>>>> Binary file ./detection-plugins/sp_appid.o matches
>>>>> ./detection-plugins/sp_appid.c:void optionAppIdFree(AppIdOptionData
>>>>> *optData)
>>>>> ./detection-plugins/sp_appid.c:        optionAppIdFree(optData);
>>>>> Binary file ./detection-plugins/libspd.a matches
>>>>> ./detection-plugins/detection_options.c:
>>>>>  optionAppIdFree(key->option_data);
>>>>> ./detection-plugins/sp_appid.h:void optionAppIdFree(AppIdOptionData
>>>>> *optData);
>>>>>
>>>>>
>>>>>  I appended the following line in snort.conf:
>>>>>  *preprocessor file_inspect: type_id, signature, capture_disk
>>>>> /home/file_capture/tmp/, capture_queue_size 5000*
>>>>>
>>>>>  While executing snort process, I got a core file with the following
>>>>> message:
>>>>>
>>>>>  File config:
>>>>>     file type: ENABLED
>>>>>     file signature: ENABLED
>>>>>     file capture: ENABLED
>>>>>     file capture directory: /home/file_capture/tmp/
>>>>>     file capture disk size: 300 (Default) megabytes
>>>>>     file sent to host: DISABLED (Default), port number: 0
>>>>>
>>>>>  *Segmentation fault (core dumped)*
>>>>>
>>>>>  The traceback of the core file points to:
>>>>>
>>>>>  root at ...17114...:~/snort_src# gdb snort -c core
>>>>> GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
>>>>> Copyright (C) 2014 Free Software Foundation, Inc.
>>>>> License GPLv3+: GNU GPL version 3 or later <
>>>>> http://gnu.org/licenses/gpl.html>
>>>>> This is free software: you are free to change and redistribute it.
>>>>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>>>>> copying"
>>>>> and "show warranty" for details.
>>>>> This GDB was configured as "x86_64-linux-gnu".
>>>>> Type "show configuration" for configuration details.
>>>>> For bug reporting instructions, please see:
>>>>> <http://www.gnu.org/software/gdb/bugs/>.
>>>>> Find the GDB manual and other documentation resources online at:
>>>>> <http://www.gnu.org/software/gdb/documentation/>.
>>>>> For help, type "help".
>>>>> Type "apropos word" to search for commands related to "word"...
>>>>> Reading symbols from snort...done.
>>>>>
>>>>>  warning: exec file is newer than core file.
>>>>> [New LWP 10904]
>>>>>
>>>>>  warning: .dynamic section for
>>>>> "/usr/local/lib/snort_dynamicengine/libsf_engine.so" is not at the expected
>>>>> address (wrong library or version mismatch?)
>>>>>
>>>>>  warning: .dynamic section for
>>>>> "/usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so" is
>>>>> not at the expected address (wrong library or version mismatch?)
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library
>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/bin/snort -c /etc/snort/snort.conf
>>>>> -Q -i eth1:eth2 -l /var/log/snort'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0  strlen () at ../sysdeps/x86_64/strlen.S:106
>>>>> 106     ../sysdeps/x86_64/strlen.S: No such file or directory.
>>>>> (gdb) bt
>>>>> *#0  strlen () at ../sysdeps/x86_64/strlen.S:106*
>>>>> *#1  0x00007f6ab63050a6 in appIdStatsInit (appFileName=0x7f6ab6628170
>>>>> <config+16> "appstats-unified.log", statsPeriod=10, rolloverSize=20971520,
>>>>> rolloverPeriod=86400) at appIdStats.c:264*
>>>>> *#2  0x00007f6ab62fa2d0 in AppIdCommonInit (memcap=268435456) at
>>>>> commonAppMatcher.c:297*
>>>>> *#3  0x00007f6ab6303798 in AppIdInit (sc=0x1eb9770, args=0x1f516e0
>>>>> "app_stats_filename appstats-unified.log, app_stats_period 10,
>>>>> app_detector_dir /usr/local/lib/openappid") at spp_appid.c:157*
>>>>> *#4  0x000000000042048e in InitVarTables (p=0x1eb9770) at
>>>>> parser.c:5728*
>>>>> *#5  0x000000000046c3d0 in CheckAppId (option_data=0x0, p=0x0) at
>>>>> sp_appid.c:342*
>>>>> *#6  0x0000000000000000 in ?? ()*
>>>>> *(gdb) Quit*
>>>>>
>>>>>  I had installed openappid as well.
>>>>>
>>>>>
>>>>> On Wed, Mar 11, 2015 at 7:00 PM, Joel Esler (jesler) <jesler at ...589...
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>>  On Mar 11, 2015, at 9:23 AM, Rishabh Shah <rishabh420 at ...11827...>
>>>>>> wrote:
>>>>>>
>>>>>>  Hi Snort Team,
>>>>>>
>>>>>>  Is it possible to extract any file during http/ftp transactions?
>>>>>> The HTTP preprocessor makes it possible to read the HTTP URI/content. Does
>>>>>> snort have the intelligence to extract the file during any transfer?
>>>>>>
>>>>>>
>>>>>>  Beginning with 2.9.6.0, Snort has had the ability to extract files
>>>>>> from streams and write them to disk.
>>>>>>
>>>>>>  Check out the README: https://www.snort.org/faq/readme-file
>>>>>>
>>>>>>  --
>>>>>> *Joel Esler*
>>>>>> Open Source Manager
>>>>>> Threat Intelligence Team Lead
>>>>>> Talos Group
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>> Regards,
>>>>> Rishabh Shah.
>>>>>
>>>>>
>>>>>  ------------------------------------------------------------------------------
>>>>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>>>>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>>>>> things parallel software development, from weekly thought leadership blogs to
>>>>> news, videos, case studies, tutorials and more. Take a look and join the
>>>>> conversation now. http://goparallel.sourceforge.net/
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Snort-users mailing listSnort-users at lists.sourceforge.net
>>>>> Go to this URL to change user options or unsubscribe:https://lists.sourceforge.net/lists/listinfo/snort-users
>>>>> Snort-users list archive:http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>>>>>
>>>>> Please visit http://blog.snort.org to stay current on all the latest Snort news!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>>>> sponsored
>>>>> by Intel and developed in partnership with Slashdot Media, is your hub
>>>>> for all
>>>>> things parallel software development, from weekly thought leadership
>>>>> blogs to
>>>>> news, videos, case studies, tutorials and more. Take a look and join
>>>>> the
>>>>> conversation now. http://goparallel.sourceforge.net/
>>>>> _______________________________________________
>>>>> Snort-users mailing list
>>>>> Snort-users at lists.sourceforge.net
>>>>> Go to this URL to change user options or unsubscribe:
>>>>> https://lists.sourceforge.net/lists/listinfo/snort-users
>>>>> Snort-users list archive:
>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>>>>>
>>>>> Please visit http://blog.snort.org to stay current on all the latest
>>>>> Snort news!
>>>>>
>>>>
>>>>
>>>>
>>>>  --
>>>> Regards,
>>>> Rishabh Shah.
>>>>
>>>>
>>>>
>>>
>>>
>>>  --
>>> Regards,
>>> Rishabh Shah.
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>>> things parallel software development, from weekly thought leadership blogs to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now. http://goparallel.sourceforge.net/
>>>
>>>
>>>
>>> _______________________________________________
>>> Snort-users mailing listSnort-users at lists.sourceforge.net
>>> Go to this URL to change user options or unsubscribe:https://lists.sourceforge.net/lists/listinfo/snort-users
>>> Snort-users list archive:http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>>>
>>> Please visit http://blog.snort.org to stay current on all the latest Snort news!
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>> sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub
>>> for all
>>> things parallel software development, from weekly thought leadership
>>> blogs to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Snort-users mailing list
>>> Snort-users at lists.sourceforge.net
>>> Go to this URL to change user options or unsubscribe:
>>> https://lists.sourceforge.net/lists/listinfo/snort-users
>>> Snort-users list archive:
>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>>>
>>> Please visit http://blog.snort.org to stay current on all the latest
>>> Snort news!
>>>
>>
>>
>>
>>  --
>> Regards,
>> Rishabh Shah.
>>
>>
>>
>
>
>  --
> Regards,
> Rishabh Shah.
>
>
>


-- 
Regards,
Rishabh Shah.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20150311/0202dd65/attachment.html>


More information about the Snort-users mailing list