[Snort-users] Getting alerts for every file Snort detects and File Services preprocessor

Jaime Nebrera jnebrera at ...16842...
Fri Mar 27 17:46:44 EDT 2015


Hi Víctor,

I'm not that technical ;) Pablo will coment on your ideas, I just tried to
give you a deeper context of what Pablo (as part of my team) was doing but
for sure, they are welcomed. I let them continue
El 27/03/2015 21:25, "Victor Roemer" <viroemer at ...589...> escribió:

> Jamie,
>
> That's neat, I'll keep an eye on it. :)
>
> Reviewing over how the "file_identify" works currently, when doing capture
> or signature
> its not possible to also get the file type event via the preprocessor.
> I'll open an bug for this.
>
> In order to get the file type events with these further settings, more
> sophisticated tracking
> needs to be in-place in the file_identify preproc; the tricky bits being
> that the file service code
> is generating the events in question; however once a verdict is passed to
> file service- an
> action will be taken at that time.
>
> To clarify the verdicts -
> As I understand it, passing FILE_VERDICT_UNKNOWN is how processing
> continues further.
> The FILE_VERDICT_PENDING when passed to file service api, should cause the
> packet to be
> dropped, in-order to give the interfacing code time to retrieve a true
> verdict when the re-transmitted
> packet is seen.
>
> On 03/27/15 16:04, Jaime Nebrera wrote:
>
>> Hi Víctor,
>>
>> Pablo's work is geared towards a more ambitious goal:
>>
>> 1) Intercept the file and send it to a S3 based storage platform for
>> further analysis or whatever
>>
>> 2) Provide as much context about such interception as possible, like from
>> were to whom, URL, email, etc
>>
>> This we hope to open source and make it public as soon as it is usable in
>> our repository www.github.com/redborder
>> El 27/03/2015 20:50, "Victor Roemer" <viroemer at ...589...> escribió:
>>
>>    Pablo,
>>>
>>> Sorry for the delay. The "file_identify" preprocessor is designed to work
>>> more as a "reputation" based system. I think you will get better millage
>>> by
>>> using the new "file_type" rule keyword in a plain-old snort rule.
>>>
>>> Something like this:
>>>
>>> alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded";
>>> flow:to_client,established; file_type:GIF; sid:1000000;)
>>> alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file downloaded";
>>> flow:to_client,established; file_type:PNG; sid:1000001;)
>>>
>>>
>>> You'll still need to have "file_magic.conf" included in your Snort
>>> configuration, but you will not need the file_identify preprocessor.
>>>
>>> ~Victor
>>>
>>> On 03/17/15 4:57, Pablo Cantos Polaino wrote:
>>>
>>> Hello all,
>>>
>>> I already sent this issue to snort-devel with the same subject since I am
>>> not sure if either I am configuring Snort in the right way or there is an
>>> internal malfunction to fix.
>>>
>>> I have been testing the new experimental preprocessor called File
>>> Services
>>> in order to get an event every time a file go through our network. To
>>> carry
>>> on these tests I have used two pcap files. The first one is a 1GB-size
>>> pcap
>>> with a great number of files and the second one is a short pcap generated
>>> on my computer when I downloaded a GIF file.
>>>
>>> My snort.conf file is configured like this at the end:
>>>
>>> include file_magic.conf
>>>
>>>   preprocessor file_inspect: type_id, signature, \
>>>               capture_queue_size 5000, \
>>>               capture_disk /home/file_capture/tmp/
>>>
>>>
>>>
>>> In both cases files are captured by the preprocessor, as you can see
>>> below
>>> (1GB pcap output):
>>>
>>> ============================================================
>>> ===================
>>>
>>>   Action Stats:
>>>       Alerts:            0 (  0.000%)
>>>       Logged:            0 (  0.000%)
>>>       Passed:            0 (  0.000%)
>>> Limits:
>>>        Match:            0
>>>        Queue:            0
>>>          Log:            0
>>>        Event:            0
>>>        Alert:            0
>>> Verdicts:
>>>        Allow:      8418451 ( 97.482%)
>>>        Block:            0 (  0.000%)
>>>      Replace:            0 (  0.000%)
>>>    Whitelist:       217492 (  2.518%)
>>>    Blacklist:            0 (  0.000%)
>>>       Ignore:            0 (  0.000%)
>>>
>>> ============================================================
>>> ===================
>>>
>>>
>>>
>>> ============================================================
>>> ===================
>>>
>>>   File Preprocessor Statistics
>>>    Total file type callbacks:            576
>>>    Total file signature callbacks:       578
>>>    Total files would saved to disk:      574
>>>    Total files saved to disk:            320
>>>    Total file data saved to disk:        483039    bytes
>>>    Total files duplicated:               254
>>>    Total files reserving failed:         2
>>>    Total file capture min:               0
>>>    Total file capture max:               2
>>>    Total file capture memcap:            0
>>>    Total files reading failed:           0
>>>    Total file agent memcap failures:     0
>>>    Total files sent:                     0
>>>    Total file data sent:                 0
>>>    Total file transfer failures:         0
>>>
>>> ============================================================
>>> ===================
>>> File type stats:
>>>           Type              Download   (Bytes)      Upload     (Bytes)
>>>            GZ( 33)          2          5580056      0          0
>>>           SWF( 52)          1          65991        0          0
>>>           GIF( 62)          7          16516        0          0
>>>           GIF( 63)          275        151718       0          0
>>>           PNG( 69)          266        256724       0          0
>>>          JPEG( 70)          2          35566        0          0
>>>           BMP(148)          2          4204         0          0
>>>           ICO(149)          21         187894       0          0
>>>              Total          576        6298669      0          0
>>> File signature stats:
>>>           Type              Download   Upload
>>>            GZ( 33)          2          0
>>>           SWF( 52)          1          0
>>>           GIF( 62)          7          0
>>>           GIF( 63)          275        0
>>>           PNG( 69)          266        0
>>>          JPEG( 70)          2          0
>>>           BMP(148)          2          0
>>>           ICO(149)          21         0
>>>              Total          576        0
>>> File type verdicts:
>>>          UNKNOWN:           576
>>>              LOG:           0
>>>             STOP:           0
>>>            BLOCK:           0
>>>           REJECT:           0
>>>          PENDING:           0
>>>     STOP CAPTURE:           0
>>>            Total:           576
>>> File signature verdicts:
>>>          UNKNOWN:           578
>>>              LOG:           0
>>>             STOP:           0
>>>            BLOCK:           0
>>>           REJECT:           0
>>>          PENDING:           0
>>>     STOP CAPTURE:           0
>>>            Total:           578
>>> Total files processed:             68985
>>> Total files data processed:        97156439  bytes
>>> Total files buffered:              576
>>> Total files released:              574
>>> Total files freed:                 2
>>> Total files captured:              574
>>> Total files within one packet:     561
>>> Total buffers allocated:           641
>>> Total buffers freed:               64
>>> Total buffers released:            577
>>> Maximum file buffers used:         64
>>> Total buffers free errors:         0
>>> Total buffers release errors:      0
>>> Total memcap failures:             0
>>> Total memcap failures at reserve:  0
>>> Total reserve failures:            0
>>> Total file capture size min:       0
>>> Total file capture size max:       0
>>> Total capture max before reserve:  2
>>> Total file signature max:          0
>>> Maximum buffers can allocate:      3196
>>> Number of buffers in use:          0
>>> Number of buffers in free list:    2619
>>> Number of buffers in release list: 577
>>>
>>> ============================================================
>>> ===================
>>>
>>>
>>>
>>> Following the instructions given as examples inside the file
>>> README.file, I
>>> have included the following rules to get an alert every time Snort
>>> detects
>>> a file:
>>>
>>> alert (msg: "GIF file"; gid:146; sid:63; rev:1; metadata: rule-type
>>>
>>>   preproc;)
>>>
>>>
>>>
>>> After that, no alert showed up.
>>>
>>> I went deep inside the code to find out what the reason is and found the
>>> following piece of code that confused me:
>>>
>>> snort/src/dynamic-preprocessors/file/file_agent.c:601-614
>>>
>>> /*
>>>
>>>    * File type callback when file type is identified
>>>   *
>>>   * For file capture or file signature, FILE_VERDICT_PENDING must be
>>> returned
>>>   */
>>> static File_Verdict file_agent_type_callback(void* p, void* ssnptr,
>>>          uint32_t file_type_id, bool upload, uint32_t file_id)
>>> {
>>>      file_inspect_stats.file_types_total++;
>>>      if (file_signature_enabled || file_capture_enabled)
>>>          return FILE_VERDICT_UNKNOWN;
>>>      else
>>>          return FILE_VERDICT_LOG;
>>> }
>>>
>>>
>>>
>>> You can read on the description that FILE_VERDICT_PENDING must be
>>> returned
>>> when file capture OR file signature is enabled, but what really the code
>>> does is to return FILE_VERDICT_UNKNOWN when capture or signature are
>>> enabled.
>>>
>>> After see that, I have modified the snort.conf by carrying on the
>>> following
>>> changes:
>>>
>>> Replace this:
>>>
>>> preprocessor file_inspect: type_id, signature, \
>>>
>>>                capture_queue_size 5000, \
>>>               capture_disk /home/file_capture/tmp/
>>>
>>>
>>>
>>> By:
>>>
>>> preprocessor file_inspect: type_id
>>>
>>>
>>> This way I forced to go through the ELSE and return a FILE_VERDICT_LOG.
>>> After this change, and using the same two alert rules, we run snort,
>>> getting alerts like these below:
>>>
>>> 03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0] {TCP}
>>>
>>>   192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/>
>>> -> 192.168.203.61:38976
>>> 03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0] {TCP}
>>> 192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/> ->
>>> 192.168.203.61:38976
>>> 03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0] {TCP}
>>> 192.168.202.78:80 <http://192.168.202.78/> <http://192.168.202.78/> ->
>>> 192.168.203.61:38977
>>>
>>>
>>> ...
>>>
>>>   and getting the following output at the end:
>>>
>>> ============================================================
>>> ===================
>>>
>>>   Action Stats:
>>>       Alerts:          275 (  0.003%)
>>>       Logged:          275 (  0.003%)
>>>       Passed:            0 (  0.000%)
>>> Limits:
>>>        Match:            0
>>>        Queue:            0
>>>          Log:            0
>>>        Event:            0
>>>        Alert:            0
>>> Verdicts:
>>>        Allow:      8418514 ( 97.482%)
>>>        Block:            0 (  0.000%)
>>>      Replace:            0 (  0.000%)
>>>    Whitelist:       217429 (  2.518%)
>>>    Blacklist:            0 (  0.000%)
>>>       Ignore:            0 (  0.000%)
>>>
>>> ============================================================
>>> ===================
>>>
>>>
>>>
>>>
>>> ============================================================
>>> ===================
>>> File Preprocessor Statistics
>>>    Total file type callbacks:            576
>>>    Total file signature callbacks:       0
>>>    Total files would saved to disk:      0
>>>    Total files saved to disk:            0
>>>    Total file data saved to disk:        0         bytes
>>>    Total files duplicated:               0
>>>    Total files reserving failed:         0
>>>    Total file capture min:               0
>>>    Total file capture max:               0
>>>    Total file capture memcap:            0
>>>    Total files reading failed:           0
>>>    Total file agent memcap failures:     0
>>>    Total files sent:                     0
>>>    Total file data sent:                 0
>>>    Total file transfer failures:         0
>>>
>>> ============================================================
>>> ===================
>>> File type stats:
>>>           Type              Download   (Bytes)      Upload     (Bytes)
>>>            GZ( 33)          2          0            0          0
>>>           SWF( 52)          1          0            0          0
>>>           GIF( 62)          7          0            0          0
>>>           GIF( 63)          275        0            0          0
>>>           PNG( 69)          266        0            0          0
>>>          JPEG( 70)          2          0            0          0
>>>           BMP(148)          2          0            0          0
>>>           ICO(149)          21         0            0          0
>>>              Total          576        0            0          0
>>> File signature stats:
>>>           Type              Download   Upload
>>>              Total          0          0
>>> File type verdicts:
>>>          UNKNOWN:           0
>>>              LOG:           576
>>>             STOP:           0
>>>            BLOCK:           0
>>>           REJECT:           0
>>>          PENDING:           0
>>>     STOP CAPTURE:           0
>>>            Total:           576
>>> File signature verdicts:
>>>          UNKNOWN:           0
>>>              LOG:           0
>>>             STOP:           0
>>>            BLOCK:           0
>>>           REJECT:           0
>>>          PENDING:           0
>>>     STOP CAPTURE:           0
>>>            Total:           0
>>> Total files processed:             68987
>>> Total files data processed:        42751396  bytes
>>> Total files buffered:              0
>>> Total files released:              0
>>> Total files freed:                 0
>>> Total files captured:              0
>>> Total files within one packet:     0
>>> Total buffers allocated:           0
>>> Total buffers freed:               0
>>> Total buffers released:            0
>>> Maximum file buffers used:         0
>>> Total buffers free errors:         0
>>> Total buffers release errors:      0
>>> Total memcap failures:             0
>>> Total memcap failures at reserve:  0
>>> Total reserve failures:            0
>>> Total file capture size min:       0
>>> Total file capture size max:       0
>>> Total capture max before reserve:  0
>>> Total file signature max:          0
>>>
>>> ============================================================
>>> ===================
>>>
>>>
>>>
>>> As you can see, in the "File type verdicts" section I got all the files
>>> with verdict LOG. Also, I got 275 alerts that match the 275 GIF files
>>> detected by Snort.
>>>
>>> I am not sure if this is the expected behavior of this feature or maybe I
>>> am not configuring Snort properly.
>>>
>>> Am I doing something wrong or configuring the preprocessor in a wrong
>>> way?
>>>
>>> Thanks for your help and best Regards,
>>>
>>> Pablo Cantosredborder.org / pcantos at ...16842...
>>>
>>>
>>>
>>> This body part will be downloaded on demand.
>>>
>>>
>>>
>>> This body part will be downloaded on demand.
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>> sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub
>>> for
>>> all
>>> things parallel software development, from weekly thought leadership
>>> blogs
>>> to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Snort-users mailing list
>>> Snort-users at lists.sourceforge.net
>>> Go to this URL to change user options or unsubscribe:
>>> https://lists.sourceforge.net/lists/listinfo/snort-users
>>> Snort-users list archive:
>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>>>
>>> Please visit http://blog.snort.org to stay current on all the latest
>>> Snort news!
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20150327/032dd11a/attachment.html>


More information about the Snort-users mailing list