[Snort-users] Getting alerts for every file Snort detects and File Services preprocessor

Pablo Cantos Polaino pcantos at ...16842...
Thu Apr 9 07:59:59 EDT 2015


Sure, here they are:

pcap: http://download.netresec.com/pcap/maccdc-2012/maccdc2012_00001.pcap.gz
snort.conf: (If you don't mind, I will attach you in a separated mail)

Best Regards,

Pablo Cantos
redborder.org / pcantos at ...16842...

2015-04-08 19:04 GMT+02:00 Victor Roemer <viroemer at ...589...>:

> Pablo, can you send me a pcap and the snort.conf(s) that show the
> inconsistent alert count?
>
>
> On 04/07/15 5:36, Pablo Cantos Polaino wrote:
>
>> Hi Victor,
>>
>> I've tried it again after configuring max_queue 100 and log 100 and adding
>> the flag "-H", getting the same result. I still get a volatile number of
>> alerts less than the total number of detected files.
>>
>> In any case, and please tell me if I'm wrong, since I'm just using 10
>> rules
>> and, in our test, is highly unlikely that more than one rule could affect
>> one single packet (a packet wouldn't contain more than one file), I think
>> maybe altering the "event_queue" configurable options wouldn't change
>> anything here.
>>
>> If we don't manage to solve this issue related to the number of alerts, we
>> will disable the signature and capture options and use the preprocessor,
>> and just in case we will need signature or capture options we will try to
>> do a workaround to not lose any functionality.
>>
>> Thanks for your help!
>>
>> Best Regards,
>>
>> Pablo Cantos
>> redborder.org / pcantos at ...16842...
>>
>> 2015-04-02 9:52 GMT+02:00 Pablo Cantos Polaino <pcantos at ...16842...>:
>>
>>  Thanks Victor! From today I will be out of the office until Monday. I'll
>>> try it then and let you know!
>>> El 1/4/2015 16:56, "Victor Roemer" <viroemer at ...589...> escribió:
>>>
>>>  Ok, I'm guessing that this is due to internal default's for event_queue
>>>> and the
>>>> RNG in rule evaluation ordering.
>>>>
>>>> Try adding the following to your snort.conf
>>>>
>>>> config event_queue: max_queue 100 log 100
>>>>
>>>> and adding the flag "-H" when running Snort.
>>>>
>>>>
>>>> The -H removes the RNG in Snort runtime, so using it in production would
>>>> not be
>>>> a good idea; we only use it for testing on our side.
>>>>
>>>>
>>>> On 04/01/15 9:18, Pablo Cantos Polaino wrote:
>>>>
>>>>  Hi Victor,
>>>>>
>>>>> Since there is no PDF files in my PCAP file, I've tried what you
>>>>> propose
>>>>> for the following file types: MSEXE, ZIP, GZ, SWF, GIF, PNG, JPEG, BMP
>>>>> and
>>>>> ICO.
>>>>>
>>>>> Using the file types below (getting from file_magic.conf) and the rules
>>>>> below I've got something quite coherent, but every time
>>>>> I run snort I get a different number of alerts, lower or equal to the
>>>>> right
>>>>> number or files. For instance, there are 566 files inside the PCAP file
>>>>> (constant number got from the File preprocessor) and I've got volatile
>>>>> number of alerts like 558, 556, 557, ...
>>>>>
>>>>> File types:
>>>>> file type:MSEXE; id:21; category:Executables,Dynamic Analysis Capable;
>>>>> msg:"Windows/DOS executable file "; rev:1; content:| 4D 5A|; offset:0;
>>>>> file type:ZIP; id:29; category:Archive; msg:"PKZIP archive file";
>>>>> rev:1;
>>>>> content:| 50 4B 03 04 |; offset:0;
>>>>> file type:GZ; id:33; category:Archive; msg:"GZ"; rev:1; content:| 1F 8B
>>>>> 08
>>>>> |; offset:0;
>>>>> file type:SWF; id:52; category:Multimedia; msg:"Flash file "; rev:1;
>>>>> content:| 43 57 53 |; offset:0;
>>>>> file type:GIF; id:62; category:Graphics; msg:"GIF"; rev:1; content:| 47
>>>>> 49
>>>>> 46 38 37 61 |; offset:0; group:multimedia;
>>>>> file type:GIF; id:63; category:Graphics; msg:"GIF"; rev:1; content:| 47
>>>>> 49
>>>>> 46 38 39 61 |; offset:0; group:multimedia;
>>>>> file type:PNG; id:69; category:Graphics; msg:"Portable Network Graphics
>>>>> file"; rev:1; content:| 89 50 4E 47 0D 0A 1A 0A |; offset:0;
>>>>> group:multimedia;
>>>>> file type:JPEG; id:70; category:Graphics; msg:"JPEG/JFIF graphics
>>>>> file";
>>>>> rev:1; content:| FF D8 FF E0 |; offset:0; group:multimedia;
>>>>> file type:BMP; id:148; category:Graphics; msg:"Bitmap image file";
>>>>> rev:1;
>>>>> content:|42  4D |; offset:0; group:multimedia;
>>>>> file type:ICO; id:149; category:Graphics; msg:"Windows icon file";
>>>>> rev:1;
>>>>> content:| 00 00 01 00 |; offset:0;
>>>>>
>>>>> Rules:
>>>>> alert tcp any any -> any any (msg:"MSEXE: MZ"; content:"|4D 5A|";
>>>>> offset:0;
>>>>> file_type:MSEXE; sid:5000001;)
>>>>> alert tcp any any -> any any (msg:"ZIP: PK"; content:"|50 4B 03 04|";
>>>>> offset:0; file_type:ZIP; sid:5000002;)
>>>>> alert tcp any any -> any any (msg:"GZ: "; content:"|1F 8B 08|";
>>>>> offset:0;
>>>>> file_type:GZ; sid:5000003;)
>>>>> alert tcp any any -> any any (msg:"SWF: "; content:"|43 57 53|";
>>>>> offset:0;
>>>>> file_type:SWF; sid:5000004;)
>>>>> alert tcp any any -> any any (msg:"GIF: GIF87a"; content:"|47 49 46 38
>>>>> 37
>>>>> 61|"; offset:0; file_type:GIF; sid:5000005;)
>>>>> alert tcp any any -> any any (msg:"GIF: GIF89a"; content:"|47 49 46 38
>>>>> 39
>>>>> 61|"; offset:0; file_type:GIF; sid:5000006;)
>>>>> alert tcp any any -> any any (msg:"PNG: "; content:"|89 50 4E 47 0D 0A
>>>>> 1A
>>>>> 0A|"; offset:0; file_type:PNG; sid:5000007;)
>>>>> alert tcp any any -> any any (msg:"JPEG: "; content:"|FF D8 FF E0|";
>>>>> offset:0; file_type:JPEG; sid:5000008;)
>>>>> alert tcp any any -> any any (msg:"BMP: "; content:"|42 4D|"; offset:0;
>>>>> file_type:BMP; sid:5000009;)
>>>>> alert tcp any any -> any any (msg:"ICO: "; content:"|00 00 01 00|";
>>>>> offset:0; file_type:ICO; sid:5000010;)
>>>>>
>>>>> If I use the rules below (going through the File preprocessor) I always
>>>>> get
>>>>> the same number of alerts that's equal to the number of files (566).
>>>>>
>>>>> alert (msg: "MSEXE file"; gid:146; sid:21;)
>>>>> alert (msg: "ZIP file"; gid:146; sid:29;)
>>>>> alert (msg: "GZ file"; gid:146; sid:33;)
>>>>> alert (msg: "SWF file"; gid:146; sid:52;)
>>>>> alert (msg: "GIF(62) file"; gid:146; sid:62;)
>>>>> alert (msg: "GIF(63) file"; gid:146; sid:63;)
>>>>> alert (msg: "PNG file"; gid:146; sid:69;)
>>>>> alert (msg: "JPEG file"; gid:146; sid:70;)
>>>>> alert (msg: "BMP file"; gid:146; sid:148;)
>>>>> alert (msg: "ICO file"; gid:146; sid:149;)
>>>>>
>>>>> I've also tried to detect just one type of files and the results are
>>>>> similar, when I use the way I get a volatile number of alerts, but I
>>>>> manage
>>>>> to get the same number of alerts if I follow the second way.
>>>>>
>>>>> To us the second way is more reliable since we don't want to skip any
>>>>> file
>>>>> detected by File preprocessor. On the other hand, we have also managed
>>>>> to
>>>>> log these alerts to unified2, so this would not be a disadvantage.
>>>>>
>>>>>  Ok, for some reason I believed that u2 logging would be possible.
>>>>
>>>> If you would like to use the preprocessor events that is fine; but as
>>>> you've already identified, the
>>>> file type events do not work when signature or capture is enabled. Until
>>>> we fix that, you'd have
>>>> to resolve that in your own version of the file_inspect preprocessor or
>>>> wait for us to address it.
>>>> (The bug was written, but is currently unknown when we will work on it)
>>>>
>>>>   Best Regards,
>>>>
>>>>> Pablo Cantos
>>>>> redborder.org / pcantos at ...16842...
>>>>>
>>>>> 2015-03-31 17:54 GMT+02:00 Victor Roemer <viroemer at ...589...>:
>>>>>
>>>>>   Pablo,
>>>>>
>>>>>> Yes, your right. Flowbits apply per-session so the rule would only be
>>>>>> capable of alerting 1x (per-session).
>>>>>>
>>>>>> In that case, you could also just do a rule like so:
>>>>>>
>>>>>> file type:PDF; content:|25 50 44 46 2d|; offset:0; id:1;
>>>>>> alert tcp any any -> any any (msg:"PDF"; content:"%PDF-"; offset:0;
>>>>>> file_type:PDF; sid:1000002;)
>>>>>>
>>>>>> What I've done is removed the flowbits and added offset:0; from the
>>>>>> previous rule. By matching
>>>>>> the content that triggers the file rule to identify PDF, this rule
>>>>>> should
>>>>>> alert on the start of every
>>>>>> PDF file seen.
>>>>>>
>>>>>> On 03/31/15 7:19, Pablo Cantos Polaino wrote:
>>>>>>
>>>>>>   Hi Victor,
>>>>>>
>>>>>>> We already thought about using flowbits, but (and please correct me
>>>>>>> if
>>>>>>> I'm
>>>>>>> wrong) we didn't considered it effective since once a bit is set
>>>>>>> after
>>>>>>> detecting a file, this rule will never trigger any alert. I've
>>>>>>> launched
>>>>>>> some tests following your proposals and the numbers I got are not
>>>>>>> much
>>>>>>> coherent. For these tests I've used a capture with 107 GIF files
>>>>>>> inside.
>>>>>>> The file preprocessor is able to detect these 107 files and below you
>>>>>>> could
>>>>>>> find the number of alerts I get using the following rules:
>>>>>>>
>>>>>>> Rules:
>>>>>>> alert (msg: "GIF(62) file"; gid:146; sid:62;)
>>>>>>> alert (msg: "GIF(63) file"; gid:146; sid:63;)
>>>>>>> Number of alerts: 107
>>>>>>>
>>>>>>> Rule:
>>>>>>> alert tcp any any -> any any (msg: "GIF file"; file_type:GIF;
>>>>>>> sid:3000011;
>>>>>>> rev:1;)
>>>>>>> Number of alerts: 111 (4 alerts more than the number of files
>>>>>>> detected.
>>>>>>> Possibly due to one or more than one file takes up one or more
>>>>>>> packet)
>>>>>>>
>>>>>>> Rule:
>>>>>>> alert tcp any any -> any any (msg: "GIF file";
>>>>>>> flow:to_client,established;
>>>>>>> file_type:GIF; sid:3000013; rev:1;)
>>>>>>> Number of alerts: 24 (much less alerts than the number of files
>>>>>>> detected.
>>>>>>> Possibly due to the direction flow, but as you can see below, adding
>>>>>>> 24
>>>>>>> and
>>>>>>> 25 we don't get the total number of files)
>>>>>>>
>>>>>>> Rule:
>>>>>>> alert tcp any any -> any any (msg: "GIF file";
>>>>>>> flow:to_server,established;
>>>>>>> file_type:GIF; sid:3000014; rev:1;)
>>>>>>> Number of alerts: 25
>>>>>>>
>>>>>>> Rule:
>>>>>>> alert tcp any any -> any any (msg: "GIF file flowbits";
>>>>>>> flowbits:isnotset,gif; file_type:GIF; flowbits:set,gif; sid:4000011;
>>>>>>> rev:1;)
>>>>>>> Number of alerts: 2
>>>>>>>
>>>>>>> Rule:
>>>>>>> alert tcp any any -> any any (msg: "GIF file flowbits";
>>>>>>> flowbits:isnotset,gif; flow:to_client,established; file_type:GIF;
>>>>>>> flowbits:set,gif; sid:4000013; rev:1;)
>>>>>>> Number of alerts: 20
>>>>>>>
>>>>>>> Rule:
>>>>>>> alert tcp any any -> any any (msg: "GIF file flowbits";
>>>>>>> flowbits:isnotset,gif; flow:to_server,established; file_type:GIF;
>>>>>>> flowbits:set,gif; sid:4000014; rev:1;)
>>>>>>> Number of alerts: 20
>>>>>>>
>>>>>>> As you can see, just using the two first rules which employ directly
>>>>>>> the
>>>>>>> File preprocessor (rules with gid:146) gives us just one alert per
>>>>>>> file,
>>>>>>> and this is what we are interested in. This way we can be sure that,
>>>>>>> through the alerts, we are getting information from every file
>>>>>>> detected in
>>>>>>> File preprocessor. The only thing we have to put care into is
>>>>>>> configuring
>>>>>>> properly the preprocessor like this below:
>>>>>>>
>>>>>>> preprocessor file_inspect: type_id
>>>>>>>
>>>>>>> and await the related bug could be fixed.
>>>>>>>
>>>>>>> In a previous mail you said "The "file_identify" preprocessor is
>>>>>>> designed
>>>>>>> to work more as a "reputation" based system. I think you will get
>>>>>>> better
>>>>>>> millage by using the new "file_type" rule keyword in a plain-old
>>>>>>> snort
>>>>>>> rule". Our conclusion is quite opposite to your recommendation, whose
>>>>>>> grounds we didn't understand, I mean, why is preferable using the
>>>>>>> file_type
>>>>>>> keyword in a plain-old snort rule instead of using a rule with
>>>>>>> gid:146?
>>>>>>>
>>>>>>> I hope I explained my self.
>>>>>>>
>>>>>>>   * Because it doesn't work the way you want it to is 'one' reason.
>>>>>>>
>>>>>> * Text rules are more managable and less error prone than custom
>>>>>> preprocessors.
>>>>>> * You can take advantage of logging to unified2 vs the one-the-side
>>>>>> mechanism currently employed
>>>>>>      by the file preprocessor.
>>>>>>
>>>>>> But specifically, your intentions were not known to me when I sent
>>>>>> that. :/
>>>>>>
>>>>>>
>>>>>>   Best Regards,
>>>>>>
>>>>>>> Pablo Cantos
>>>>>>> redborder.org / pcantos at ...16842...
>>>>>>>
>>>>>>> 2015-03-30 19:22 GMT+02:00 Victor Roemer <viroemer at ...589...>:
>>>>>>>
>>>>>>>    The solution too the "too many alerts" can be resolved with the
>>>>>>>
>>>>>>>  application of "flowbits".
>>>>>>>>
>>>>>>>> For example:
>>>>>>>>
>>>>>>>> file type:PDF; content:|25 50 44 46 2d|; offset:0; id:1;
>>>>>>>> alert tcp any any -> any any (msg:"PDF"; flowbits:isnotset,pdf;
>>>>>>>> file_type:PDF; flowbits:set,pdf; sid:1000000;)
>>>>>>>>
>>>>>>>> However, it is also recommended that you add a "content" option too
>>>>>>>> the
>>>>>>>> rule so that you may take advantage of the fast pattern matcher.
>>>>>>>> This may seem odd, but it will be faster. In the next rule, the
>>>>>>>> content
>>>>>>>> being matched "mirrors" the content in the "file" rule above
>>>>>>>> exactly.
>>>>>>>>
>>>>>>>> alert tcp any any -> any any (msg:"PDF"; flowbits:isnotset,pdf;
>>>>>>>> content:"%PDF-"; file_type:PDF; flowbits:set,pdf; sid:1000001;)
>>>>>>>>
>>>>>>>>
>>>>>>>> ~Victor
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/30/15 5:53, Pablo Cantos Polaino wrote:
>>>>>>>>
>>>>>>>>    Hi Victor,
>>>>>>>>
>>>>>>>>  First of all, thanks for your reply!
>>>>>>>>>
>>>>>>>>> Regarding your proposal below:
>>>>>>>>>
>>>>>>>>> The "file_identify" preprocessor is designed to work more as a
>>>>>>>>> "reputation"
>>>>>>>>>
>>>>>>>>>    based system. I think you will get better millage by using the
>>>>>>>>> new
>>>>>>>>>
>>>>>>>>>  "file_type" rule keyword in a plain-old snort rule.
>>>>>>>>>> Something like this:
>>>>>>>>>> alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file
>>>>>>>>>> downloaded";
>>>>>>>>>> flow:to_client,established; file_type:GIF; sid:1000000;)
>>>>>>>>>> alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file
>>>>>>>>>> downloaded";
>>>>>>>>>> flow:to_client,established; file_type:PNG; sid:1000001;)
>>>>>>>>>>
>>>>>>>>>>    We already tried that and we got a lot of alerts. The sizer the
>>>>>>>>>> file
>>>>>>>>>>
>>>>>>>>>>  is
>>>>>>>>> the
>>>>>>>>> more alerts are fired, and this is not the behavior we expected
>>>>>>>>> since we
>>>>>>>>> would overpopulate the disk with repeated events. For instance,
>>>>>>>>> after
>>>>>>>>> downloading a BZ file with size of 7,5MB, we've got 855 alerts from
>>>>>>>>> one
>>>>>>>>> just rule:
>>>>>>>>>
>>>>>>>>> alert tcp any any -> any any (msg: "BZ file";
>>>>>>>>> flow:to_client,established;
>>>>>>>>> file_type:BZ; sid:3000033;)
>>>>>>>>>
>>>>>>>>> And we get just one alert if we use the rule below and the
>>>>>>>>> following
>>>>>>>>> configuration:
>>>>>>>>>
>>>>>>>>> Conf (just type_id, neither signature nor capture)
>>>>>>>>>
>>>>>>>>> preprocessor file_inspect: type_id
>>>>>>>>>
>>>>>>>>> Rule:
>>>>>>>>>
>>>>>>>>> alert (msg: "BZ file"; gid:146; sid:32;)
>>>>>>>>>
>>>>>>>>> This is why we discarded some weeks ago the way of writing the rule
>>>>>>>>> as
>>>>>>>>> you
>>>>>>>>> proposed now. If it were a way to get just one alert per file it
>>>>>>>>> would
>>>>>>>>> be
>>>>>>>>> great. Meanwhile, we await the bug can be fixed.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>>
>>>>>>>>> Pablo Cantos
>>>>>>>>> redborder.org / pcantos at ...16842...
>>>>>>>>>
>>>>>>>>> 2015-03-27 22:46 GMT+01:00 Jaime Nebrera <jnebrera at ...16842...>:
>>>>>>>>>
>>>>>>>>>     Hi Víctor,
>>>>>>>>>
>>>>>>>>>   I'm not that technical ;) Pablo will coment on your ideas, I just
>>>>>>>>>
>>>>>>>>>> tried
>>>>>>>>>> to
>>>>>>>>>> give you a deeper context of what Pablo (as part of my team) was
>>>>>>>>>> doing
>>>>>>>>>> but
>>>>>>>>>> for sure, they are welcomed. I let them continue
>>>>>>>>>> El 27/03/2015 21:25, "Victor Roemer" <viroemer at ...589...>
>>>>>>>>>> escribió:
>>>>>>>>>>
>>>>>>>>>>     Jamie,
>>>>>>>>>>
>>>>>>>>>>   That's neat, I'll keep an eye on it. :)
>>>>>>>>>>
>>>>>>>>>>> Reviewing over how the "file_identify" works currently, when
>>>>>>>>>>> doing
>>>>>>>>>>> capture or signature
>>>>>>>>>>> its not possible to also get the file type event via the
>>>>>>>>>>> preprocessor.
>>>>>>>>>>> I'll open an bug for this.
>>>>>>>>>>>
>>>>>>>>>>> In order to get the file type events with these further settings,
>>>>>>>>>>> more
>>>>>>>>>>> sophisticated tracking
>>>>>>>>>>> needs to be in-place in the file_identify preproc; the tricky
>>>>>>>>>>> bits
>>>>>>>>>>> being
>>>>>>>>>>> that the file service code
>>>>>>>>>>> is generating the events in question; however once a verdict is
>>>>>>>>>>> passed
>>>>>>>>>>> to
>>>>>>>>>>> file service- an
>>>>>>>>>>> action will be taken at that time.
>>>>>>>>>>>
>>>>>>>>>>> To clarify the verdicts -
>>>>>>>>>>> As I understand it, passing FILE_VERDICT_UNKNOWN is how
>>>>>>>>>>> processing
>>>>>>>>>>> continues further.
>>>>>>>>>>> The FILE_VERDICT_PENDING when passed to file service api, should
>>>>>>>>>>> cause
>>>>>>>>>>> the packet to be
>>>>>>>>>>> dropped, in-order to give the interfacing code time to retrieve a
>>>>>>>>>>> true
>>>>>>>>>>> verdict when the re-transmitted
>>>>>>>>>>> packet is seen.
>>>>>>>>>>>
>>>>>>>>>>> On 03/27/15 16:04, Jaime Nebrera wrote:
>>>>>>>>>>>
>>>>>>>>>>>     Hi Víctor,
>>>>>>>>>>>
>>>>>>>>>>>   Pablo's work is geared towards a more ambitious goal:
>>>>>>>>>>>
>>>>>>>>>>>> 1) Intercept the file and send it to a S3 based storage platform
>>>>>>>>>>>> for
>>>>>>>>>>>> further analysis or whatever
>>>>>>>>>>>>
>>>>>>>>>>>> 2) Provide as much context about such interception as possible,
>>>>>>>>>>>> like
>>>>>>>>>>>> from
>>>>>>>>>>>> were to whom, URL, email, etc
>>>>>>>>>>>>
>>>>>>>>>>>> This we hope to open source and make it public as soon as it is
>>>>>>>>>>>> usable
>>>>>>>>>>>> in
>>>>>>>>>>>> our repository www.github.com/redborder
>>>>>>>>>>>> El 27/03/2015 20:50, "Victor Roemer" <viroemer at ...589...>
>>>>>>>>>>>> escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>        Pablo,
>>>>>>>>>>>>
>>>>>>>>>>>>    Sorry for the delay. The "file_identify" preprocessor is
>>>>>>>>>>>> designed to
>>>>>>>>>>>>
>>>>>>>>>>>>  work
>>>>>>>>>>>>> more as a "reputation" based system. I think you will get
>>>>>>>>>>>>> better
>>>>>>>>>>>>> millage by
>>>>>>>>>>>>> using the new "file_type" rule keyword in a plain-old snort
>>>>>>>>>>>>> rule.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Something like this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file
>>>>>>>>>>>>> downloaded";
>>>>>>>>>>>>> flow:to_client,established; file_type:GIF; sid:1000000;)
>>>>>>>>>>>>> alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"GIF file
>>>>>>>>>>>>> downloaded";
>>>>>>>>>>>>> flow:to_client,established; file_type:PNG; sid:1000001;)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> You'll still need to have "file_magic.conf" included in your
>>>>>>>>>>>>> Snort
>>>>>>>>>>>>> configuration, but you will not need the file_identify
>>>>>>>>>>>>> preprocessor.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ~Victor
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/17/15 4:57, Pablo Cantos Polaino wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I already sent this issue to snort-devel with the same subject
>>>>>>>>>>>>> since I
>>>>>>>>>>>>> am
>>>>>>>>>>>>> not sure if either I am configuring Snort in the right way or
>>>>>>>>>>>>> there
>>>>>>>>>>>>> is
>>>>>>>>>>>>> an
>>>>>>>>>>>>> internal malfunction to fix.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have been testing the new experimental preprocessor called
>>>>>>>>>>>>> File
>>>>>>>>>>>>> Services
>>>>>>>>>>>>> in order to get an event every time a file go through our
>>>>>>>>>>>>> network.
>>>>>>>>>>>>> To
>>>>>>>>>>>>> carry
>>>>>>>>>>>>> on these tests I have used two pcap files. The first one is a
>>>>>>>>>>>>> 1GB-size
>>>>>>>>>>>>> pcap
>>>>>>>>>>>>> with a great number of files and the second one is a short pcap
>>>>>>>>>>>>> generated
>>>>>>>>>>>>> on my computer when I downloaded a GIF file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My snort.conf file is configured like this at the end:
>>>>>>>>>>>>>
>>>>>>>>>>>>> include file_magic.conf
>>>>>>>>>>>>>
>>>>>>>>>>>>>       preprocessor file_inspect: type_id, signature, \
>>>>>>>>>>>>>                   capture_queue_size 5000, \
>>>>>>>>>>>>>                   capture_disk /home/file_capture/tmp/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In both cases files are captured by the preprocessor, as you
>>>>>>>>>>>>> can
>>>>>>>>>>>>> see
>>>>>>>>>>>>> below
>>>>>>>>>>>>> (1GB pcap output):
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>>
>>>>>>>>>>>>>       Action Stats:
>>>>>>>>>>>>>           Alerts:            0 (  0.000%)
>>>>>>>>>>>>>           Logged:            0 (  0.000%)
>>>>>>>>>>>>>           Passed:            0 (  0.000%)
>>>>>>>>>>>>> Limits:
>>>>>>>>>>>>>            Match:            0
>>>>>>>>>>>>>            Queue:            0
>>>>>>>>>>>>>              Log:            0
>>>>>>>>>>>>>            Event:            0
>>>>>>>>>>>>>            Alert:            0
>>>>>>>>>>>>> Verdicts:
>>>>>>>>>>>>>            Allow:      8418451 ( 97.482%)
>>>>>>>>>>>>>            Block:            0 (  0.000%)
>>>>>>>>>>>>>          Replace:            0 (  0.000%)
>>>>>>>>>>>>>        Whitelist:       217492 (  2.518%)
>>>>>>>>>>>>>        Blacklist:            0 (  0.000%)
>>>>>>>>>>>>>           Ignore:            0 (  0.000%)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>>
>>>>>>>>>>>>>       File Preprocessor Statistics
>>>>>>>>>>>>>        Total file type callbacks:            576
>>>>>>>>>>>>>        Total file signature callbacks:       578
>>>>>>>>>>>>>        Total files would saved to disk:      574
>>>>>>>>>>>>>        Total files saved to disk:            320
>>>>>>>>>>>>>        Total file data saved to disk:        483039    bytes
>>>>>>>>>>>>>        Total files duplicated:               254
>>>>>>>>>>>>>        Total files reserving failed:         2
>>>>>>>>>>>>>        Total file capture min:               0
>>>>>>>>>>>>>        Total file capture max:               2
>>>>>>>>>>>>>        Total file capture memcap:            0
>>>>>>>>>>>>>        Total files reading failed:           0
>>>>>>>>>>>>>        Total file agent memcap failures:     0
>>>>>>>>>>>>>        Total files sent:                     0
>>>>>>>>>>>>>        Total file data sent:                 0
>>>>>>>>>>>>>        Total file transfer failures:         0
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>> File type stats:
>>>>>>>>>>>>>               Type              Download   (Bytes)      Upload
>>>>>>>>>>>>>     (Bytes)
>>>>>>>>>>>>>                GZ( 33)          2          5580056      0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               SWF( 52)          1          65991        0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               GIF( 62)          7          16516        0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               GIF( 63)          275        151718       0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               PNG( 69)          266        256724       0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>              JPEG( 70)          2          35566        0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               BMP(148)          2          4204         0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               ICO(149)          21         187894       0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>                  Total          576        6298669      0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>> File signature stats:
>>>>>>>>>>>>>               Type              Download   Upload
>>>>>>>>>>>>>                GZ( 33)          2          0
>>>>>>>>>>>>>               SWF( 52)          1          0
>>>>>>>>>>>>>               GIF( 62)          7          0
>>>>>>>>>>>>>               GIF( 63)          275        0
>>>>>>>>>>>>>               PNG( 69)          266        0
>>>>>>>>>>>>>              JPEG( 70)          2          0
>>>>>>>>>>>>>               BMP(148)          2          0
>>>>>>>>>>>>>               ICO(149)          21         0
>>>>>>>>>>>>>                  Total          576        0
>>>>>>>>>>>>> File type verdicts:
>>>>>>>>>>>>>              UNKNOWN:           576
>>>>>>>>>>>>>                  LOG:           0
>>>>>>>>>>>>>                 STOP:           0
>>>>>>>>>>>>>                BLOCK:           0
>>>>>>>>>>>>>               REJECT:           0
>>>>>>>>>>>>>              PENDING:           0
>>>>>>>>>>>>>         STOP CAPTURE:           0
>>>>>>>>>>>>>                Total:           576
>>>>>>>>>>>>> File signature verdicts:
>>>>>>>>>>>>>              UNKNOWN:           578
>>>>>>>>>>>>>                  LOG:           0
>>>>>>>>>>>>>                 STOP:           0
>>>>>>>>>>>>>                BLOCK:           0
>>>>>>>>>>>>>               REJECT:           0
>>>>>>>>>>>>>              PENDING:           0
>>>>>>>>>>>>>         STOP CAPTURE:           0
>>>>>>>>>>>>>                Total:           578
>>>>>>>>>>>>> Total files processed:             68985
>>>>>>>>>>>>> Total files data processed:        97156439  bytes
>>>>>>>>>>>>> Total files buffered:              576
>>>>>>>>>>>>> Total files released:              574
>>>>>>>>>>>>> Total files freed:                 2
>>>>>>>>>>>>> Total files captured:              574
>>>>>>>>>>>>> Total files within one packet:     561
>>>>>>>>>>>>> Total buffers allocated:           641
>>>>>>>>>>>>> Total buffers freed:               64
>>>>>>>>>>>>> Total buffers released:            577
>>>>>>>>>>>>> Maximum file buffers used:         64
>>>>>>>>>>>>> Total buffers free errors:         0
>>>>>>>>>>>>> Total buffers release errors:      0
>>>>>>>>>>>>> Total memcap failures:             0
>>>>>>>>>>>>> Total memcap failures at reserve:  0
>>>>>>>>>>>>> Total reserve failures:            0
>>>>>>>>>>>>> Total file capture size min:       0
>>>>>>>>>>>>> Total file capture size max:       0
>>>>>>>>>>>>> Total capture max before reserve:  2
>>>>>>>>>>>>> Total file signature max:          0
>>>>>>>>>>>>> Maximum buffers can allocate:      3196
>>>>>>>>>>>>> Number of buffers in use:          0
>>>>>>>>>>>>> Number of buffers in free list:    2619
>>>>>>>>>>>>> Number of buffers in release list: 577
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Following the instructions given as examples inside the file
>>>>>>>>>>>>> README.file, I
>>>>>>>>>>>>> have included the following rules to get an alert every time
>>>>>>>>>>>>> Snort
>>>>>>>>>>>>> detects
>>>>>>>>>>>>> a file:
>>>>>>>>>>>>>
>>>>>>>>>>>>> alert (msg: "GIF file"; gid:146; sid:63; rev:1; metadata:
>>>>>>>>>>>>> rule-type
>>>>>>>>>>>>>
>>>>>>>>>>>>>       preproc;)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> After that, no alert showed up.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I went deep inside the code to find out what the reason is and
>>>>>>>>>>>>> found
>>>>>>>>>>>>> the
>>>>>>>>>>>>> following piece of code that confused me:
>>>>>>>>>>>>>
>>>>>>>>>>>>> snort/src/dynamic-preprocessors/file/file_agent.c:601-614
>>>>>>>>>>>>>
>>>>>>>>>>>>> /*
>>>>>>>>>>>>>
>>>>>>>>>>>>>        * File type callback when file type is identified
>>>>>>>>>>>>>       *
>>>>>>>>>>>>>       * For file capture or file signature,
>>>>>>>>>>>>> FILE_VERDICT_PENDING
>>>>>>>>>>>>> must
>>>>>>>>>>>>> be
>>>>>>>>>>>>> returned
>>>>>>>>>>>>>       */
>>>>>>>>>>>>> static File_Verdict file_agent_type_callback(void* p, void*
>>>>>>>>>>>>> ssnptr,
>>>>>>>>>>>>>              uint32_t file_type_id, bool upload, uint32_t
>>>>>>>>>>>>> file_id)
>>>>>>>>>>>>> {
>>>>>>>>>>>>>          file_inspect_stats.file_types_total++;
>>>>>>>>>>>>>          if (file_signature_enabled || file_capture_enabled)
>>>>>>>>>>>>>              return FILE_VERDICT_UNKNOWN;
>>>>>>>>>>>>>          else
>>>>>>>>>>>>>              return FILE_VERDICT_LOG;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can read on the description that FILE_VERDICT_PENDING must
>>>>>>>>>>>>> be
>>>>>>>>>>>>> returned
>>>>>>>>>>>>> when file capture OR file signature is enabled, but what really
>>>>>>>>>>>>> the
>>>>>>>>>>>>> code
>>>>>>>>>>>>> does is to return FILE_VERDICT_UNKNOWN when capture or
>>>>>>>>>>>>> signature
>>>>>>>>>>>>> are
>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>
>>>>>>>>>>>>> After see that, I have modified the snort.conf by carrying on
>>>>>>>>>>>>> the
>>>>>>>>>>>>> following
>>>>>>>>>>>>> changes:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Replace this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> preprocessor file_inspect: type_id, signature, \
>>>>>>>>>>>>>
>>>>>>>>>>>>>                    capture_queue_size 5000, \
>>>>>>>>>>>>>                   capture_disk /home/file_capture/tmp/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> By:
>>>>>>>>>>>>>
>>>>>>>>>>>>> preprocessor file_inspect: type_id
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This way I forced to go through the ELSE and return a
>>>>>>>>>>>>> FILE_VERDICT_LOG.
>>>>>>>>>>>>> After this change, and using the same two alert rules, we run
>>>>>>>>>>>>> snort,
>>>>>>>>>>>>> getting alerts like these below:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0]
>>>>>>>>>>>>> {TCP}
>>>>>>>>>>>>>
>>>>>>>>>>>>>       192.168.202.78:80 <http://192.168.202.78/> <
>>>>>>>>>>>>> http://192.168.202.78/
>>>>>>>>>>>>> -> 192.168.203.61:38976
>>>>>>>>>>>>> 03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0]
>>>>>>>>>>>>> {TCP}
>>>>>>>>>>>>> 192.168.202.78:80 <http://192.168.202.78/> <
>>>>>>>>>>>>> http://192.168.202.78/>
>>>>>>>>>>>>> ->
>>>>>>>>>>>>> 192.168.203.61:38976
>>>>>>>>>>>>> 03/16-12:50:22.350000  [**] [146:63:1] GIF [**] [Priority: 0]
>>>>>>>>>>>>> {TCP}
>>>>>>>>>>>>> 192.168.202.78:80 <http://192.168.202.78/> <
>>>>>>>>>>>>> http://192.168.202.78/>
>>>>>>>>>>>>> ->
>>>>>>>>>>>>> 192.168.203.61:38977
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>       and getting the following output at the end:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>>
>>>>>>>>>>>>>       Action Stats:
>>>>>>>>>>>>>           Alerts:          275 (  0.003%)
>>>>>>>>>>>>>           Logged:          275 (  0.003%)
>>>>>>>>>>>>>           Passed:            0 (  0.000%)
>>>>>>>>>>>>> Limits:
>>>>>>>>>>>>>            Match:            0
>>>>>>>>>>>>>            Queue:            0
>>>>>>>>>>>>>              Log:            0
>>>>>>>>>>>>>            Event:            0
>>>>>>>>>>>>>            Alert:            0
>>>>>>>>>>>>> Verdicts:
>>>>>>>>>>>>>            Allow:      8418514 ( 97.482%)
>>>>>>>>>>>>>            Block:            0 (  0.000%)
>>>>>>>>>>>>>          Replace:            0 (  0.000%)
>>>>>>>>>>>>>        Whitelist:       217429 (  2.518%)
>>>>>>>>>>>>>        Blacklist:            0 (  0.000%)
>>>>>>>>>>>>>           Ignore:            0 (  0.000%)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>> File Preprocessor Statistics
>>>>>>>>>>>>>        Total file type callbacks:            576
>>>>>>>>>>>>>        Total file signature callbacks:       0
>>>>>>>>>>>>>        Total files would saved to disk:      0
>>>>>>>>>>>>>        Total files saved to disk:            0
>>>>>>>>>>>>>        Total file data saved to disk:        0         bytes
>>>>>>>>>>>>>        Total files duplicated:               0
>>>>>>>>>>>>>        Total files reserving failed:         0
>>>>>>>>>>>>>        Total file capture min:               0
>>>>>>>>>>>>>        Total file capture max:               0
>>>>>>>>>>>>>        Total file capture memcap:            0
>>>>>>>>>>>>>        Total files reading failed:           0
>>>>>>>>>>>>>        Total file agent memcap failures:     0
>>>>>>>>>>>>>        Total files sent:                     0
>>>>>>>>>>>>>        Total file data sent:                 0
>>>>>>>>>>>>>        Total file transfer failures:         0
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>> File type stats:
>>>>>>>>>>>>>               Type              Download   (Bytes)      Upload
>>>>>>>>>>>>>     (Bytes)
>>>>>>>>>>>>>                GZ( 33)          2          0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               SWF( 52)          1          0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               GIF( 62)          7          0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               GIF( 63)          275        0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               PNG( 69)          266        0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>              JPEG( 70)          2          0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               BMP(148)          2          0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>               ICO(149)          21         0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>>                  Total          576        0            0
>>>>>>>>>>>>>    0
>>>>>>>>>>>>> File signature stats:
>>>>>>>>>>>>>               Type              Download   Upload
>>>>>>>>>>>>>                  Total          0          0
>>>>>>>>>>>>> File type verdicts:
>>>>>>>>>>>>>              UNKNOWN:           0
>>>>>>>>>>>>>                  LOG:           576
>>>>>>>>>>>>>                 STOP:           0
>>>>>>>>>>>>>                BLOCK:           0
>>>>>>>>>>>>>               REJECT:           0
>>>>>>>>>>>>>              PENDING:           0
>>>>>>>>>>>>>         STOP CAPTURE:           0
>>>>>>>>>>>>>                Total:           576
>>>>>>>>>>>>> File signature verdicts:
>>>>>>>>>>>>>              UNKNOWN:           0
>>>>>>>>>>>>>                  LOG:           0
>>>>>>>>>>>>>                 STOP:           0
>>>>>>>>>>>>>                BLOCK:           0
>>>>>>>>>>>>>               REJECT:           0
>>>>>>>>>>>>>              PENDING:           0
>>>>>>>>>>>>>         STOP CAPTURE:           0
>>>>>>>>>>>>>                Total:           0
>>>>>>>>>>>>> Total files processed:             68987
>>>>>>>>>>>>> Total files data processed:        42751396  bytes
>>>>>>>>>>>>> Total files buffered:              0
>>>>>>>>>>>>> Total files released:              0
>>>>>>>>>>>>> Total files freed:                 0
>>>>>>>>>>>>> Total files captured:              0
>>>>>>>>>>>>> Total files within one packet:     0
>>>>>>>>>>>>> Total buffers allocated:           0
>>>>>>>>>>>>> Total buffers freed:               0
>>>>>>>>>>>>> Total buffers released:            0
>>>>>>>>>>>>> Maximum file buffers used:         0
>>>>>>>>>>>>> Total buffers free errors:         0
>>>>>>>>>>>>> Total buffers release errors:      0
>>>>>>>>>>>>> Total memcap failures:             0
>>>>>>>>>>>>> Total memcap failures at reserve:  0
>>>>>>>>>>>>> Total reserve failures:            0
>>>>>>>>>>>>> Total file capture size min:       0
>>>>>>>>>>>>> Total file capture size max:       0
>>>>>>>>>>>>> Total capture max before reserve:  0
>>>>>>>>>>>>> Total file signature max:          0
>>>>>>>>>>>>>
>>>>>>>>>>>>> ============================================================
>>>>>>>>>>>>> ===================
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> As you can see, in the "File type verdicts" section I got all
>>>>>>>>>>>>> the
>>>>>>>>>>>>> files
>>>>>>>>>>>>> with verdict LOG. Also, I got 275 alerts that match the 275 GIF
>>>>>>>>>>>>> files
>>>>>>>>>>>>> detected by Snort.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am not sure if this is the expected behavior of this feature
>>>>>>>>>>>>> or
>>>>>>>>>>>>> maybe
>>>>>>>>>>>>> I
>>>>>>>>>>>>> am not configuring Snort properly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am I doing something wrong or configuring the preprocessor in a
>>>>>>>>>>>>> wrong
>>>>>>>>>>>>> way?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your help and best Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pablo Cantosredborder.org / pcantos at ...16842...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This body part will be downloaded on demand.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This body part will be downloaded on demand.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>>>>> ------------------
>>>>>>>>>>>>> Dive into the World of Parallel Programming The Go Parallel
>>>>>>>>>>>>> Website,
>>>>>>>>>>>>> sponsored
>>>>>>>>>>>>> by Intel and developed in partnership with Slashdot Media, is
>>>>>>>>>>>>> your
>>>>>>>>>>>>> hub
>>>>>>>>>>>>> for
>>>>>>>>>>>>> all
>>>>>>>>>>>>> things parallel software development, from weekly thought
>>>>>>>>>>>>> leadership
>>>>>>>>>>>>> blogs
>>>>>>>>>>>>> to
>>>>>>>>>>>>> news, videos, case studies, tutorials and more. Take a look and
>>>>>>>>>>>>> join
>>>>>>>>>>>>> the
>>>>>>>>>>>>> conversation now. http://goparallel.sourceforge.net/
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Snort-users mailing list
>>>>>>>>>>>>> Snort-users at lists.sourceforge.net
>>>>>>>>>>>>> Go to this URL to change user options or unsubscribe:
>>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/snort-users
>>>>>>>>>>>>> Snort-users list archive:
>>>>>>>>>>>>> http://sourceforge.net/mailarchive/forum.php?forum_
>>>>>>>>>>>>> name=snort-users
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please visit http://blog.snort.org to stay current on all the
>>>>>>>>>>>>> latest
>>>>>>>>>>>>> Snort news!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20150409/a133a89a/attachment.html>


More information about the Snort-users mailing list