[Snort-devel] [snort-devel] Chainning pre-processors

Emiliano Fausto emiliano.fausto at ...2499...
Thu Dec 5 14:19:06 EST 2013


>
> Hello Hui,
>
>  the "-k none" did the trick, I really really thank you for all your help.
>
>  As a summary (in case anyone else has the some doubts as me), correct me
> if I'm wrong:
>
>  1) It's possible to chain preprocessors and dynamic-preprocesors in
> SNORT. In fact, they are chained automatically by SNORT by default. The
> order will be given for the priority set when they are registered (for
> example with the function AddPreproc()).
>
>  2) The priorities are set in the file preprocids.h.
>
>  3) Using the stream5 preprocessor is as explained in the official
> Stream5.README documentation, we can add that if you are seeing too much
> discarded TCP packets in the statistics, you may want to add the "-k none"
> parameter when starting the snort process (tells snort to ignore checksum
> errors in the TCP packets).
>
>  4) Inside the preprocessor code, you may use the filter: (SFSnortPacket*)
> mypacket->flags & FLAG_REBUILT_STREAM, to be sure that this is a rebuilt
> packet.
>
>  My preprocessor and the TCP reassembling is working perfectly now with
> your help.
>
>  Again Hui, thanks a lot for everything!
>  Emiliano.
>
>
>
> 2013/12/5 Hui Cao <hcao at ...402...>
>
>>  Did you see many discards in snort exit. If so, you need to disable
>> checksum. Use -k none when you run the command.
>>
>> Best,
>> Hui.
>>
>> On 12/05/2013 01:40 PM, Emiliano Fausto wrote:
>>
>> Hui,
>>
>>  I was really trying to get Stream5 working and I couldn't, I'm pretty
>> sure that the problem is not related to Stream5, but I just can't get the
>> preprocessor to rebuilt TCP segments.
>>
>>  In wireshark I can see there are lot of TCP segmented packets and if I
>> select to reassemble the HTTP ones, it does it well. Backing to SRNOT, for
>> some reason the statistics that the Stream5 are showing are that there
>> where some TCP sessions, there were discarded TCP packets, etc. But in the
>> line: "TCP Rebuilt packets" shows a 0.
>>
>>  I tried with HTTP so that I can have a known protocol to start with as
>> you suggested me, and tried to download a wallpaper, which (due to its
>> size) was delivered to me in several packets.
>>
>>  I changed the snort.conf configuration file, to look like this:
>>
>>  preprocessor stream5_global: track_tcp yes, \
>>     track_udp no, \
>>     track_icmp no, \
>>      max_tcp 262144, \
>>     max_active_responses 2, \
>>      min_response_seconds 5, \
>>     show_rebuilt_packets
>> preprocessor stream5_tcp: policy linux, \
>>     overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
>>     ports both all
>>
>>  I also played a little bit with my own preprocessor to print the Flags
>> that the packet brings, and they change based on the priority I assign to
>> the AddPreproc(). But I tried different ones, from FIRST to LAST and none
>> of them matched the & FLAG_REBUILT_STREAM flag.
>>
>>  I was looking at the stream5 documentation (the README file), also
>> searched different threads in the snort users and devel community, but none
>> of them helped me.
>>
>>  Do you know where can I get more information or examples on how to use
>> properly the Stream5 preprocess, or if I'm missing something else to make
>> it works? maybe create a generic rule, or something like that?
>>
>>  I really thank you in advance for all your help on this,
>>  Emiliano.
>>
>>
>>
>> 2013/12/5 Hui Cao <hcao at ...402...>
>>
>>>  Make sure you have frag3 enabled. You can try the application
>>> preprocessors (such as http, smtp, ssl, sip etc) to see how it works.  If
>>> the packets are reassembled, you can see that from snort exit stats for
>>> stream etc.
>>>
>>> Stream will reassemble the payloads of TCP.
>>>
>>> Best,
>>> Hui.
>>>
>>>
>>> On 12/05/2013 08:26 AM, Emiliano Fausto wrote:
>>>
>>> Hello Hui,
>>>
>>>  you are totally right. I was confused, I didn't have to assemble
>>> fragmented IP packets.
>>>
>>>  So, I configured the Stream5 preprocessor in order it to take into
>>> account the port of my application (it's 9090).
>>>
>>>  I let my AddPreproc() with priority set to PRIORITY_APPLICATION so
>>> that it's registered after the Stream5.
>>>
>>>  Then, I tried to get any packet with (p->flags & FLAG_REBUILT_STREAM),
>>> but none of them matched, it seems that the Stream5 preprocessor isn't
>>> reassembling my TCP segments.
>>>
>>>  The SNORT configuration file is set like this:
>>>
>>>  preprocessor stream5_global: track_tcp yes, \
>>>     track_udp yes, \
>>>     track_icmp no, \
>>>     max_tcp 262144, \
>>>     max_udp 131072, \
>>>     max_active_responses 2, \
>>>     min_response_seconds 5
>>> preprocessor stream5_tcp: policy windows, detect_anomalies, require_3whs
>>> 180, \
>>>     overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
>>>     ports client 21 22, \
>>>     ports both 9090
>>> preprocessor stream5_udp: timeout 180
>>>
>>>  My app works on port 9090.
>>>
>>>  Maybe, the problem is not related to stream5, maybe the problem is
>>> originated because the one in charge of doing the segments reassembly is my
>>> preprocessor, at the application level, and not the stream5 preprocessor at
>>> the TCP level.
>>>
>>>  More on this, into wireshark I can see all the packets of the protocol
>>> TCP segmented, and if I go to preferences-->protocols-->TCP it has an
>>> option that says: "allow dissectors to reassemble TCP". It means that the
>>> reassemble isn't being done at the TCP layer, but into the proper
>>> application.
>>>
>>>  Do you know if SNORT has something of this? Or the Stream5 should
>>> reassemble the payloads of the TCP packets?
>>>
>>>  Thanks a lot in advance,
>>> Emiliano.
>>>
>>>
>>> 2013/12/4 Hui Cao <hcao at ...402...>
>>>
>>>>  Frag3 will deal with IP fragmentation. You might want reassembled data
>>>> from TCP. This means your preprocessor should be after stream and you
>>>> should check
>>>> (SFSnortPacket*)ipacketp)->flags & FLAG_REBUILT_STREAM. Snort will only
>>>> reassemble up to the flush point defined (typically 16K). You can't expect
>>>> snort will give all the data from one reassembled packet.
>>>>
>>>> Fragmentation from wireshark might mean TCP segmentation. You might get
>>>> full IP packets.
>>>>
>>>> Best,
>>>> Hui.
>>>>  On 12/04/2013 03:17 PM, Emiliano Fausto wrote:
>>>>
>>>> Hui,
>>>>
>>>>  yes, I understood what you told me about filtering just the
>>>> de-fragmented packets, that's why I added these 3 lines:
>>>>
>>>>  SFSnortPacket *p = (SFSnortPacket *)pkt;
>>>> int fla = (p->flags & FLAG_REBUILT_FRAG);
>>>> _dpd.logMsg("Flags set? %d",fla);
>>>>
>>>>  But I generate fragmented packets (which for instance Wireshark
>>>> detects and show as fragmented and the possibility of reassembly them) and
>>>> I'm always getting with each one the output: "Flags set? 0".
>>>>
>>>>  I receive all the fragmented packets, but not the last one
>>>> reassembled.
>>>>
>>>>  Regards,
>>>> Emiliano.
>>>>
>>>>
>>>> 2013/12/4 Hui Cao <hcao at ...402...>
>>>>
>>>>> It will still get all packets including the raw packets. You need to
>>>>> use packet flags to filter them.
>>>>>
>>>>> Best,
>>>>> Hui.
>>>>>
>>>>> On Wed, Dec 4, 2013 at 3:00 PM, Emiliano Fausto
>>>>>  <emiliano.fausto at ...2499...> wrote:
>>>>> > Yes,
>>>>> >
>>>>> > that's what I thought, but for some reason the TCP packets keep on
>>>>> coming
>>>>> > fragmented to my preprocessor.
>>>>> >
>>>>> > No I took off all the preprocessors and just let the frag3 and
>>>>> mines, I'll
>>>>> > try to figure out if they are called in the correct order, but they
>>>>> are not
>>>>> > being chained.
>>>>> >
>>>>> > Thanks,
>>>>> > Emiliano.
>>>>> >
>>>>> >
>>>>> > 2013/12/4 Hui Cao <hcao at ...402...>
>>>>> >>
>>>>> >> It looks good to me.
>>>>> >>
>>>>> >> Best,
>>>>> >> Hui.
>>>>> >>
>>>>> >> On Wed, Dec 4, 2013 at 2:44 PM, Emiliano Fausto
>>>>> >> <emiliano.fausto at ...2499...> wrote:
>>>>> >> > Hi Hui,
>>>>> >> >
>>>>> >> > I've seen that I was using PRIORITY_TRANSPORT, which is lower
>>>>> than the
>>>>> >> > PRIORITY_NETWORK that uses frag3.
>>>>> >> >
>>>>> >> > Anyway, I put the priority: PRIORITY_LAST to my own preprocessor,
>>>>> but
>>>>> >> > when
>>>>> >> > the TCP packets keep arriving fragmented to my preprocessor.
>>>>> >> >
>>>>> >> > Is there anything else I should take into account?
>>>>> >> >
>>>>> >> > Thanks in advance,
>>>>> >> > Emiliano.
>>>>> >> >
>>>>> >> >
>>>>> >> > 2013/12/4 Hui Cao <hcao at ...402...>
>>>>> >> >>
>>>>> >> >> In sr/preprocids.h
>>>>> >> >>
>>>>> >> >> Best,
>>>>> >> >> Hui.
>>>>> >> >>
>>>>> >> >> On 12/04/2013 02:36 PM, Emiliano Fausto wrote:
>>>>> >> >>
>>>>> >> >> Great,
>>>>> >> >>
>>>>> >> >> so, the pre-processors are "chained" by default, and the order
>>>>> that
>>>>> >> >> SNORT
>>>>> >> >> follows to call them is set by the PRIORITY variable.
>>>>> >> >>
>>>>> >> >> Do you know where's defined this PRIORITY variable? Because I
>>>>> saw that
>>>>> >> >> the
>>>>> >> >> frag3 is being registered with PRIORITY_NETWORK, so I'd like to
>>>>> set the
>>>>> >> >> priority of my own preprocessor as (PRIORITY_NETWORK -1).
>>>>> >> >>
>>>>> >> >> Thanks in advance,
>>>>> >> >> Emiliano
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> 2013/12/4 Hui Cao <hcao at ...402...>
>>>>> >> >>>
>>>>> >> >>> sc means snort configuration. We use PRIORITY to sort the
>>>>> processing.
>>>>> >> >>> All
>>>>> >> >>> processors enabled will be called and processed based on
>>>>> priority. You
>>>>> >> >>> have
>>>>> >> >>> to rely on the code to figure out what exactly snort does.
>>>>> >> >>>
>>>>> >> >>> The checking is correct. You will only process rebuilt packets.
>>>>> >> >>>
>>>>> >> >>> Best,
>>>>> >> >>> Hui.
>>>>> >> >>> On 12/04/2013 02:19 PM, Emiliano Fausto wrote:
>>>>> >> >>>
>>>>> >> >>> Hello Hui,
>>>>> >> >>>
>>>>> >> >>> thanks a lot for your answer.
>>>>> >> >>>
>>>>> >> >>> Right now I have registered my preprocessor (let's call it
>>>>> >> >>> examplePreprocess as you said, because right now I'm using the
>>>>> one
>>>>> >> >>> provided
>>>>> >> >>> with the DPX) with this line:
>>>>> >> >>>
>>>>> >> >>> _dpd.addPreproc(ExampleProcess, PRIORITY_TRANSPORT, 10000,
>>>>> >> >>> PROTO_BIT__TCP);
>>>>> >> >>>
>>>>> >> >>> So, the only change is to add previous to the parameter
>>>>> >> >>> ExampleProcess,
>>>>> >> >>> the "sc". What does it mean? Do you know if there's any
>>>>> documentation
>>>>> >> >>> about
>>>>> >> >>> this chaining preprocesses?
>>>>> >> >>>
>>>>> >> >>> So, checking the flags, should be:
>>>>> >> >>>
>>>>> >> >>> (SFSnortPacket*)tcppacket)->flags & FLAG_REBUILT_FRAG
>>>>> >> >>>
>>>>> >> >>> right?
>>>>> >> >>>
>>>>> >> >>> Thanks again!
>>>>> >> >>> Emiliano.
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>> Then, I'll have to register my own preprocessor where?
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>> 2013/12/4 Hui Cao <hcao at ...402...>
>>>>> >> >>>>
>>>>> >> >>>> Yes, it is possible. You can register you preprocessor like
>>>>> this:
>>>>> >> >>>>
>>>>> >> >>>> _dpd.addPreproc( sc, ExampleProcess, PRIORITY_TRANSPORT,
>>>>> You_PP_ID,
>>>>> >> >>>> PROTO_BIT__IP );
>>>>> >> >>>>
>>>>> >> >>>> Remember check the following flag in your ExampleProcess:
>>>>> >> >>>>
>>>>> >> >>>> (SFSnortPacket*)ipacketp)->flags & FLAG_REBUILT_FRAG
>>>>> >> >>>>
>>>>> >> >>>> Best,
>>>>> >> >>>> Hui.
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> On 12/04/2013 12:52 PM, Emiliano Fausto wrote:
>>>>> >> >>>>
>>>>> >> >>>> Hi everybody,
>>>>> >> >>>>
>>>>> >> >>>> I'm creating a new preprocessor which needs to have the whole
>>>>> content
>>>>> >> >>>> in
>>>>> >> >>>> a packet which was fragmented.
>>>>> >> >>>>
>>>>> >> >>>> So I thought of using the frag3 preprocessor to re-assembly the
>>>>> >> >>>> packets,
>>>>> >> >>>> and then, when this reassembly is done, sent it to my own
>>>>> >> >>>> preprocessor.
>>>>> >> >>>>
>>>>> >> >>>> Do you know if this is possible? May I have the output of
>>>>> frag3 being
>>>>> >> >>>> the input of my own preprocessor?
>>>>> >> >>>>
>>>>> >> >>>> Regards,
>>>>> >> >>>> Emiliano.
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> ------------------------------------------------------------------------------
>>>>> >> >>>> Sponsored by Intel(R) XDK
>>>>> >> >>>> Develop, test and display web and hybrid apps with a single
>>>>> code
>>>>> >> >>>> base.
>>>>> >> >>>> Download it for free now!
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> _______________________________________________
>>>>> >> >>>> Snort-devel mailing list
>>>>> >> >>>> Snort-devel at lists.sourceforge.net
>>>>> >> >>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>> >> >>>> Archive:
>>>>> >> >>>>
>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>> >> >>>>
>>>>> >> >>>> Please visit http://blog.snort.org for the latest news about
>>>>> Snort!
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> ------------------------------------------------------------------------------
>>>>> >> >>>> Sponsored by Intel(R) XDK
>>>>> >> >>>> Develop, test and display web and hybrid apps with a single
>>>>> code
>>>>> >> >>>> base.
>>>>> >> >>>> Download it for free now!
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
>>>>> >> >>>> _______________________________________________
>>>>> >> >>>> Snort-devel mailing list
>>>>> >> >>>> Snort-devel at lists.sourceforge.net
>>>>> >> >>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>>> >> >>>> Archive:
>>>>> >> >>>>
>>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>> >> >>>>
>>>>> >> >>>> Please visit http://blog.snort.org for the latest news about
>>>>> Snort!
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20131205/bb5f22ef/attachment.html>


More information about the Snort-devel mailing list