[Snort-devel] Snort-devel Digest, Vol 104, Issue 18

Muhammad Ridwan Zalbina zalbinaridwan at ...2499...
Tue Mar 31 10:19:58 EDT 2015


Hello, dear snort developers and users .
I would like to know how to activate http inspect preprocessor on snort ?
and where we place our rules ?


2015-03-31 17:23 GMT+07:00 <snort-devel-request at lists.sourceforge.net>:

> Send Snort-devel mailing list submissions to
>         snort-devel at lists.sourceforge.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.sourceforge.net/lists/listinfo/snort-devel
> or, via email, send a message with subject or body 'help' to
>         snort-devel-request at lists.sourceforge.net
>
> You can reach the person managing the list at
>         snort-devel-owner at lists.sourceforge.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Snort-devel digest..."
>
>
> Today's Topics:
>
>    1. Re: Stream5 issue (Emiliano Fausto)
>    2. Re: Stream5 issue (Arun Koshal)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 30 Mar 2015 10:35:05 -0300
> From: Emiliano Fausto <emiliano.fausto at ...2499...>
> Subject: Re: [Snort-devel] Stream5 issue
> To: Arun Koshal <akoshal04 at ...2499...>
> Cc: "snort-devel at lists.sourceforge.net"
>         <snort-devel at lists.sourceforge.net>
> Message-ID:
>         <
> CAD2H3x-bL8w0VN94k48_0_v8RQ16Vox3yRCQ78YCxefaWfvcCg at ...2500...>
> Content-Type: text/plain; charset="utf-8"
>
> Hello Arun,
>
> I've found more information about your second issue, maybe these articles
> help.
>
> Link: https://wiki.wireshark.org/Development/LibpcapFileFormat
>
> It seems that you'll want to set the snaplen to the maximum size possible:
> "*A
> captured packet in a capture file does not necessarily contain all the data
> in the packet as it appeared on the network; the capture file might contain
> at most the first N bytes of each packet, for some value of N. The value
> of N, in such a capture, is called the "snapshot length" or "snaplen" of
> the capture. N might be a value larger than the largest possible packet, to
> ensure that no packet in the capture is "sliced" short; a value of 65535
> will typically be used in this case.*"
>
> Link:
>
> http://www.sans.org/reading-room/whitepapers/detection/analysis-snort-data-acquisition-modules-34027
>
> It says: "*Snort developers recommend setting the snap length to the
> highest possible value since fragmented packets are defragmented before
> being moved to the QUEUE (Snort Team, daq-0.6.2 README):*
>
> *    config snaplen: 65535*
>
>
> *Chris Murphy, chrismrph0 at ...2499... <chrismrph0 at ...2499...>*
> *With this maximum value, Snort will be sure to see the entire packet.*"
>
> Link:
>
> http://taosecurity.blogspot.com.ar/2005/04/using-snorts-k-option-i-was-looking.html
>
> In this issue, the problem was caused because of the snap-len and the "k"
> option, which I see you have already setted, so it just validates you
> should leave it in -k none, but probably won't solve your second issue.
>
> In my opinion then, I think you are doing good in increasing the snaplen
> size, since it's even necessary in most cases.
>
> About the first problem I didn't try it, so I can't talk about it by now.
> If I find something I'll let you know.
>
> Hope it helps!
> Emiliano.
>
> On Mon, Mar 30, 2015 at 6:53 AM, Arun Koshal <akoshal04 at ...2499...> wrote:
>
> > Hi Emiliano,
> >
> > Thanks a lot for your inputs!!!
> >
> > I am facing the issue # 1 even after disabling NIC segmentation features.
> > Please find below the output of the ethtool command:
> > -bash-4.1# ethtool -k eth1
> > Features for eth1:
> > rx-checksumming: off
> > tx-checksumming: off
> > scatter-gather: off
> > tcp-segmentation-offload: off
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: off
> > generic-receive-offload: off
> > large-receive-offload: off
> > rx-vlan-offload: on
> > tx-vlan-offload: on
> > ntuple-filters: off
> > receive-hashing: off
> >
> > Please find below the stream5 configuration in my setup:
> > preprocessor stream5_global: track_tcp yes, \
> >    track_udp yes, \
> >    track_icmp no, \
> >    max_tcp 262144, \
> >    max_udp 131072, \
> >    max_active_responses 2, \
> >    min_response_seconds 5
> > preprocessor stream5_tcp: policy linux, detect_anomalies, require_3whs
> > 180, \
> >    overlap_limit 10, small_segments 3 bytes 150, timeout 180,
> > max_queued_bytes 0, max_queued_segs 0,\
> >     ports client 21 22 23 25 42 53 70 79 109 110 111 113 119 135 136 137
> > 139 143 \
> >         161 445 513 514 587 593 691 1433 1521 1741 2100 3306 6070 6633
> > 6653 6665 6666 6667 6668 6669 \
> >         7000 8181 32770 32771 32772 32773 32774 32775 32776 32777 32778
> > 32779, \
> >     ports both 36 80 81 82 83 84 85 86 87 88 89 90 110 311 383 443 465
> 563
> > 555 591 593 631 636 801 808 818 901 972 989 992 993 994 995 1158 1220
> 1414
> > 1533 1741 1830 1942 2231 2301 2381 2809 2980 3029 3037 3057 3128 3443
> 3702
> > 4000 4343 4848 5000 5117 5250 5600 6080 6173 6988 7907 7000 7001 7071
> 7144
> > 7145 7510 7802 7770 7777 7778 7779 \
> >         7801 7900 7901 7902 7903 7904 7905 7906 7908 7909 7910 7911 7912
> > 7913 7914 7915 7916 \
> >         7917 7918 7919 7920 8000 8008 8014 8028 8080 8081 8082 8085 8088
> > 8090 8118 8123 8180 8181 8222 8243 8280 8300 8333 8344 8500 8509 8800
> 8888
> > 8899 8983 9000 9060 9080 9090 9091 9111 9290 9443 9999 10000 11371 12601
> > 13014 15489 29991 33300 34412 34443 34444 41080 44449 50000 50002 51423
> > 53331 55252 55555 56712
> > preprocessor stream5_udp: timeout 180
> >
> > As mentioned earlier, I have Snort running in pcap mode, with following
> > options:
> > ./snort -A fast -b -d -k none -i eth1 --daq pcap --daq-var
> > buffer_size=536870912 -u root -g root -c /etc/snort/snort.conf -l
> > /var/log/snort -A unsock
> >
> > I am streaming the client side traffic only.
> >
> > Please let me know how should I debug this problem. The first issue has
> > really become a bottleneck.
> >
> > Thanks and regards,
> > Arun
> >
> >
> >
> > On Sat, Mar 28, 2015 at 6:59 PM, Emiliano Fausto <
> > emiliano.fausto at ...2499...> wrote:
> >
> >> Hello Arun,
> >>
> >> I'd suggest you to read this article:
> >>
> http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html
> >>
> >> I had similar troubles too some time ago because of it.
> >>
> >> Also depending on what you are trying to capture, this option could also
> >> be influencing the capture:
> >>
> >> -k *checksum-mode*
> >>
> >> Controls which packet checksums Snort computes and verifies. Valid
> >> checksum modes include all, noip, notcp, noudp, noicmp, and none. This
> >> can be used to eliminate packets that fail their checksums - caused
> either
> >> by network faults or IDS evasion attempts
> >>
> >> Maybe if  your checksum is known to fail, you may  put: "-k none"
> >>
> >> Hope it helps!!
> >>
> >> Regards,
> >> Emiliano.
> >>
> >> On Sat, Mar 28, 2015 at 1:34 AM, Arun Koshal <akoshal04 at ...2499...>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> I have written a dynamic preprocessor for inspecting some custom
> >>> application. This dynamic preprocessor depends on stream5 preprocessor
> from
> >>> getting the TCP stream. I am using snort in PCAP mode. I am facing
> >>> following very strange problems -
> >>>
> >>> 1. The data in the rebuilt stream given by stream5 does not match with
> >>> the TCP sequence number. For example - for a given TCP packet the
> sequence
> >>> number in packet (pkt->tcp_header->sequence) is 507351850, but the
> data in
> >>> pkt->payload is actually same as that of some old packet, having TCP
> >>> sequence number 507343162. This scenario happens in case there are lot
> of
> >>> packets getting dropped. I confirmed this with wireshark packet
> capture and
> >>> I have observed multiple such instances. I also noticed that I am
> getting
> >>> all the packets in the same buffer (pkt->payload remains same for all
> >>> packets). So it seems like that I am getting a new packet with new
> header
> >>> but old data. If I configure the pcap buffer_size as 512MB, the
> packets do
> >>> not drop and this problem does not happen.
> >>>
> >>> 2. The second problem that I faced was with the pcap snap_len. In my
> >>> setup, I had snort running with default pcap snap_len. I noticed that
> snort
> >>> was not receiving packets having 1448 bytes data (1500 bytes, excluding
> >>> Ethernet header). On reducing the MTU of the traffic generator from
> 1500 to
> >>> 1484, Snort started receiving those packets. As a workaround, I
> increased
> >>> the snap_len in sfdaq.c to 1546. Is this behaviour correct?
> >>>
> >>> It would be really great if someone can provide some inputs on these
> >>> issues.
> >>>
> >>> Thanks and regards.
> >>> Arun
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Dive into the World of Parallel Programming The Go Parallel Website,
> >>> sponsored
> >>> by Intel and developed in partnership with Slashdot Media, is your hub
> >>> for all
> >>> things parallel software development, from weekly thought leadership
> >>> blogs to
> >>> news, videos, case studies, tutorials and more. Take a look and join
> the
> >>> conversation now. http://goparallel.sourceforge.net/
> >>> _______________________________________________
> >>> Snort-devel mailing list
> >>> Snort-devel at lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/snort-devel
> >>> Archive:
> >>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> >>>
> >>> Please visit http://blog.snort.org for the latest news about Snort!
> >>>
> >>
> >>
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
> Message: 2
> Date: Tue, 31 Mar 2015 15:53:42 +0530
> From: Arun Koshal <akoshal04 at ...2499...>
> Subject: Re: [Snort-devel] Stream5 issue
> To: Emiliano Fausto <emiliano.fausto at ...2499...>
> Cc: "snort-devel at lists.sourceforge.net"
>         <snort-devel at lists.sourceforge.net>
> Message-ID:
>         <
> CAOUJN1C4fcLPAU+a94b_FAhAWvHk2zKni_SX3kKbz2cekT+w4A at ...2500...>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Emiliano,
>
> Thanks a lot for your inputs!!! This information is really helpful.
>
> I am still working on the first issue and any suggestions on that will
> really be helpful.
>
> In order to ensure that what I have implemented is correct, please find
> below few points elaborating the implementation:
> 1. Setting up the client side reassembly (this is done if
> pkt->stream_session_ptr is not NULL):
>
>
>
> *if (!(_dpd.streamAPI->get_reassembly_direction(pkt->stream_session_ptr) &
> SSN_DIR_FROM_CLIENT)) {
> _dpd.streamAPI->set_reassembly(pkt->stream_session_ptr,
> STREAM_FLPOLICY_FOOTPRINT, SSN_DIR_FROM_CLIENT,
> STREAM_FLPOLICY_SET_ABSOLUTE);    }*
> 2. Skip the packet if *(pkt->flags & FLAG_STREAM_INSERT)* is true.
>
> 3. I am processing the packets having *(pkt->flags & FLAG_REBUILT_STREAM)*
> set.
>
> 4. For stream rebuilt packets (FLAG_REBUILT_STREAM set), I am calling
> function _dpd.streamAPI->missing_in_reassembled() to check if there is any
> missing packet. In that case I am doing a cleanup of the application state
> and re-initializing it.
>
> This works fine if packet rate is not too high. On high packet rate, I
> start getting the second issue. To debug the issue I added code to match
> the TCP sequence numbers. But it was really surprising that I am getting
> old data with a new sequence number. I have handled the scenario of missing
> packets in my application code but this behaviour (old data with new
> sequence number) seems to be difficult to handle.
>
> Please suggest if I am missing something.
>
> Thanks again for your valuable inputs!
>
> Thanks and regards,
> Arun
>
> On Mon, Mar 30, 2015 at 7:05 PM, Emiliano Fausto <
> emiliano.fausto at ...2499...>
> wrote:
>
> > Hello Arun,
> >
> > I've found more information about your second issue, maybe these articles
> > help.
> >
> > Link: https://wiki.wireshark.org/Development/LibpcapFileFormat
> >
> > It seems that you'll want to set the snaplen to the maximum size
> possible:
> > "*A captured packet in a capture file does not necessarily contain all
> > the data in the packet as it appeared on the network; the capture file
> > might contain at most the first N bytes of each packet, for some value
> > of N. The value of N, in such a capture, is called the "snapshot length"
> or
> > "snaplen" of the capture. N might be a value larger than the largest
> > possible packet, to ensure that no packet in the capture is "sliced"
> short;
> > a value of 65535 will typically be used in this case.*"
> >
> > Link:
> >
> http://www.sans.org/reading-room/whitepapers/detection/analysis-snort-data-acquisition-modules-34027
> >
> > It says: "*Snort developers recommend setting the snap length to the
> > highest possible value since fragmented packets are defragmented before
> > being moved to the QUEUE (Snort Team, daq-0.6.2 README):*
> >
> > *    config snaplen: 65535*
> >
> >
> > *Chris Murphy, chrismrph0 at ...2499... <chrismrph0 at ...2499...>*
> > *With this maximum value, Snort will be sure to see the entire packet.*"
> >
> > Link:
> >
> http://taosecurity.blogspot.com.ar/2005/04/using-snorts-k-option-i-was-looking.html
> >
> > In this issue, the problem was caused because of the snap-len and the "k"
> > option, which I see you have already setted, so it just validates you
> > should leave it in -k none, but probably won't solve your second issue.
> >
> > In my opinion then, I think you are doing good in increasing the snaplen
> > size, since it's even necessary in most cases.
> >
> > About the first problem I didn't try it, so I can't talk about it by now.
> > If I find something I'll let you know.
> >
> > Hope it helps!
> > Emiliano.
> >
> > On Mon, Mar 30, 2015 at 6:53 AM, Arun Koshal <akoshal04 at ...2499...>
> wrote:
> >
> >> Hi Emiliano,
> >>
> >> Thanks a lot for your inputs!!!
> >>
> >> I am facing the issue # 1 even after disabling NIC segmentation
> features.
> >> Please find below the output of the ethtool command:
> >> -bash-4.1# ethtool -k eth1
> >> Features for eth1:
> >> rx-checksumming: off
> >> tx-checksumming: off
> >> scatter-gather: off
> >> tcp-segmentation-offload: off
> >> udp-fragmentation-offload: off
> >> generic-segmentation-offload: off
> >> generic-receive-offload: off
> >> large-receive-offload: off
> >> rx-vlan-offload: on
> >> tx-vlan-offload: on
> >> ntuple-filters: off
> >> receive-hashing: off
> >>
> >> Please find below the stream5 configuration in my setup:
> >> preprocessor stream5_global: track_tcp yes, \
> >>    track_udp yes, \
> >>    track_icmp no, \
> >>    max_tcp 262144, \
> >>    max_udp 131072, \
> >>    max_active_responses 2, \
> >>    min_response_seconds 5
> >> preprocessor stream5_tcp: policy linux, detect_anomalies, require_3whs
> >> 180, \
> >>    overlap_limit 10, small_segments 3 bytes 150, timeout 180,
> >> max_queued_bytes 0, max_queued_segs 0,\
> >>     ports client 21 22 23 25 42 53 70 79 109 110 111 113 119 135 136 137
> >> 139 143 \
> >>         161 445 513 514 587 593 691 1433 1521 1741 2100 3306 6070 6633
> >> 6653 6665 6666 6667 6668 6669 \
> >>         7000 8181 32770 32771 32772 32773 32774 32775 32776 32777 32778
> >> 32779, \
> >>     ports both 36 80 81 82 83 84 85 86 87 88 89 90 110 311 383 443 465
> >> 563 555 591 593 631 636 801 808 818 901 972 989 992 993 994 995 1158
> 1220
> >> 1414 1533 1741 1830 1942 2231 2301 2381 2809 2980 3029 3037 3057 3128
> 3443
> >> 3702 4000 4343 4848 5000 5117 5250 5600 6080 6173 6988 7907 7000 7001
> 7071
> >> 7144 7145 7510 7802 7770 7777 7778 7779 \
> >>         7801 7900 7901 7902 7903 7904 7905 7906 7908 7909 7910 7911 7912
> >> 7913 7914 7915 7916 \
> >>         7917 7918 7919 7920 8000 8008 8014 8028 8080 8081 8082 8085 8088
> >> 8090 8118 8123 8180 8181 8222 8243 8280 8300 8333 8344 8500 8509 8800
> 8888
> >> 8899 8983 9000 9060 9080 9090 9091 9111 9290 9443 9999 10000 11371 12601
> >> 13014 15489 29991 33300 34412 34443 34444 41080 44449 50000 50002 51423
> >> 53331 55252 55555 56712
> >> preprocessor stream5_udp: timeout 180
> >>
> >> As mentioned earlier, I have Snort running in pcap mode, with following
> >> options:
> >> ./snort -A fast -b -d -k none -i eth1 --daq pcap --daq-var
> >> buffer_size=536870912 -u root -g root -c /etc/snort/snort.conf -l
> >> /var/log/snort -A unsock
> >>
> >> I am streaming the client side traffic only.
> >>
> >> Please let me know how should I debug this problem. The first issue has
> >> really become a bottleneck.
> >>
> >> Thanks and regards,
> >> Arun
> >>
> >>
> >>
> >> On Sat, Mar 28, 2015 at 6:59 PM, Emiliano Fausto <
> >> emiliano.fausto at ...2499...> wrote:
> >>
> >>> Hello Arun,
> >>>
> >>> I'd suggest you to read this article:
> >>>
> http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html
> >>>
> >>> I had similar troubles too some time ago because of it.
> >>>
> >>> Also depending on what you are trying to capture, this option could
> also
> >>> be influencing the capture:
> >>>
> >>> -k *checksum-mode*
> >>>
> >>> Controls which packet checksums Snort computes and verifies. Valid
> >>> checksum modes include all, noip, notcp, noudp, noicmp, and none. This
> >>> can be used to eliminate packets that fail their checksums - caused
> either
> >>> by network faults or IDS evasion attempts
> >>>
> >>> Maybe if  your checksum is known to fail, you may  put: "-k none"
> >>>
> >>> Hope it helps!!
> >>>
> >>> Regards,
> >>> Emiliano.
> >>>
> >>> On Sat, Mar 28, 2015 at 1:34 AM, Arun Koshal <akoshal04 at ...2499...>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I have written a dynamic preprocessor for inspecting some custom
> >>>> application. This dynamic preprocessor depends on stream5
> preprocessor from
> >>>> getting the TCP stream. I am using snort in PCAP mode. I am facing
> >>>> following very strange problems -
> >>>>
> >>>> 1. The data in the rebuilt stream given by stream5 does not match with
> >>>> the TCP sequence number. For example - for a given TCP packet the
> sequence
> >>>> number in packet (pkt->tcp_header->sequence) is 507351850, but the
> data in
> >>>> pkt->payload is actually same as that of some old packet, having TCP
> >>>> sequence number 507343162. This scenario happens in case there are
> lot of
> >>>> packets getting dropped. I confirmed this with wireshark packet
> capture and
> >>>> I have observed multiple such instances. I also noticed that I am
> getting
> >>>> all the packets in the same buffer (pkt->payload remains same for all
> >>>> packets). So it seems like that I am getting a new packet with new
> header
> >>>> but old data. If I configure the pcap buffer_size as 512MB, the
> packets do
> >>>> not drop and this problem does not happen.
> >>>>
> >>>> 2. The second problem that I faced was with the pcap snap_len. In my
> >>>> setup, I had snort running with default pcap snap_len. I noticed that
> snort
> >>>> was not receiving packets having 1448 bytes data (1500 bytes,
> excluding
> >>>> Ethernet header). On reducing the MTU of the traffic generator from
> 1500 to
> >>>> 1484, Snort started receiving those packets. As a workaround, I
> increased
> >>>> the snap_len in sfdaq.c to 1546. Is this behaviour correct?
> >>>>
> >>>> It would be really great if someone can provide some inputs on these
> >>>> issues.
> >>>>
> >>>> Thanks and regards.
> >>>> Arun
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------------------------
> >>>> Dive into the World of Parallel Programming The Go Parallel Website,
> >>>> sponsored
> >>>> by Intel and developed in partnership with Slashdot Media, is your hub
> >>>> for all
> >>>> things parallel software development, from weekly thought leadership
> >>>> blogs to
> >>>> news, videos, case studies, tutorials and more. Take a look and join
> the
> >>>> conversation now. http://goparallel.sourceforge.net/
> >>>> _______________________________________________
> >>>> Snort-devel mailing list
> >>>> Snort-devel at lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
> >>>> Archive:
> >>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> >>>>
> >>>> Please visit http://blog.snort.org for the latest news about Snort!
> >>>>
> >>>
> >>>
> >>
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
>
> ------------------------------
>
> _______________________________________________
> Snort-devel mailing list
> Snort-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/snort-devel
>
>
> End of Snort-devel Digest, Vol 104, Issue 18
> ********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20150331/be0b9fbb/attachment.html>


More information about the Snort-devel mailing list