[Snort-devel] Snort-devel Digest, Vol 104, Issue 18

Al Lewis (allewi) allewi at ...3461...
Tue Mar 31 10:35:32 EDT 2015


Hello,

From the manual on http_inspect:

http://manual.snort.org/node17.html#SECTION00327000000000000000


Ideally you should put your rules (the ones that YOU write) in the local.rules file.


The path to your rules is set in your snort.conf file.


Hope this helps.


Albert Lewis
QA Software Engineer
SOURCEfire, Inc. now part of Cisco
9780 Patuxent Woods Drive
Columbia, MD 21046
Phone: (office) 443.430.7112
Email: allewi at ...3461...

From: Muhammad Ridwan Zalbina [mailto:zalbinaridwan at ...2499...]
Sent: Tuesday, March 31, 2015 10:20 AM
To: snort-devel at lists.sourceforge.net
Subject: Re: [Snort-devel] Snort-devel Digest, Vol 104, Issue 18

Hello, dear snort developers and users .
I would like to know how to activate http inspect preprocessor on snort ?
and where we place our rules ?

2015-03-31 17:23 GMT+07:00 <snort-devel-request at lists.sourceforge.net<mailto:snort-devel-request at lists.sourceforge.net>>:
Send Snort-devel mailing list submissions to
        snort-devel at lists.sourceforge.net<mailto:snort-devel at lists.sourceforge.net>

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.sourceforge.net/lists/listinfo/snort-devel
or, via email, send a message with subject or body 'help' to
        snort-devel-request at lists.sourceforge.net<mailto:snort-devel-request at lists.sourceforge.net>

You can reach the person managing the list at
        snort-devel-owner at lists.sourceforge.net<mailto:snort-devel-owner at lists.sourceforge.net>

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Snort-devel digest..."


Today's Topics:

   1. Re: Stream5 issue (Emiliano Fausto)
   2. Re: Stream5 issue (Arun Koshal)


----------------------------------------------------------------------

Message: 1
Date: Mon, 30 Mar 2015 10:35:05 -0300
From: Emiliano Fausto <emiliano.fausto at ...2499...<mailto:emiliano.fausto at ...2499...>>
Subject: Re: [Snort-devel] Stream5 issue
To: Arun Koshal <akoshal04 at ...2499...<mailto:akoshal04 at ...2499...>>
Cc: "snort-devel at lists.sourceforge.net<mailto:snort-devel at lists.sourceforge.net>"
        <snort-devel at lists.sourceforge.net<mailto:snort-devel at lists.sourceforge.net>>
Message-ID:
        <CAD2H3x-bL8w0VN94k48_0_v8RQ16Vox3yRCQ78YCxefaWfvcCg at ...2500...<mailto:CAD2H3x-bL8w0VN94k48_0_v8RQ16Vox3yRCQ78YCxefaWfvcCg at ...2500...>>
Content-Type: text/plain; charset="utf-8"

Hello Arun,

I've found more information about your second issue, maybe these articles
help.

Link: https://wiki.wireshark.org/Development/LibpcapFileFormat

It seems that you'll want to set the snaplen to the maximum size possible: "*A
captured packet in a capture file does not necessarily contain all the data
in the packet as it appeared on the network; the capture file might contain
at most the first N bytes of each packet, for some value of N. The value
of N, in such a capture, is called the "snapshot length" or "snaplen" of
the capture. N might be a value larger than the largest possible packet, to
ensure that no packet in the capture is "sliced" short; a value of 65535
will typically be used in this case.*"

Link:
http://www.sans.org/reading-room/whitepapers/detection/analysis-snort-data-acquisition-modules-34027

It says: "*Snort developers recommend setting the snap length to the
highest possible value since fragmented packets are defragmented before
being moved to the QUEUE (Snort Team, daq-0.6.2 README):*

*    config snaplen: 65535*


*Chris Murphy, chrismrph0 at ...2499...<mailto:chrismrph0 at ...2499...> <chrismrph0 at ...2499...<mailto:chrismrph0 at ...2499...>>*
*With this maximum value, Snort will be sure to see the entire packet.*"

Link:
http://taosecurity.blogspot.com.ar/2005/04/using-snorts-k-option-i-was-looking.html

In this issue, the problem was caused because of the snap-len and the "k"
option, which I see you have already setted, so it just validates you
should leave it in -k none, but probably won't solve your second issue.

In my opinion then, I think you are doing good in increasing the snaplen
size, since it's even necessary in most cases.

About the first problem I didn't try it, so I can't talk about it by now.
If I find something I'll let you know.

Hope it helps!
Emiliano.

On Mon, Mar 30, 2015 at 6:53 AM, Arun Koshal <akoshal04 at ...2499...<mailto:akoshal04 at ...2499...>> wrote:

> Hi Emiliano,
>
> Thanks a lot for your inputs!!!
>
> I am facing the issue # 1 even after disabling NIC segmentation features.
> Please find below the output of the ethtool command:
> -bash-4.1# ethtool -k eth1
> Features for eth1:
> rx-checksumming: off
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: off
>
> Please find below the stream5 configuration in my setup:
> preprocessor stream5_global: track_tcp yes, \
>    track_udp yes, \
>    track_icmp no, \
>    max_tcp 262144, \
>    max_udp 131072, \
>    max_active_responses 2, \
>    min_response_seconds 5
> preprocessor stream5_tcp: policy linux, detect_anomalies, require_3whs
> 180, \
>    overlap_limit 10, small_segments 3 bytes 150, timeout 180,
> max_queued_bytes 0, max_queued_segs 0,\
>     ports client 21 22 23 25 42 53 70 79 109 110 111 113 119 135 136 137
> 139 143 \
>         161 445 513 514 587 593 691 1433 1521 1741 2100 3306 6070 6633
> 6653 6665 6666 6667 6668 6669 \
>         7000 8181 32770 32771 32772 32773 32774 32775 32776 32777 32778
> 32779, \
>     ports both 36 80 81 82 83 84 85 86 87 88 89 90 110 311 383 443 465 563
> 555 591 593 631 636 801 808 818 901 972 989 992 993 994 995 1158 1220 1414
> 1533 1741 1830 1942 2231 2301 2381 2809 2980 3029 3037 3057 3128 3443 3702
> 4000 4343 4848 5000 5117 5250 5600 6080 6173 6988 7907 7000 7001 7071 7144
> 7145 7510 7802 7770 7777 7778 7779 \
>         7801 7900 7901 7902 7903 7904 7905 7906 7908 7909 7910 7911 7912
> 7913 7914 7915 7916 \
>         7917 7918 7919 7920 8000 8008 8014 8028 8080 8081 8082 8085 8088
> 8090 8118 8123 8180 8181 8222 8243 8280 8300 8333 8344 8500 8509 8800 8888
> 8899 8983 9000 9060 9080 9090 9091 9111 9290 9443 9999 10000 11371 12601
> 13014 15489 29991 33300 34412 34443 34444 41080 44449 50000 50002 51423
> 53331 55252 55555 56712
> preprocessor stream5_udp: timeout 180
>
> As mentioned earlier, I have Snort running in pcap mode, with following
> options:
> ./snort -A fast -b -d -k none -i eth1 --daq pcap --daq-var
> buffer_size=536870912 -u root -g root -c /etc/snort/snort.conf -l
> /var/log/snort -A unsock
>
> I am streaming the client side traffic only.
>
> Please let me know how should I debug this problem. The first issue has
> really become a bottleneck.
>
> Thanks and regards,
> Arun
>
>
>
> On Sat, Mar 28, 2015 at 6:59 PM, Emiliano Fausto <
> emiliano.fausto at ...2499...<mailto:emiliano.fausto at ...2499...>> wrote:
>
>> Hello Arun,
>>
>> I'd suggest you to read this article:
>> http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html
>>
>> I had similar troubles too some time ago because of it.
>>
>> Also depending on what you are trying to capture, this option could also
>> be influencing the capture:
>>
>> -k *checksum-mode*
>>
>> Controls which packet checksums Snort computes and verifies. Valid
>> checksum modes include all, noip, notcp, noudp, noicmp, and none. This
>> can be used to eliminate packets that fail their checksums - caused either
>> by network faults or IDS evasion attempts
>>
>> Maybe if  your checksum is known to fail, you may  put: "-k none"
>>
>> Hope it helps!!
>>
>> Regards,
>> Emiliano.
>>
>> On Sat, Mar 28, 2015 at 1:34 AM, Arun Koshal <akoshal04 at ...2499...<mailto:akoshal04 at ...2499...>> wrote:
>>
>>> Hi,
>>>
>>> I have written a dynamic preprocessor for inspecting some custom
>>> application. This dynamic preprocessor depends on stream5 preprocessor from
>>> getting the TCP stream. I am using snort in PCAP mode. I am facing
>>> following very strange problems -
>>>
>>> 1. The data in the rebuilt stream given by stream5 does not match with
>>> the TCP sequence number. For example - for a given TCP packet the sequence
>>> number in packet (pkt->tcp_header->sequence) is 507351850, but the data in
>>> pkt->payload is actually same as that of some old packet, having TCP
>>> sequence number 507343162. This scenario happens in case there are lot of
>>> packets getting dropped. I confirmed this with wireshark packet capture and
>>> I have observed multiple such instances. I also noticed that I am getting
>>> all the packets in the same buffer (pkt->payload remains same for all
>>> packets). So it seems like that I am getting a new packet with new header
>>> but old data. If I configure the pcap buffer_size as 512MB, the packets do
>>> not drop and this problem does not happen.
>>>
>>> 2. The second problem that I faced was with the pcap snap_len. In my
>>> setup, I had snort running with default pcap snap_len. I noticed that snort
>>> was not receiving packets having 1448 bytes data (1500 bytes, excluding
>>> Ethernet header). On reducing the MTU of the traffic generator from 1500 to
>>> 1484, Snort started receiving those packets. As a workaround, I increased
>>> the snap_len in sfdaq.c to 1546. Is this behaviour correct?
>>>
>>> It would be really great if someone can provide some inputs on these
>>> issues.
>>>
>>> Thanks and regards.
>>> Arun
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>> sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub
>>> for all
>>> things parallel software development, from weekly thought leadership
>>> blogs to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Snort-devel mailing list
>>> Snort-devel at lists.sourceforge.net<mailto:Snort-devel at lists.sourceforge.net>
>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>> Archive:
>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>
>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...

------------------------------

Message: 2
Date: Tue, 31 Mar 2015 15:53:42 +0530
From: Arun Koshal <akoshal04 at ...2499...<mailto:akoshal04 at ...2499...>>
Subject: Re: [Snort-devel] Stream5 issue
To: Emiliano Fausto <emiliano.fausto at ...2499...<mailto:emiliano.fausto at ...2499...>>
Cc: "snort-devel at lists.sourceforge.net<mailto:snort-devel at lists.sourceforge.net>"
        <snort-devel at lists.sourceforge.net<mailto:snort-devel at lists.sourceforge.net>>
Message-ID:
        <CAOUJN1C4fcLPAU+a94b_FAhAWvHk2zKni_SX3kKbz2cekT+w4A at ...2500...<mailto:CAOUJN1C4fcLPAU%2Ba94b_FAhAWvHk2zKni_SX3kKbz2cekT%2Bw4A at ...2500...>>
Content-Type: text/plain; charset="utf-8"

Hi Emiliano,

Thanks a lot for your inputs!!! This information is really helpful.

I am still working on the first issue and any suggestions on that will
really be helpful.

In order to ensure that what I have implemented is correct, please find
below few points elaborating the implementation:
1. Setting up the client side reassembly (this is done if
pkt->stream_session_ptr is not NULL):



*if (!(_dpd.streamAPI->get_reassembly_direction(pkt->stream_session_ptr) &
SSN_DIR_FROM_CLIENT)) {
_dpd.streamAPI->set_reassembly(pkt->stream_session_ptr,
STREAM_FLPOLICY_FOOTPRINT, SSN_DIR_FROM_CLIENT,
STREAM_FLPOLICY_SET_ABSOLUTE);    }*
2. Skip the packet if *(pkt->flags & FLAG_STREAM_INSERT)* is true.

3. I am processing the packets having *(pkt->flags & FLAG_REBUILT_STREAM)*
set.

4. For stream rebuilt packets (FLAG_REBUILT_STREAM set), I am calling
function _dpd.streamAPI->missing_in_reassembled() to check if there is any
missing packet. In that case I am doing a cleanup of the application state
and re-initializing it.

This works fine if packet rate is not too high. On high packet rate, I
start getting the second issue. To debug the issue I added code to match
the TCP sequence numbers. But it was really surprising that I am getting
old data with a new sequence number. I have handled the scenario of missing
packets in my application code but this behaviour (old data with new
sequence number) seems to be difficult to handle.

Please suggest if I am missing something.

Thanks again for your valuable inputs!

Thanks and regards,
Arun

On Mon, Mar 30, 2015 at 7:05 PM, Emiliano Fausto <emiliano.fausto at ...2499...<mailto:emiliano.fausto at ...2499...>>
wrote:

> Hello Arun,
>
> I've found more information about your second issue, maybe these articles
> help.
>
> Link: https://wiki.wireshark.org/Development/LibpcapFileFormat
>
> It seems that you'll want to set the snaplen to the maximum size possible:
> "*A captured packet in a capture file does not necessarily contain all
> the data in the packet as it appeared on the network; the capture file
> might contain at most the first N bytes of each packet, for some value
> of N. The value of N, in such a capture, is called the "snapshot length" or
> "snaplen" of the capture. N might be a value larger than the largest
> possible packet, to ensure that no packet in the capture is "sliced" short;
> a value of 65535 will typically be used in this case.*"
>
> Link:
> http://www.sans.org/reading-room/whitepapers/detection/analysis-snort-data-acquisition-modules-34027
>
> It says: "*Snort developers recommend setting the snap length to the
> highest possible value since fragmented packets are defragmented before
> being moved to the QUEUE (Snort Team, daq-0.6.2 README):*
>
> *    config snaplen: 65535*
>
>
> *Chris Murphy, chrismrph0 at ...2499...<mailto:chrismrph0 at ...2499...> <chrismrph0 at ...2499...<mailto:chrismrph0 at ...2499...>>*
> *With this maximum value, Snort will be sure to see the entire packet.*"
>
> Link:
> http://taosecurity.blogspot.com.ar/2005/04/using-snorts-k-option-i-was-looking.html
>
> In this issue, the problem was caused because of the snap-len and the "k"
> option, which I see you have already setted, so it just validates you
> should leave it in -k none, but probably won't solve your second issue.
>
> In my opinion then, I think you are doing good in increasing the snaplen
> size, since it's even necessary in most cases.
>
> About the first problem I didn't try it, so I can't talk about it by now.
> If I find something I'll let you know.
>
> Hope it helps!
> Emiliano.
>
> On Mon, Mar 30, 2015 at 6:53 AM, Arun Koshal <akoshal04 at ...2499...<mailto:akoshal04 at ...2499...>> wrote:
>
>> Hi Emiliano,
>>
>> Thanks a lot for your inputs!!!
>>
>> I am facing the issue # 1 even after disabling NIC segmentation features.
>> Please find below the output of the ethtool command:
>> -bash-4.1# ethtool -k eth1
>> Features for eth1:
>> rx-checksumming: off
>> tx-checksumming: off
>> scatter-gather: off
>> tcp-segmentation-offload: off
>> udp-fragmentation-offload: off
>> generic-segmentation-offload: off
>> generic-receive-offload: off
>> large-receive-offload: off
>> rx-vlan-offload: on
>> tx-vlan-offload: on
>> ntuple-filters: off
>> receive-hashing: off
>>
>> Please find below the stream5 configuration in my setup:
>> preprocessor stream5_global: track_tcp yes, \
>>    track_udp yes, \
>>    track_icmp no, \
>>    max_tcp 262144, \
>>    max_udp 131072, \
>>    max_active_responses 2, \
>>    min_response_seconds 5
>> preprocessor stream5_tcp: policy linux, detect_anomalies, require_3whs
>> 180, \
>>    overlap_limit 10, small_segments 3 bytes 150, timeout 180,
>> max_queued_bytes 0, max_queued_segs 0,\
>>     ports client 21 22 23 25 42 53 70 79 109 110 111 113 119 135 136 137
>> 139 143 \
>>         161 445 513 514 587 593 691 1433 1521 1741 2100 3306 6070 6633
>> 6653 6665 6666 6667 6668 6669 \
>>         7000 8181 32770 32771 32772 32773 32774 32775 32776 32777 32778
>> 32779, \
>>     ports both 36 80 81 82 83 84 85 86 87 88 89 90 110 311 383 443 465
>> 563 555 591 593 631 636 801 808 818 901 972 989 992 993 994 995 1158 1220
>> 1414 1533 1741 1830 1942 2231 2301 2381 2809 2980 3029 3037 3057 3128 3443
>> 3702 4000 4343 4848 5000 5117 5250 5600 6080 6173 6988 7907 7000 7001 7071
>> 7144 7145 7510 7802 7770 7777 7778 7779 \
>>         7801 7900 7901 7902 7903 7904 7905 7906 7908 7909 7910 7911 7912
>> 7913 7914 7915 7916 \
>>         7917 7918 7919 7920 8000 8008 8014 8028 8080 8081 8082 8085 8088
>> 8090 8118 8123 8180 8181 8222 8243 8280 8300 8333 8344 8500 8509 8800 8888
>> 8899 8983 9000 9060 9080 9090 9091 9111 9290 9443 9999 10000 11371 12601
>> 13014 15489 29991 33300 34412 34443 34444 41080 44449 50000 50002 51423
>> 53331 55252 55555 56712
>> preprocessor stream5_udp: timeout 180
>>
>> As mentioned earlier, I have Snort running in pcap mode, with following
>> options:
>> ./snort -A fast -b -d -k none -i eth1 --daq pcap --daq-var
>> buffer_size=536870912 -u root -g root -c /etc/snort/snort.conf -l
>> /var/log/snort -A unsock
>>
>> I am streaming the client side traffic only.
>>
>> Please let me know how should I debug this problem. The first issue has
>> really become a bottleneck.
>>
>> Thanks and regards,
>> Arun
>>
>>
>>
>> On Sat, Mar 28, 2015 at 6:59 PM, Emiliano Fausto <
>> emiliano.fausto at ...2499...<mailto:emiliano.fausto at ...2499...>> wrote:
>>
>>> Hello Arun,
>>>
>>> I'd suggest you to read this article:
>>> http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html
>>>
>>> I had similar troubles too some time ago because of it.
>>>
>>> Also depending on what you are trying to capture, this option could also
>>> be influencing the capture:
>>>
>>> -k *checksum-mode*
>>>
>>> Controls which packet checksums Snort computes and verifies. Valid
>>> checksum modes include all, noip, notcp, noudp, noicmp, and none. This
>>> can be used to eliminate packets that fail their checksums - caused either
>>> by network faults or IDS evasion attempts
>>>
>>> Maybe if  your checksum is known to fail, you may  put: "-k none"
>>>
>>> Hope it helps!!
>>>
>>> Regards,
>>> Emiliano.
>>>
>>> On Sat, Mar 28, 2015 at 1:34 AM, Arun Koshal <akoshal04 at ...2499...<mailto:akoshal04 at ...2499...>>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have written a dynamic preprocessor for inspecting some custom
>>>> application. This dynamic preprocessor depends on stream5 preprocessor from
>>>> getting the TCP stream. I am using snort in PCAP mode. I am facing
>>>> following very strange problems -
>>>>
>>>> 1. The data in the rebuilt stream given by stream5 does not match with
>>>> the TCP sequence number. For example - for a given TCP packet the sequence
>>>> number in packet (pkt->tcp_header->sequence) is 507351850, but the data in
>>>> pkt->payload is actually same as that of some old packet, having TCP
>>>> sequence number 507343162. This scenario happens in case there are lot of
>>>> packets getting dropped. I confirmed this with wireshark packet capture and
>>>> I have observed multiple such instances. I also noticed that I am getting
>>>> all the packets in the same buffer (pkt->payload remains same for all
>>>> packets). So it seems like that I am getting a new packet with new header
>>>> but old data. If I configure the pcap buffer_size as 512MB, the packets do
>>>> not drop and this problem does not happen.
>>>>
>>>> 2. The second problem that I faced was with the pcap snap_len. In my
>>>> setup, I had snort running with default pcap snap_len. I noticed that snort
>>>> was not receiving packets having 1448 bytes data (1500 bytes, excluding
>>>> Ethernet header). On reducing the MTU of the traffic generator from 1500 to
>>>> 1484, Snort started receiving those packets. As a workaround, I increased
>>>> the snap_len in sfdaq.c to 1546. Is this behaviour correct?
>>>>
>>>> It would be really great if someone can provide some inputs on these
>>>> issues.
>>>>
>>>> Thanks and regards.
>>>> Arun
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>>> sponsored
>>>> by Intel and developed in partnership with Slashdot Media, is your hub
>>>> for all
>>>> things parallel software development, from weekly thought leadership
>>>> blogs to
>>>> news, videos, case studies, tutorials and more. Take a look and join the
>>>> conversation now. http://goparallel.sourceforge.net/
>>>> _______________________________________________
>>>> Snort-devel mailing list
>>>> Snort-devel at lists.sourceforge.net<mailto:Snort-devel at lists.sourceforge.net>
>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>> Archive:
>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>
>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...

------------------------------

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/

------------------------------

_______________________________________________
Snort-devel mailing list
Snort-devel at lists.sourceforge.net<mailto:Snort-devel at lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/snort-devel


End of Snort-devel Digest, Vol 104, Issue 18
********************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20150331/73d5969e/attachment.html>


More information about the Snort-devel mailing list