[Snort-devel] Stream5 issue

Arun Koshal akoshal04 at ...2499...
Tue Mar 31 06:23:42 EDT 2015


Hi Emiliano,

Thanks a lot for your inputs!!! This information is really helpful.

I am still working on the first issue and any suggestions on that will
really be helpful.

In order to ensure that what I have implemented is correct, please find
below few points elaborating the implementation:
1. Setting up the client side reassembly (this is done if
pkt->stream_session_ptr is not NULL):



*if (!(_dpd.streamAPI->get_reassembly_direction(pkt->stream_session_ptr) &
SSN_DIR_FROM_CLIENT)) {
_dpd.streamAPI->set_reassembly(pkt->stream_session_ptr,
STREAM_FLPOLICY_FOOTPRINT, SSN_DIR_FROM_CLIENT,
STREAM_FLPOLICY_SET_ABSOLUTE);    }*
2. Skip the packet if *(pkt->flags & FLAG_STREAM_INSERT)* is true.

3. I am processing the packets having *(pkt->flags & FLAG_REBUILT_STREAM)*
set.

4. For stream rebuilt packets (FLAG_REBUILT_STREAM set), I am calling
function _dpd.streamAPI->missing_in_reassembled() to check if there is any
missing packet. In that case I am doing a cleanup of the application state
and re-initializing it.

This works fine if packet rate is not too high. On high packet rate, I
start getting the second issue. To debug the issue I added code to match
the TCP sequence numbers. But it was really surprising that I am getting
old data with a new sequence number. I have handled the scenario of missing
packets in my application code but this behaviour (old data with new
sequence number) seems to be difficult to handle.

Please suggest if I am missing something.

Thanks again for your valuable inputs!

Thanks and regards,
Arun

On Mon, Mar 30, 2015 at 7:05 PM, Emiliano Fausto <emiliano.fausto at ...2499...>
wrote:

> Hello Arun,
>
> I've found more information about your second issue, maybe these articles
> help.
>
> Link: https://wiki.wireshark.org/Development/LibpcapFileFormat
>
> It seems that you'll want to set the snaplen to the maximum size possible:
> "*A captured packet in a capture file does not necessarily contain all
> the data in the packet as it appeared on the network; the capture file
> might contain at most the first N bytes of each packet, for some value
> of N. The value of N, in such a capture, is called the "snapshot length" or
> "snaplen" of the capture. N might be a value larger than the largest
> possible packet, to ensure that no packet in the capture is "sliced" short;
> a value of 65535 will typically be used in this case.*"
>
> Link:
> http://www.sans.org/reading-room/whitepapers/detection/analysis-snort-data-acquisition-modules-34027
>
> It says: "*Snort developers recommend setting the snap length to the
> highest possible value since fragmented packets are defragmented before
> being moved to the QUEUE (Snort Team, daq-0.6.2 README):*
>
> *    config snaplen: 65535*
>
>
> *Chris Murphy, chrismrph0 at ...2499... <chrismrph0 at ...2499...>*
> *With this maximum value, Snort will be sure to see the entire packet.*"
>
> Link:
> http://taosecurity.blogspot.com.ar/2005/04/using-snorts-k-option-i-was-looking.html
>
> In this issue, the problem was caused because of the snap-len and the "k"
> option, which I see you have already setted, so it just validates you
> should leave it in -k none, but probably won't solve your second issue.
>
> In my opinion then, I think you are doing good in increasing the snaplen
> size, since it's even necessary in most cases.
>
> About the first problem I didn't try it, so I can't talk about it by now.
> If I find something I'll let you know.
>
> Hope it helps!
> Emiliano.
>
> On Mon, Mar 30, 2015 at 6:53 AM, Arun Koshal <akoshal04 at ...2499...> wrote:
>
>> Hi Emiliano,
>>
>> Thanks a lot for your inputs!!!
>>
>> I am facing the issue # 1 even after disabling NIC segmentation features.
>> Please find below the output of the ethtool command:
>> -bash-4.1# ethtool -k eth1
>> Features for eth1:
>> rx-checksumming: off
>> tx-checksumming: off
>> scatter-gather: off
>> tcp-segmentation-offload: off
>> udp-fragmentation-offload: off
>> generic-segmentation-offload: off
>> generic-receive-offload: off
>> large-receive-offload: off
>> rx-vlan-offload: on
>> tx-vlan-offload: on
>> ntuple-filters: off
>> receive-hashing: off
>>
>> Please find below the stream5 configuration in my setup:
>> preprocessor stream5_global: track_tcp yes, \
>>    track_udp yes, \
>>    track_icmp no, \
>>    max_tcp 262144, \
>>    max_udp 131072, \
>>    max_active_responses 2, \
>>    min_response_seconds 5
>> preprocessor stream5_tcp: policy linux, detect_anomalies, require_3whs
>> 180, \
>>    overlap_limit 10, small_segments 3 bytes 150, timeout 180,
>> max_queued_bytes 0, max_queued_segs 0,\
>>     ports client 21 22 23 25 42 53 70 79 109 110 111 113 119 135 136 137
>> 139 143 \
>>         161 445 513 514 587 593 691 1433 1521 1741 2100 3306 6070 6633
>> 6653 6665 6666 6667 6668 6669 \
>>         7000 8181 32770 32771 32772 32773 32774 32775 32776 32777 32778
>> 32779, \
>>     ports both 36 80 81 82 83 84 85 86 87 88 89 90 110 311 383 443 465
>> 563 555 591 593 631 636 801 808 818 901 972 989 992 993 994 995 1158 1220
>> 1414 1533 1741 1830 1942 2231 2301 2381 2809 2980 3029 3037 3057 3128 3443
>> 3702 4000 4343 4848 5000 5117 5250 5600 6080 6173 6988 7907 7000 7001 7071
>> 7144 7145 7510 7802 7770 7777 7778 7779 \
>>         7801 7900 7901 7902 7903 7904 7905 7906 7908 7909 7910 7911 7912
>> 7913 7914 7915 7916 \
>>         7917 7918 7919 7920 8000 8008 8014 8028 8080 8081 8082 8085 8088
>> 8090 8118 8123 8180 8181 8222 8243 8280 8300 8333 8344 8500 8509 8800 8888
>> 8899 8983 9000 9060 9080 9090 9091 9111 9290 9443 9999 10000 11371 12601
>> 13014 15489 29991 33300 34412 34443 34444 41080 44449 50000 50002 51423
>> 53331 55252 55555 56712
>> preprocessor stream5_udp: timeout 180
>>
>> As mentioned earlier, I have Snort running in pcap mode, with following
>> options:
>> ./snort -A fast -b -d -k none -i eth1 --daq pcap --daq-var
>> buffer_size=536870912 -u root -g root -c /etc/snort/snort.conf -l
>> /var/log/snort -A unsock
>>
>> I am streaming the client side traffic only.
>>
>> Please let me know how should I debug this problem. The first issue has
>> really become a bottleneck.
>>
>> Thanks and regards,
>> Arun
>>
>>
>>
>> On Sat, Mar 28, 2015 at 6:59 PM, Emiliano Fausto <
>> emiliano.fausto at ...2499...> wrote:
>>
>>> Hello Arun,
>>>
>>> I'd suggest you to read this article:
>>> http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html
>>>
>>> I had similar troubles too some time ago because of it.
>>>
>>> Also depending on what you are trying to capture, this option could also
>>> be influencing the capture:
>>>
>>> -k *checksum-mode*
>>>
>>> Controls which packet checksums Snort computes and verifies. Valid
>>> checksum modes include all, noip, notcp, noudp, noicmp, and none. This
>>> can be used to eliminate packets that fail their checksums - caused either
>>> by network faults or IDS evasion attempts
>>>
>>> Maybe if  your checksum is known to fail, you may  put: "-k none"
>>>
>>> Hope it helps!!
>>>
>>> Regards,
>>> Emiliano.
>>>
>>> On Sat, Mar 28, 2015 at 1:34 AM, Arun Koshal <akoshal04 at ...2499...>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have written a dynamic preprocessor for inspecting some custom
>>>> application. This dynamic preprocessor depends on stream5 preprocessor from
>>>> getting the TCP stream. I am using snort in PCAP mode. I am facing
>>>> following very strange problems -
>>>>
>>>> 1. The data in the rebuilt stream given by stream5 does not match with
>>>> the TCP sequence number. For example - for a given TCP packet the sequence
>>>> number in packet (pkt->tcp_header->sequence) is 507351850, but the data in
>>>> pkt->payload is actually same as that of some old packet, having TCP
>>>> sequence number 507343162. This scenario happens in case there are lot of
>>>> packets getting dropped. I confirmed this with wireshark packet capture and
>>>> I have observed multiple such instances. I also noticed that I am getting
>>>> all the packets in the same buffer (pkt->payload remains same for all
>>>> packets). So it seems like that I am getting a new packet with new header
>>>> but old data. If I configure the pcap buffer_size as 512MB, the packets do
>>>> not drop and this problem does not happen.
>>>>
>>>> 2. The second problem that I faced was with the pcap snap_len. In my
>>>> setup, I had snort running with default pcap snap_len. I noticed that snort
>>>> was not receiving packets having 1448 bytes data (1500 bytes, excluding
>>>> Ethernet header). On reducing the MTU of the traffic generator from 1500 to
>>>> 1484, Snort started receiving those packets. As a workaround, I increased
>>>> the snap_len in sfdaq.c to 1546. Is this behaviour correct?
>>>>
>>>> It would be really great if someone can provide some inputs on these
>>>> issues.
>>>>
>>>> Thanks and regards.
>>>> Arun
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>>> sponsored
>>>> by Intel and developed in partnership with Slashdot Media, is your hub
>>>> for all
>>>> things parallel software development, from weekly thought leadership
>>>> blogs to
>>>> news, videos, case studies, tutorials and more. Take a look and join the
>>>> conversation now. http://goparallel.sourceforge.net/
>>>> _______________________________________________
>>>> Snort-devel mailing list
>>>> Snort-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/snort-devel
>>>> Archive:
>>>> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>>>>
>>>> Please visit http://blog.snort.org for the latest news about Snort!
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20150331/54db5db3/attachment.html>


More information about the Snort-devel mailing list