[Snort-users] Snort only partially alerting

Frank Calone fc10011001 at ...11827...
Fri Jul 19 15:43:36 EDT 2013


A thank you to all who helped to resolve this issue with partial alerting
on events.  It turns out that we had packet truncation (due to a
large framing size)  which was causing us to miss alerts.  This was noticed
when I ran snort in the foreground.  The following message showed up:
(snort_decoder) WARNING: IP dgm len > captured len

We had an older Snort system that was alerting when our new one was not.
The network tap is a vlan though I don't know if that contributed to the
problem.  Finding an external site where I was able to consistenly
reproduce the problem helped because I was then able to do both tcpdump and
snort binary captures. Comparing between the two showed clearly Snort was
not processing everything.   I think because there were so many truncated
packets, the syslog supressed the alerts when I was running Snort as a
Daemon.  I actually had two issues to blame for partial alerting.
The first fix was to set the packet size to the maximum with this setting
-P 65535
The second fix was to tell Snort to ignore checksum errors
-k none

I checked twice this week to compare results and each time I had a 1 to 1
match between the two systems.  Joel had advised of a better rule to use,
so I am glad to say we are actually getting a few more hits than our old
content rule was alerting on.  Great stuff.  Thanks for the help!

Frank

On Mon, Jul 15, 2013 at 1:32 PM, Joel Esler <jesler at ...1935...> wrote:

> Sounds good.  Thanks.
>
>  On Jul 15, 2013, at 10:56 AM, Frank Calone <fc10011001 at ...11827...> wrote:
>
>  Joel,
>   Thanks much.  I'm just so glad to have your tool finally working.  Now
> hopefully I can get my boss the exe files he has been wanting for a long
> time.  I will compare the alerts between the two systems this week and am
> hopeful everything matches.  I will let you know how that goes.  Thanks
> again!!!
>
> Frank
>
> On Fri, Jul 12, 2013 at 5:11 PM, Joel Esler <jesler at ...1935...> wrote:
>
>> If you remember, you were getting warnings:
>>
>>  (snort_decoder) WARNING: IP dgm len > captured len
>>
>>
>> The problem is that we had to figure out *why* we were getting those
>> warnings.  Turns out apparently that the device that is passing packets to
>> your machine is going so in very large “jumbo” frames.  While 99% of Snort
>> users don’t have to deal with this, every once in awhile, we do find this
>> issue.  You just happened to be in that 1%.  Annoying, I know.
>>
>>
>>  On Jul 12, 2013, at 4:46 PM, Frank Calone <fc10011001 at ...11827...> wrote:
>>
>>  Joel,
>>    Russ was able to determine the problem.  I will forward you our
>> correspondence if you want to see it (I left you off our correspondence as
>> I am sure you have plenty just watching all the user group posts).  Packet
>> truncation was occuring.  Using a setting of -P 65535 resolved the issue.
>> Well, I hope it has fixed it anyways.  I plan to compare the new server
>> hits to the old one all of next week.  If we match, I will be ecstatic!  At
>> that time I ought to post to the whole community the resolution (end of
>> next week).  I am troubled though by what I see as a weakness in your tool
>> alerting me to this condition.  I don't mean to be critical as there are a
>> lot of complexities and this tool has tried to deal with all of them.  It
>> is not a simple thing to do.  I think highly of what this tool does and all
>> of your efforts.  It does make a real difference in helping to protect
>> networks which is an absolute must given the poor state of computing
>> security today (it all starts with the sheer number of vulnerabilities each
>> day/week/month.  I am 30+ years of just IT security experience and I am
>> disgusted with the state of IT security today.  Just my personal
>> opinion.).  Even so, truncating packets is a serious problem and a error
>> warning message should have shown up in bold letters somewhere and not just
>> at shutdown of the snort process.  For that matter, even an abort would
>> have been better until the truncation was addressed in some fashion (say a
>> forced override or parameter setting).  What worries me is how many other
>> Snort users may have packet truncation going on such that they too are only
>> getting partial alerting and they don't even know it.  Instead they think
>> everything is being checked thoroughly and they are safe.
>>
>> Frank
>>
>> On Wed, Jun 26, 2013 at 3:42 PM, Joel Esler <jesler at ...1935...>
>> wrote:
>>
>>> OH NO, HAIR!
>>>
>>>
>>>  On Jun 26, 2013, at 3:41 PM, Frank Calone <fc10011001 at ...11827...> wrote:
>>>
>>>  I'd be glad to provide any other testing or results you might need to
>>> further diagnose what is happening, just let me know.  In the end it will
>>> make your product better which is good for everyone.  I have little hair
>>> left on my head at this time, hopefully it will grow back!
>>>
>>> Frank
>>>
>>> On Wed, Jun 26, 2013 at 3:39 PM, Joel Esler <jesler at ...1935...>
>>> wrote:
>>>
>>>> :)
>>>>
>>>> I'll talk to development about it, and what we can do.
>>>>
>>>> --
>>>> *Joel Esler*
>>>> Senior Research Engineer, VRT
>>>> OpenSource Community Manager
>>>> Sourcefire
>>>>
>>>>  On Jun 26, 2013, at 3:36 PM, Frank Calone <fc10011001 at ...11827...>
>>>> wrote:
>>>>
>>>>  The errors only showed up when I ran it in the foreground.  Running
>>>> it in the background (which is how I was running snort for the first few
>>>> weeks) resulted in no warning messages in the syslog file.  At the very
>>>> least then you would think I would have gotten 1514 bytes, but that does
>>>> not look to have happened either.  Still seems like this should have
>>>> aborted or alerted better.  I guess I am thinking that after about 3-4
>>>> weeks of struggling with this, it came down to an issue of the capture
>>>> length not being right.  Why, I don't know.  How that is handled by Snort I
>>>> think is where there is room for improvement.  Thanks again for all the
>>>> help with this.  I am relieved, well almost.  Once I do some more testing
>>>> and have the alerts show up, then I can celebrate.
>>>>
>>>> Frank
>>>>
>>>> On Wed, Jun 26, 2013 at 3:25 PM, Joel Esler <jesler at ...1935...>
>>>> wrote:
>>>>
>>>>> Well, Snort won't know the size of the packets being presented to it.
>>>>>  You *were* getting an error.  The one about your IP dgm length being
>>>>> smaller than the packet received.
>>>>>
>>>>> but yeah, by default, Snort's snaplen is 1514.  The packets you are
>>>>> presenting to Snort are apparently bigger than that. (Jumbo frames?)
>>>>>
>>>>> But yeah, you were receiving errors, we just had to figure out what
>>>>> was causing it.
>>>>>
>>>>>
>>>>>
>>>>>  On Jun 26, 2013, at 3:20 PM, Frank Calone <fc10011001 at ...11827...>
>>>>> wrote:
>>>>>
>>>>>  Joel,
>>>>>    So the million dollar question is what is going wrong such that
>>>>> the capture length must be hard coded in this way?  Isn't there still a
>>>>> fundamental flaw somewhere?  I see no real mention of this as an example of
>>>>> a command line option all users should be using to start Snort with.  What
>>>>> is equally concerning is the lack of any error message in the syslog
>>>>> stating that the capture length is being truncated.  If I hadn't had
>>>>> another snort server to compare my results against I would have thought
>>>>> everything is working fine.  It did get occasional hits, at least
>>>>> initially.  Just my opinion, but this kind of packet truncating ought to
>>>>> cause Snort to abort with a clear error message noting the problem.
>>>>> Otherwise, one might think they are doing monitoring when in reality most
>>>>> everything is going to a bit bucket someplace.  Please don't take this the
>>>>> wrong way, I really like the Snort tool already and I've hardly begun using
>>>>> it.   The out of the box startup though had the appearance of working when
>>>>> it was really failing misserably.  I'll do some testing (downloads to make
>>>>> sure it is getting them).  If it works, I'll post it to the entire group
>>>>> list.
>>>>>
>>>>> Frank
>>>>>
>>>>> On Wed, Jun 26, 2013 at 3:11 PM, Joel Esler <jesler at ...1935...>
>>>>> wrote:
>>>>>
>>>>>> Okay, that works.
>>>>>>
>>>>>> Make sure you start snort with that as well
>>>>>>
>>>>>>
>>>>>> Snort -c /etc/snort.conf -P 9000 -l ./log -i p1p1 -D
>>>>>>
>>>>>> or whatever.
>>>>>>
>>>>>>
>>>>>>  On Jun 26, 2013, at 2:59 PM, Frank Calone <fc10011001 at ...11827...>
>>>>>> wrote:
>>>>>>
>>>>>>  Joel,
>>>>>>    As requested.  Looking better.
>>>>>>
>>>>>> Frank
>>>>>>
>>>>>> On Wed, Jun 26, 2013 at 2:48 PM, Joel Esler <jesler at ...1935...>
>>>>>> wrote:
>>>>>>
>>>>>>> Try "snort -l ./logall -b -i p1p1 -P 9000 src 157.98.75.158 or dst
>>>>>>> 157.98.75.158"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  On Jun 26, 2013, at 2:37 PM, Frank Calone <fc10011001 at ...11827...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Joel,
>>>>>>>    I ran both captures at the same time and I downloaded putty.exe
>>>>>>> from the Internet.  Here are the command lines I used and the respective
>>>>>>> output files.
>>>>>>>
>>>>>>> snort -l ./logall -b -i p1p1 src 157.98.75.158 or dst 157.98.75.158
>>>>>>>
>>>>>>> tcpdump -i p1p1 -N -w tcpdump.jun26.v1.pcap src 157.98.75.158 or dst
>>>>>>> 157.98.75.158
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Frank
>>>>>>>
>>>>>>> On Wed, Jun 26, 2013 at 2:27 PM, Joel Esler <jesler at ...1935...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> use the same bpf syntax you use for tcpdump.
>>>>>>>>
>>>>>>>> host 157.98.75.158
>>>>>>>>
>>>>>>>> or whatever.
>>>>>>>>
>>>>>>>>
>>>>>>>>  On Jun 26, 2013, at 2:19 PM, Frank Calone <fc10011001 at ...11827...>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  something is wrong with the command syntax-
>>>>>>>>
>>>>>>>>
>>>>>>>> root at ...16442... /var/log/snort $ snort -l ./logall -b -i p1p1 ip
>>>>>>>> 157.98.75.158
>>>>>>>> Running in packet logging mode
>>>>>>>>         --== Initializing Snort ==--
>>>>>>>> Initializing Output Plugins!
>>>>>>>> Snort BPF option: ip 157.98.75.158
>>>>>>>> Log directory = ./logall
>>>>>>>> pcap DAQ configured to passive.
>>>>>>>> Acquiring network traffic from "p1p1".
>>>>>>>> ERROR: Can't set DAQ BPF filter to 'ip 157.98.75.158'
>>>>>>>> (pcap_daq_set_filter: pcap_compile: syntax error)!
>>>>>>>> Fatal Error, Quitting..
>>>>>>>>
>>>>>>>> Frank
>>>>>>>> On Wed, Jun 26, 2013 at 1:38 PM, Joel Esler <jesler at ...1935...>
>>>>>>>>  wrote:
>>>>>>>>
>>>>>>>>> You can filter to just an IP with Snort the same way you do with
>>>>>>>>> tcpdump.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> snort -l ./log -b ip 123.123.123.123
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you send me the actual pcaps from both tcpdump and Snort on
>>>>>>>>> the same box, at the same time capturing the same thing?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  On Jun 26, 2013, at 1:35 PM, Frank Calone <fc10011001 at ...14542....>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>  Joel,
>>>>>>>>>    Here are two samples of the snort -dvr playback of binary
>>>>>>>>> captures.  The 2 MB capture represents the TCPDUMP pcap capture I made
>>>>>>>>> where I filtered for just my IP in either the src or dst, IP address.  It
>>>>>>>>> shows the putty.exe download.  The second playback is where I ran snort
>>>>>>>>> options to capture all packets into a binary file:
>>>>>>>>>
>>>>>>>>> ./snort -l ./log -b
>>>>>>>>> I wrote a quick PERL prgram to take the playback file and when it
>>>>>>>>> found my IP address, it took all output lines thereafter until it hit a
>>>>>>>>> delimiter (the =+=+=+=+ boundry lines).  This way you can see what Snort is
>>>>>>>>> showing as the packets to be analyzed versus what TCPDUMP showed.  The
>>>>>>>>> TCPDUMP program was included in the CENTOS OS that was loaded onto our
>>>>>>>>> Snort system.  So the pcap capture was made on the same system monitoring
>>>>>>>>> the same interface (p1p1) as what Snort is running on.   In looking at the
>>>>>>>>> Snort binary capture it becomes readily apparent there is a problem.
>>>>>>>>> Interesting that I do seem to get the initial http session request and
>>>>>>>>> server response.  After that, virtually nothing other than headers.
>>>>>>>>>
>>>>>>>>> Frank
>>>>>>>>>
>>>>>>>>> On Wed, Jun 26, 2013 at 10:49 AM, Joel Esler <
>>>>>>>>> jesler at ...1935...> wrote:
>>>>>>>>>
>>>>>>>>>> BTW -- Here is an older presentation on Razorback:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  On Jun 25, 2013, at 6:04 PM, Frank Calone <fc10011001 at ...14459.....>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>  I will look into what razorback offers.  I do think though I
>>>>>>>>>> will want to run most if not all of your rules at some time.  Perhaps not
>>>>>>>>>> to do individual captures on an alert, but rather just to be alerted so we
>>>>>>>>>> can look at the web traffic and such to then determine if it is a real
>>>>>>>>>> problem or not.  At this time I'm just trying to get my boss what he wants,
>>>>>>>>>> which is just exe captures.
>>>>>>>>>>
>>>>>>>>>> About what the networking guys are giving us, it sure looks like
>>>>>>>>>> they are giving us all the traffic.  The TCPDUMP testing I did sure looks
>>>>>>>>>> just fine.  the datagram sizes look normal and have payload.  something is
>>>>>>>>>> being translated (rather truncated) when Snort tries to look at it.  I'm
>>>>>>>>>> wondering if it is possible to have Snort do a full capture for my IP
>>>>>>>>>> only.  My boss might be okay with me sending that to you so you can see
>>>>>>>>>> what I am seeing here.  The size ought to be small enough to that I can
>>>>>>>>>> just email.  If this is an option, can you let me know what command to
>>>>>>>>>> use.  I can see how to do it with tcpdump, I'm not sure how to do it with
>>>>>>>>>> Snort.
>>>>>>>>>>
>>>>>>>>>> Frank
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 25, 2013 at 5:54 PM, Joel Esler <
>>>>>>>>>> jesler at ...1935...> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sounds like the networking guys may be confused about what they
>>>>>>>>>>> are giving you.  Or you aren't getting the full thing, or whatever.
>>>>>>>>>>>
>>>>>>>>>>> In any case it sounds like what you are trying to do is
>>>>>>>>>>> Razorback (We have a project for that!)
>>>>>>>>>>>
>>>>>>>>>>> http://labs.snort.org/razorback/
>>>>>>>>>>>
>>>>>>>>>>> It sniffs all files off the wire (not only exes) and then judges
>>>>>>>>>>> them to be malicious or not through a variety of methods.
>>>>>>>>>>>
>>>>>>>>>>>  --
>>>>>>>>>>> *Joel Esler*
>>>>>>>>>>> Senior Research Engineer, VRT
>>>>>>>>>>> OpenSource Community Manager
>>>>>>>>>>> Sourcefire
>>>>>>>>>>>  On Jun 25, 2013, at 5:50 PM, Frank Calone <fc10011001 at ...13704......>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>  What I am told is that the networking guys gave us a vlan
>>>>>>>>>>> tap.  Our intent is to run the full Snort server on a standalone system
>>>>>>>>>>> rather than on the appliance.  We were unable to get the logging to work on
>>>>>>>>>>> the appliance.  We want to have session captures so when we get an alert,
>>>>>>>>>>> we can pull the exe file out of the session data.  Then we can take the exe
>>>>>>>>>>> and run it on fireeye to see if it gets tagged as malicious.  We can also
>>>>>>>>>>> check the md5 against known values that are malicious.  So, having a log
>>>>>>>>>>> capture is really our end goal.  I do think at some point we may turn up
>>>>>>>>>>> the other rules, just depending upon what that does to system load.  My
>>>>>>>>>>> bosses main goal is to get the exe files first.  We did have a nice java
>>>>>>>>>>> hit that I got from the emerging threats.  There certainly is real value in
>>>>>>>>>>> using all the rules.  We see a lot of exe downloads each day and the
>>>>>>>>>>> problem is knowing whether they are malicious or not.  This is why he wants
>>>>>>>>>>> the session packet capture when an alert is triggered.  I hope I answered
>>>>>>>>>>> your question.  I'm the one that is supposed to be confused.  I must be
>>>>>>>>>>> confusing you then.
>>>>>>>>>>>
>>>>>>>>>>> Frank
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 25, 2013 at 5:42 PM, Joel Esler <
>>>>>>>>>>> jesler at ...1935...> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> And you are trying to capture it on the box you have sitting
>>>>>>>>>>>> somewhere else?
>>>>>>>>>>>>
>>>>>>>>>>>> I'm confused.
>>>>>>>>>>>>
>>>>>>>>>>>>  On Jun 25, 2013, at 5:37 PM, Frank Calone <
>>>>>>>>>>>> fc10011001 at ...11827...> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>  It's an IBM Proventia 4000 or 5000 series systems.  As far as
>>>>>>>>>>>> the lan type being the same, I would say not the same.  These devises are
>>>>>>>>>>>> IPS units, so they are seeing the raw traffic in stream.
>>>>>>>>>>>>
>>>>>>>>>>>> Frank
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jun 25, 2013 at 5:31 PM, Joel Esler <
>>>>>>>>>>>> jesler at ...1935...> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, that's what that error means. (Packet says its bigger
>>>>>>>>>>>>> than it actually is).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is the box you are trying to sniff getting the same span as
>>>>>>>>>>>>> the appliance?  What kind of appliance is it?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  On Jun 25, 2013, at 5:28 PM, Frank Calone <
>>>>>>>>>>>>> fc10011001 at ...11827...> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Turn snort on to do full packet captures, all sessions:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ./snort -l ./log -b
>>>>>>>>>>>>> I've turned this on for like a minute or so, did my download,
>>>>>>>>>>>>> then did a playback of the capture and looked for just traffic tied to my
>>>>>>>>>>>>> PC's IP.  I can see the initial get request and a reply, but after that it
>>>>>>>>>>>>> is almost nothing as far as a payload goes.  If this is what snort is
>>>>>>>>>>>>> testing the rules against, it is no wonder I am getting no hits.  The
>>>>>>>>>>>>> content string i'm testing for doesn't show up.  So, when I say partial
>>>>>>>>>>>>> packets, I mean packets with no payload.  I think this has something to do
>>>>>>>>>>>>> with the error message I saw when I did the snort testing in the foreground:
>>>>>>>>>>>>>
>>>>>>>>>>>>>  (snort_decoder) WARNING: IP dgm len > captured len
>>>>>>>>>>>>>
>>>>>>>>>>>>> It simply looks like all of the payload is being truncated,
>>>>>>>>>>>>> which is what this error message seems to be telling me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Frank
>>>>>>>>>>>>>
>>>>>>>>>>>>> ps: Is the file-identify simply a category you use to put
>>>>>>>>>>>>> rules into, say much like "exploit-kit"?  I really don't think the rules
>>>>>>>>>>>>> are the problem here.  I've even tried using the exact same Snort rules my
>>>>>>>>>>>>> co-worker has to test for exe files.  So, the rule works on our appliance,
>>>>>>>>>>>>> but not for me.  Why has been this weeks old headache I have had.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jun 25, 2013 at 5:11 PM, Joel Esler <
>>>>>>>>>>>>> jesler at ...1935...> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jun 25, 2013, at 5:10 PM, Frank Calone <
>>>>>>>>>>>>>> fc10011001 at ...11827...> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > Joel,
>>>>>>>>>>>>>> >    What confuses me is if I turn on full packet capture
>>>>>>>>>>>>>> mode, I ought to see complete traffic regardless of any rules.  If what I
>>>>>>>>>>>>>> am seeing in that mode is partial packets, then it sure seems any rules I
>>>>>>>>>>>>>> may test for are going to be intermitent.  I'm not sure how to set the
>>>>>>>>>>>>>> file-identify rule category you note.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you mean "turn on full packet capture mode"  and what
>>>>>>>>>>>>>> do you mean "partial packets"?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> <test.24.75.158.txt><testpcap.v3.jun20.txt>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> <tcpdump.jun26.v1.pcap><snort.log.1372271318>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> <snort.log.1372272837>
>>>>>>
>>>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20130719/ab3d3462/attachment.html>


More information about the Snort-users mailing list