[Snort-users] Announce: StreamDB

Joel Esler jesler at ...1935...
Wed Feb 2 10:32:45 EST 2011


This looks very interesting.  Nice work, I'll personally take a look at that
later.

Joel

On Tue, Feb 1, 2011 at 3:26 PM, Martin Holste <mcholste at ...11827...> wrote:

> I wanted to let these communities know about a new open-source project
> called StreamDB (http://code.google.com/p/streamdb/) I've just
> published that's proven to be extremely helpful for my analysts.  It
> is a fast and simple tool for quickly viewing traffic related to IDS
> alerts (or any IP-based event) which specializes in ultra-fast
> retrievals from very large data sets.  It can hook into Snorby as it
> is OpenFPC compatible.  It is also very effective for PCRE searching
> traffic from a given source or destination IP address.  Streams are
> rotated out based on configured retention size in a ring-buffer
> fashion.  From the project home page:
>
> StreamDB is a high-performance framework for storing network streams.
> The current version uses Vortex IDS to read the streams from a file or
> network interface and saves them to an indexed DB and data file. Web
> code provides an URL-based query interface. There is also a
> command-line interface which includes the ability to read piped
> queries from STDIN. In addition to almost instant retrieval by IP
> address, StreamDB also allows PCRE searches and file type searches on
> streams if an IP address is provided as an initial filter. The system
> can handle recording gigabit line-speed networks and can retrieve
> arbitrary streams from terabytes of data in milliseconds. It is
> designed to be a complimentary tool to intrusion detection systems to
> aid security analysts.
>
> Here are some query examples:
>
> http://streamdb/?srcip=10.0.0.1
> http://streamdb/?srcip=10.0.0.1&dstip=1.1.1.1&dstport!80
> http://streamdb/?srcip=10.0.0.1&dstip=1.1.1.1&dstport=80&start=2 weeks
> ago&end=now
> http://streamdb/?srcip=10.0.0.1&pcre=example.com
> http://streamdb/?srcip=10.0.0.1&pcre=MZ.*PE\x00\x00
> http://streamdb/?srcip=10.0.0.1&sort=1&as_hex=1
> http://streamdb/?srcip=10.0.0.1&raw=1
> http://streamdb/?srcip=10.0.0.1&offset=1000&limit=200
> http://streamdb/?srcip=10.0.0.1&filetype=executable
>
> Examples from the CLI:
> ./sdb --srcip 10.0.0.1 --filetype pdf --headers-only
> tail -f /var/log/snort/alert | ./sdb > alert_streams.txt
>
> All of these queries will return in a second or two, assuming that the
> IP's referred to aren't busy web servers or NAT points.  IP's with
> many connections will benefit from more specific search filters for
> time and/or destination IP address.  Some non-scientific benchmarks on
> commodity hardware searching 5 TB of data: PCRE search for a given
> srcip with 1000 connections completes in about one second.  A similar
> search for a srcip with 50,000 connections will take about two
> minutes.  A lookup for a given srcip/dstip pair will complete in less
> than a second, including browser render time.
>
> Example output as text/plain:
>
> Returning 2 of 2 at offset 0 from Sun Jan 30 11:56:11 2011 to Sun Jan
> 30 11:56:11 2011
>
> 2011-01-30 11:56:11 192.168.58.52:4099 -> 131.243.2.191:80 13s 512
> bytes FIN ASCII text, with CRLF line terminators
>
> GET /bro-workshop-2009-2/slides/Installation.pdf HTTP/1.1
> Host: www.bro-ids.org
> User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13)
> Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-us,en;q=0.5
> Accept-Encoding: gzip,deflate
> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
> Keep-Alive: 115
> Referer: http://www.bro-ids.org/bro-workshop-2009-2/slides/
> X-Do-Not-Track: 1
> Connection: keep-alive
>
>
>
> 2011-01-30 11:56:11 192.168.58.52:4099 <- 131.243.2.191:80 13s 778247
> bytes FIN PDF document, version 1.3
>
> 200 OK
> Connection: Keep-Alive
> Date: Sun, 30 Jan 2011 17:56:11 GMT
> Accept-Ranges: bytes
> ETag: "8f724e-bde6b-47617252eebc0"
> Server: Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 OpenSSL/0.9.8e DAV/2
> PHP/5.3.5 with Suhosin-Patch mod_python/3.3.1 Python/2.6.6
> mod_wsgi/2.8
> Content-Length: 777835
> Content-Type: application/pdf
> Last-Modified: Sat, 17 Oct 2009 01:08:07 GMT
> Keep-Alive: timeout=5, max=100
> X-HTTP-Version: 1.1
>
> %PDF-1.3
> %...........
> 4 0 obj
> << /Length 5 0 R /Filter /FlateDecode >>
> stream
> x..VM..7
..W.T..z,i4....I.-.bS..!.a...nl..=. ...%O.Q.....|.Y.(..|.......).
> 9........l.h.'E....-....&.7]....... .}Fvr....}.x...
> .)...^k."U.rC.....w.N<...Z..u<..Z..e.j....4T.Hpj.........u...../g.....n.....o.......R....*.Do.9<.*]>...(...I8....ikJ_.T...:......c|..Ki..Q..>.U.MZ...*...!........jKik~7.qg.iw#.......|..............D/.\Yx..v...2<..d.
> O./...<...&.DDF....x..K.
> Qy...|..f
> F>.2k.v....^v..{^<..Q..v....t}.f;x.e.S..]...U.7......l..uD...
> <clipped for brevity>
>
> I hope that others have found it as useful as we have for rapidly
> investigating IDS alerts.  Please use the project page or email me to
> let me know about any questions, issues, or suggestions.
>
> Thanks,
>
> Martin
>
> --
Joel Esler
Skype:eslerjoel
http://blog.snort.org && http://blog.clamav.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20110202/5a0df804/attachment.html>


More information about the Snort-users mailing list