[Snort-users] Base Barnyard and Unified Logs

Wes Young wcyoung at ...12754...
Sat Mar 26 06:22:26 EST 2005


These hacks all are great in theory...i'd rather just rip out the CID 
in the signature table.
  I really like populating the sig table pre-emptively, that I might do 
reguardless, but software
ppl need to revolve their "viewing" software around the sid. I think 
the PK that originally was in place (cid or whatever) was before
SID's were even involved and everything was just PK'd on the msg... 
(hense making the CID important in the DB schema, but not in
real life. If you re-vamp the output plugin along with the schema to 
reflect everything being key'd on the sid, the system would scale
much higher when you start incorporating more databases with teh 
product (i have about 4 diff db's that rely on the one snort_log
alone, not to mention the snort_alerts, all work well untill I have to 
clear one of them, i clear one, 2 of them have to be flushed and
re-init'd as well because of the stupid CID auto-increment stuff, and 
no, aanval (atleast the older version) isn't totally exempt
from this problem either). If they were all SID based, the joins would 
be fine reguardless of what i flush.

Actually the db plugin doesn't really have to be re-written come to 
think of it, just rip out the CID and base your software on the SID.
IMO that key shouldn't even be there.

I think the original problem I was having was because i re-init'd one 
of my databases and it threw off the rest, i didnt check to make sure 
base was ok until i had a 1G database of logs, by then it was too late 
and it labeled every sig as it's SID within base...

On Mar 26, 2005, at 5:47 AM, Dirk Geschke wrote:

> Hi Jerry,
>
>> Just saw the discussion about barnyard and DB's.  Here is some info
>> I gained in having to deal with consolidating data from two snort
>> DB's in to a single application.
>>
>> Now that generators have been assigned to various parts of snort,
>> they need to be employed in the DB schema (generator:sid:rev) as
>> a key to a signature.  The generator-id is needed since the
>> pre-processors usually start the SIDS=1!  The problem becomes more
>> complicated in that the signature, sensor, reference, and
>> classification tables are built on the fly by the DB-plugins.  The
>> plugins first try to grab the signature from the DB using msg
>> (sig_name), Rev (sig_rev) and SID (sig_sid). If found then use
>> the assigned (via MySql auto-increment) sig_id.  If not, create the
>> record.  Note that the generator-id is never mentioned in the DB.
>>
>> The signature, sensor, reference and classification tables are
>> "normalized" tables created on-the-fly by the database plugin.  Their
>> ordinal (created by the order of insertion) is used in the
>> other tables (eg. event) to save time and space.
>> If you are only using a single DB, there isn't any problem, except as
>> Joel wrote below, if you have to clean the DB, your mapping between
>> SID ->(sig_name,sig_sid,sig_rev) is lost.  If you are combining the
>> two DB's, for example an inside and an outside, into a single
>> application/DB like we are, you run in to data collisions and race
>> conditions.
>>
>>
>> To solve these issues, I ended up writing scripts to insert (read
>> preload) the following tables:
>> 	.signature, from all of the rules
>> 	.sensor (including the 'read from file' entries)
>> 	.reference (reference.config), and
>> 	.classification (classification.config)
>> The input to the scripts will never shrink.  Thus I will maintain
>> the mapping.
>
> a similar script exists as part of FLoP: rules.pl. It inserts all
> rules of the signature files to the database. This would also speed
> up insertiion of alerts since the signature is already part of the
> database.
>
> To solve the generator-id problem we use a hack, if the alert is
> not created by a "normal" rule, e.g created by a pre-processor
> then we insert the generator id in the field sig_rev since the
> revision is neither defined for preprocessors nor would there
> be any change within it. The big problem is that the generator
> id was never thought of as the database design was made. And
> more complicated, ACID/BASE won't use it...
>
> To learn more about FLoP take a look at
>
>   http://www.geschke-online.de/FLoP/
>
> Maybe you can use some of the tools which are part of the project
> or you can use it at all...
>
> Best regards
>
> Dirk
>
>
>
--
Wes

I wish I could come up with some witty sigs
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/enriched
Size: 4385 bytes
Desc: not available
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20050326/2c5c820c/attachment.bin>


More information about the Snort-users mailing list