[Snort-devel] Snort full log files need heirarchy

David Ford david+cert at ...1022...
Sat Mar 23 12:24:11 EST 2002


Here is some food for thought.

Store the directories in two char zero prefixed hexidecimal format.  For 
example, 208.179.59.1 would become /d0/b3/3b/01/data

The advantages of this are:

a) each directory is always two chars long
b) a directory listing can be naturally sorted at zero cost.  01-ff is 
natural the entire way v.s. trying to to naturally sort 20 v.s. 199.
c) IPs are already known in hex form, a simple type bitshift && mask|add 
int<>char is all that's needed to convert either direction

The disadvantages of this are:

a) those who don't understand hex might be confused?

To get each octet:
#define o1(value) ((value>>24) & 0xff)
#define o2(value) ((value>>16) & 0xff)
#define o3(value) ((value>>8) & 0xff)
#define o4(value) (value & 0xff)

To reassemble:
#define reassemble(o1, o2, o3, o4) (o1<<24 + o2<<16 + o3<<8 + o4)

To create the directory string:
sprintf(string, "%02x/%02x/%02x/%02x/", o1, o2, o3, o4);

All very simple, and pretty quick too.  Adjust for IPv6 if desired.

David

Bit Stream wrote:

>
>
>> From: James Hoagland <hoagland at ...60...>
>
> ...
>
>>> I'd propose that for the next version of snort, that you create a
>>> directory heirarchy.  Where IP 1.2.3.4 becomes $LOGDIR/1/2/3/4/ when
>>> snort is running in full log mode.  This would solve a lot of
>>> problems by limiting the total number of directories in any
>>> directory to 255.  Perhaps it is problematic performance wise, but
>>> real performance oriented people shouldn't be running in full log
>>> mode in real-time anyway.
>>
>>
>> This might be a good idea.  It is not a particularly good idea to let
>> the attacker control your performance.  Especially if they can push
>> it over a file system limit.  (Although, of course, they can control
>> how many alerts you get.)  I would suggest this as an option though,
>> to preserve backwards compatibility.
>
>
> Let me reinforce a point that I thought I made.  I'm talking about 
> doing this expansion out of band (relative to the snort collection).  
> That is, I am planning to run the collector then postprocess the log 
> file to a more useful form for analysis.  In this sense, yes, allowing 
> an attacker to control performance is a bad thing, on the other hand, 
> it's out of band, so it's not necessarily critical.  As for time spent 
> by the collector recording all the data, there is a performance 
> implication there, but I suspect we could spend many days and beers 
> arguing about the value of different rules.  This is why most IDS 
> vendors are trying to find ways to make the number of rules v. packet 
> processing time a low-order equation.
>
>> SnortSnarf used to place all its files in one directory too.  But
>> now, it stores it in heirarchical directories for reasons like you
>> describe.  The pages corresponding to IP files are in nested
>> directories named after the first 3 bytes of the IP address.  I could
>> see why you might use 4 bytes here though.
>
>
> I actually contemplated 2 bytes and decided that was too little and 
> never contemplated stopping at 3.  Brian wrote to me separately and 
> discusssed inode depletion, which could be an issue, but at least 
> inodes are a function of mkfs and disk size, where as files-per-dir is 
> static per OS.
>
> I could code this in myself, and am willing to do the development work 
> and submit it if people other than me think it's a good idea (and 
> there isn't a consensus that it's a bad idea). Since I haven't tracked 
> snort development for all that long, I also wanted to make sure it 
> wasn't being done by someone already.
>
> Eric 







More information about the Snort-devel mailing list