[Snort-devel] Snort full log files need heirarchy
david+cert at ...1022...
Sat Mar 23 12:24:11 EST 2002
Here is some food for thought.
Store the directories in two char zero prefixed hexidecimal format. For
example, 22.214.171.124 would become /d0/b3/3b/01/data
The advantages of this are:
a) each directory is always two chars long
b) a directory listing can be naturally sorted at zero cost. 01-ff is
natural the entire way v.s. trying to to naturally sort 20 v.s. 199.
c) IPs are already known in hex form, a simple type bitshift && mask|add
int<>char is all that's needed to convert either direction
The disadvantages of this are:
a) those who don't understand hex might be confused?
To get each octet:
#define o1(value) ((value>>24) & 0xff)
#define o2(value) ((value>>16) & 0xff)
#define o3(value) ((value>>8) & 0xff)
#define o4(value) (value & 0xff)
#define reassemble(o1, o2, o3, o4) (o1<<24 + o2<<16 + o3<<8 + o4)
To create the directory string:
sprintf(string, "%02x/%02x/%02x/%02x/", o1, o2, o3, o4);
All very simple, and pretty quick too. Adjust for IPv6 if desired.
Bit Stream wrote:
>> From: James Hoagland <hoagland at ...60...>
>>> I'd propose that for the next version of snort, that you create a
>>> directory heirarchy. Where IP 126.96.36.199 becomes $LOGDIR/1/2/3/4/ when
>>> snort is running in full log mode. This would solve a lot of
>>> problems by limiting the total number of directories in any
>>> directory to 255. Perhaps it is problematic performance wise, but
>>> real performance oriented people shouldn't be running in full log
>>> mode in real-time anyway.
>> This might be a good idea. It is not a particularly good idea to let
>> the attacker control your performance. Especially if they can push
>> it over a file system limit. (Although, of course, they can control
>> how many alerts you get.) I would suggest this as an option though,
>> to preserve backwards compatibility.
> Let me reinforce a point that I thought I made. I'm talking about
> doing this expansion out of band (relative to the snort collection).
> That is, I am planning to run the collector then postprocess the log
> file to a more useful form for analysis. In this sense, yes, allowing
> an attacker to control performance is a bad thing, on the other hand,
> it's out of band, so it's not necessarily critical. As for time spent
> by the collector recording all the data, there is a performance
> implication there, but I suspect we could spend many days and beers
> arguing about the value of different rules. This is why most IDS
> vendors are trying to find ways to make the number of rules v. packet
> processing time a low-order equation.
>> SnortSnarf used to place all its files in one directory too. But
>> now, it stores it in heirarchical directories for reasons like you
>> describe. The pages corresponding to IP files are in nested
>> directories named after the first 3 bytes of the IP address. I could
>> see why you might use 4 bytes here though.
> I actually contemplated 2 bytes and decided that was too little and
> never contemplated stopping at 3. Brian wrote to me separately and
> discusssed inode depletion, which could be an issue, but at least
> inodes are a function of mkfs and disk size, where as files-per-dir is
> static per OS.
> I could code this in myself, and am willing to do the development work
> and submit it if people other than me think it's a good idea (and
> there isn't a consensus that it's a bad idea). Since I haven't tracked
> snort development for all that long, I also wanted to make sure it
> wasn't being done by someone already.
More information about the Snort-devel