[Snort-users] Snort Architecture Part 1: The Packet Decoder

Martin Roesch roesch at ...421...
Wed Mar 14 00:46:58 EST 2001


Welcome Snorters and Snortettes, tonight I'm going to begin a new
experiment.  I've been receiving a lot of pokes and hints and such from
various people requesting better architectural documentation for Snort,
so rather than sitting down and trying to describe the whole thing at
once, I figured I'd try to attack the problem on a subsystem by
subsystem basis.  Tonight is the first of these little architectural
overviews, I hope you guys find them useful!

For part 1, we'll discuss one of the simpler subsystems, the packet
decoder.  The Snort packet decoder is built to be as fast as possible
while providing accurate decodes and information on the packets as they
arrive in the program.  The basis of the architecture is the concept of
overlaying structure on raw data using predefined structs that represent
the actual layout of a particular protocol header in memory.  When a
packet arrives, a pointer to the raw data is set and cast to the type of
the particular protocol header that we're pointing at.  For example, we
have the following struct for the IP header:

typedef struct _IPHdr
{
#if defined(WORDS_BIGENDIAN)
    u_int8_t ip_ver:4,      /* IP version */
    ip_hlen:4;     /* IP header length */
#else
    u_int8_t ip_hlen:4, ip_ver:4;
#endif
    u_int8_t ip_tos;        /* type of service */
    u_int16_t ip_len;       /* datagram length */
    u_int16_t ip_id;        /* identification  */
    u_int16_t ip_off;       /* fragment offset */
    u_int8_t ip_ttl;        /* time to live field */
    u_int8_t ip_proto;      /* datagram protocol */
    u_int16_t ip_csum;      /* checksum */
    struct in_addr ip_src;  /* source IP */
    struct in_addr ip_dst;  /* dest IP */

}      IPHdr;

If you check your Stevens (or whatever TCP/IP book you like) you'll see
that this structure corresponds to the actual layout of an IP header as
it's represented in the book (and more importantly, RFC791). There are a
few things to note while we're at this point.  

Before I continue, I'll answer a question I get from time to time: "Why
does Snort define its own protocol headers when they're available in
/usr/include/netinet on most OSs?"  The answer is portability. 
Depending on every operating system to implement these structs the same
way with the same variable names is not practical, so we define them
ourselves and try to give them names to avoid namespace collisions with
the native OS that we're running on.

There is an overarching packet structure called "Packet" that contains
pointers to all of these structs to allow for referencing of every
available protocol decode that Snort can analyze.  The structure for
this struct is very straightforward and it contains a number of "helper"
variables and flags:

(NOTE: this struct is abbreviated for the purposes of illustration, see
decode.h for the full definition!)
typedef struct _Packet
{
    /* data link layer */
    EtherHdr *eh;               /* standard TCP/IP/Ethernet/ARP headers
*/
    EtherARP *ah;

    /* network layer */
    IPHdr *iph, *orig_iph;      /* and orig. headers for ICMP_*_UNREACH
family */

    /* transport layer */
    TCPHdr *tcph, *orig_tcph;
    UDPHdr *udph, *orig_udph;
    ICMPHdr *icmph, *orig_icmph;

    /* application layer */
    u_int8_t *data;         /* packet payload pointer */
    u_int16_t dsize;        /* packet payload size */

    /* helpers */
    u_int8_t frag_flag;     /* flag to indicate a fragmented packet */
    u_int16_t frag_offset;  /* fragment offset number */
    u_int8_t mf;            /* more fragments flag */
    u_int8_t df;            /* don't fragment flag */
    u_int8_t rf;            /* IP reserved bit */

    u_int16_t sp;           /* source port (TCP/UDP) */
    u_int16_t dp;           /* dest port (TCP/UDP) */

    Options ip_options[40]; /* ip options decode structure */
    u_int32_t ip_option_count;  /* number of options in this packet */
    
    Options tcp_options[40];    /* tcp options decode struct */
    u_int32_t tcp_option_count;
    u_int8_t csum_flags;        /* checksum flags */

} Packet;

When a packet arrives, the appropriate pointers for the protocols
contained within those packets are set by the decoders.  If a protocol
pointer is not set to point within a packet, it is left NULL.  The
helper vaiables are set for packet fields that are either frequently
referenced or somewhat complex to decode.  For example, the source and
destination ports of a TCP/UDP packet are frequently referenced in
Snort, so it makes sense to convert them from network order and store
them in a variable that can be readily referenced anywhere it's needed
in the program instead of having to call the ntohs() function repeatedly
at run time for each reference to a port.  

So anyway, when a packet arrives in Snort, the "grinder" function is
called, which is a function pointer to the data link layer decoder that
has been selected for the particular interface by the libpcap
initilization code.  We'll talk about how that happens in a later
paper.  Suffice to say, the proper layer 2 (data link) decoder is called
when a packet is acquired.  For the sake of argument (and this
document), we'll say that the hypothetical Snort run that we're
discussing right now is using the Ethernet interface.  Once the packet
is handed to the Ethernet decoder, there are several steps that are
followed.  Lets take a look at the DecodeEthPkt function and follow
along with what happens (we'll abbreviate this function as well for the
sake of simplicity, look in decode.c for more info):

First off, note the function's argument list.  A pointer to a
pre-allocated packet structure is passed, as is a pointer to the libpcap
packet header (a struct with timestamp and packet size information), and
a pointer to the raw packet called "pkt".
void DecodeEthPkt(Packet * p, struct pcap_pkthdr * pkthdr, u_int8_t *
pkt)
{

These variables just help us keep track of how big the packet is:     
    u_int32_t pkt_len;      
    u_int32_t cap_len;     

We clean out the packet data struct to make sure all the pointers are
initialized to NULL and that all the other variables are clean as well:
    bzero((char *) p, sizeof(Packet));

Set the packet lengths.  The *packet* length might be different than the
*captured* length.  The maximum size of the caplen is determined by the
program snaplen.  This variable is adjustable in Snort with the -P
command line option:
    pkt_len = pkthdr->len;  /* total packet length */
    cap_len = pkthdr->caplen;   /* captured packet length */

Make sure the packet length isn't greater than the captured length, that
could cause problems down the road:
    if(snaplen < pkt_len)
        pkt_len = cap_len;

Verify that we have an ethernet header that's at least 14-bytes long
(i.e. the size of an ethernet header):
    if(p->pkth->caplen < ETHERNET_HEADER_LEN)
    {
        if(pv.verbose_flag)
        {
            ErrorMessage("Captured data length < Ethernet header length!
(%d bytes)\n", p->pkth->caplen);
        }
        return;
    }

If everything else has validated properly up to this point, we set the
Ethernet header pointer to point to the start of the raw data.  Once
this pointer has been set, the structure of the raw data is set by the
casting to the protocol header type.
    p->eh = (EtherHdr *) pkt;

Now that this layer of structure has been applied to the raw data, we
can make references to header fields directly through pointer
dereferences and get the raw data nice and packaged up for our nefarious
purposes:
    switch(ntohs(p->eh->ether_type))
    {

If the ether_type (i.e. protocol) field of the packet indicates that
we've got an IP packet (protocol number 0x800) we call the IP header
decoder.  Note that we pass the length of the captured data minus the
length of this header, and we move the raw data pointer up to the end of
this header as well:
        case ETHERNET_TYPE_IP:
            DecodeIP(p->pkt + ETHERNET_HEADER_LEN, cap_len -
ETHERNET_HEADER_LEN, p);

Once we're done decoding the packet, we return from whence we came:
            return;
    }
}

The IP decoder is much like the Ethernet decoder, the packet lengths are
validated for protocol specification minimums, then the structure is
overlaid on the raw data.  The offsets and data lengths are updated as
the protocol is decoded and the next layer down is decoded and so on.  

There are a few things to be aware of.  

1) If the packet is an IP fragment the IP protocol field will be set but
the transport layer pointers will be left NULL.  This is done because
reliable data cannot be extracted from the information beyond the IP
header until the fragmented packet is reassembled.  When a fragmented
packet comes in, the frag_flag variable is set for easy reference within
the plugins.  The actual fragment payload of a fragmented packet is
available in the p->data pointer and its size is noted in the p->dsize
member.

2) The MF, DF and reserved bit (RB) flags are available and are set by
the IP decoder.

3) IP and TCP option arrays are filled in as needed.  Easy ways to check
if there are options present in a packet are by checking the
ip_option_count or tcp_option_count.

4) The application layer is left undecoded by the decoder stage. 
Preprocessors may be used to perform this task currently, in the future
we'll have better methods.

Once a packet has been decoded and run through the various detection and
processing mechanisms, it is thrown away.  If a packet is to be kept
around for later usage, space must be allocated for it and it must be
duplicated into the new memory (which slows everything down).

That's pretty much the nickel tour of how the packet decoder works.  Did
you find it helpful?  Useless?  Did you want more information?  Less? 
Let me know what you think of this paper and I'll make changes and
improvements for the next one.

     -Marty

--
Martin Roesch
roesch at ...421...
http://www.snort.org




More information about the Snort-users mailing list