[Snort-users] File preprocessor fails to capture files

Pablo Cantos Polaino pcantos at ...16842...
Fri May 8 17:59:42 EDT 2015


Hi Hui,

I've replaced config paf_max: 16000 by 60000 as you propose.

File type was not identified because I had disabled type_id option in
preprocessor file_inspect. I've replayed the tests with paf_max = 60000 and
both type_id enabled and disabled. In both cases the capture files are the
same (number and size) when sniffering from an interface and reading from a
PCAP file. So I'm pasting below the exit stats when type_id is enabled and
paf_max = 60000:

Exit stats when reading the PCAP file and type_id enabled:

===============================================================================
Run time for packet processing was 3.978146 seconds
Snort processed 3326 packets.
Snort ran for 0 days 0 hours 0 minutes 3 seconds
   Pkts/sec:         1108
===============================================================================
Memory usage summary:
  Total non-mmapped bytes (arena):       10190848
  Bytes in mapped regions (hblkhd):      122081280
  Total allocated space (uordblks):      8072896
  Total free space (fordblks):           2117952
  Topmost releasable block (keepcost):   133008
===============================================================================
Packet I/O Totals:
   Received:         3326
   Analyzed:         3326 (100.000%)
    Dropped:            0 (  0.000%)
   Filtered:            0 (  0.000%)
Outstanding:            0 (  0.000%)
   Injected:            0
===============================================================================
Breakdown by protocol (includes rebuilt packets):
        Eth:         3333 (100.000%)
       VLAN:            0 (  0.000%)
        IP4:         3333 (100.000%)
       Frag:            0 (  0.000%)
       ICMP:            0 (  0.000%)
        UDP:           40 (  1.200%)
        TCP:         3293 ( 98.800%)
        IP6:            0 (  0.000%)
    IP6 Ext:            0 (  0.000%)
   IP6 Opts:            0 (  0.000%)
      Frag6:            0 (  0.000%)
      ICMP6:            0 (  0.000%)
       UDP6:            0 (  0.000%)
       TCP6:            0 (  0.000%)
     Teredo:            0 (  0.000%)
    ICMP-IP:            0 (  0.000%)
    IP4/IP4:            0 (  0.000%)
    IP4/IP6:            0 (  0.000%)
    IP6/IP4:            0 (  0.000%)
    IP6/IP6:            0 (  0.000%)
        GRE:            0 (  0.000%)
    GRE Eth:            0 (  0.000%)
   GRE VLAN:            0 (  0.000%)
    GRE IP4:            0 (  0.000%)
    GRE IP6:            0 (  0.000%)
GRE IP6 Ext:            0 (  0.000%)
   GRE PPTP:            0 (  0.000%)
    GRE ARP:            0 (  0.000%)
    GRE IPX:            0 (  0.000%)
   GRE Loop:            0 (  0.000%)
       MPLS:            0 (  0.000%)
        ARP:            0 (  0.000%)
        IPX:            0 (  0.000%)
   Eth Loop:            0 (  0.000%)
   Eth Disc:            0 (  0.000%)
   IP4 Disc:            0 (  0.000%)
   IP6 Disc:            0 (  0.000%)
   TCP Disc:            0 (  0.000%)
   UDP Disc:            0 (  0.000%)
  ICMP Disc:            0 (  0.000%)
All Discard:            0 (  0.000%)
      Other:            0 (  0.000%)
Bad Chk Sum:            0 (  0.000%)
    Bad TTL:            0 (  0.000%)
     S5 G 1:            3 (  0.090%)
     S5 G 2:            4 (  0.120%)
      Total:         3333
===============================================================================
Action Stats:
     Alerts:            0 (  0.000%)
     Logged:            0 (  0.000%)
     Passed:            0 (  0.000%)
Limits:
      Match:            0
      Queue:            0
        Log:            0
      Event:            0
      Alert:            0
Verdicts:
      Allow:         3326 (100.000%)
      Block:            0 (  0.000%)
    Replace:            0 (  0.000%)
  Whitelist:            0 (  0.000%)
  Blacklist:            0 (  0.000%)
     Ignore:            0 (  0.000%)
===============================================================================
Frag3 statistics:
        Total Fragments: 0
      Frags Reassembled: 0
               Discards: 0
          Memory Faults: 0
               Timeouts: 0
               Overlaps: 0
              Anomalies: 0
                 Alerts: 0
                  Drops: 0
     FragTrackers Added: 0
    FragTrackers Dumped: 0
FragTrackers Auto Freed: 0
    Frag Nodes Inserted: 0
     Frag Nodes Deleted: 0
===============================================================================
===============================================================================
Stream statistics:
            Total sessions: 24
              TCP sessions: 14
              UDP sessions: 10
             ICMP sessions: 0
               IP sessions: 0
                TCP Prunes: 0
                UDP Prunes: 0
               ICMP Prunes: 0
                 IP Prunes: 0
TCP StreamTrackers Created: 14
TCP StreamTrackers Deleted: 14
              TCP Timeouts: 0
              TCP Overlaps: 0
       TCP Segments Queued: 2394
     TCP Segments Released: 2394
       TCP Rebuilt Packets: 793
         TCP Segments Used: 2393
              TCP Discards: 0
                  TCP Gaps: 0
      UDP Sessions Created: 10
      UDP Sessions Deleted: 10
              UDP Timeouts: 0
              UDP Discards: 0
                    Events: 1
           Internal Events: 0
           TCP Port Filter
                  Filtered: 0
                 Inspected: 0
                   Tracked: 3286
           UDP Port Filter
                  Filtered: 0
                 Inspected: 0
                   Tracked: 10
===============================================================================
HTTP Inspect - encodings (Note: stream-reassembled packets included):
    POST methods:                         0
    GET methods:                          10
    HTTP Request Headers extracted:       10
    HTTP Request Cookies extracted:       0
    Post parameters extracted:            0
    HTTP response Headers extracted:      10
    HTTP Response Cookies extracted:      0
    Unicode:                              0
    Double unicode:                       0
    Non-ASCII representable:              0
    Directory traversals:                 0
    Extra slashes ("//"):                 0
    Self-referencing paths ("./"):        0
    HTTP Response Gzip packets extracted: 0
    Gzip Compressed Data Processed:       n/a
    Gzip Decompressed Data Processed:     n/a
    Total packets processed:              2433
===============================================================================
SMTP Preprocessor Statistics
  Total sessions                                    : 0
  Max concurrent sessions                           : 0
===============================================================================
dcerpc2 Preprocessor Statistics
  Total sessions: 0
===============================================================================
===============================================================================
SIP Preprocessor Statistics
  Total sessions: 0
===============================================================================
File Preprocessor Statistics
  Total file type callbacks:            7
  Total file signature callbacks:       7
  Total files would saved to disk:      7
  Total files saved to disk:            7
  Total file data saved to disk:        47466737  bytes
  Total files duplicated:               0
  Total files reserving failed:         0
  Total file capture min:               0
  Total file capture max:               0
  Total file capture memcap:            0
  Total files reading failed:           0
  Total file agent memcap failures:     0
  Total files sent:                     0
  Total file data sent:                 0
  Total file transfer failures:         0
===============================================================================
File type stats:
         Type              Download   (Bytes)      Upload     (Bytes)
          GZ( 33)          2          6848054      0          0
         MP3( 64)          2          37257592     0          0
        JPEG( 70)          2          3360645      0          0
         BMP(148)          1          446          0          0
            Total          7          47466737     0          0

File signature stats:
         Type              Download   Upload
          GZ( 33)          2          0
         MP3( 64)          2          0
         PNG( 69)          1          0
        JPEG( 70)          2          0
            Total          7          0

File type verdicts:
        UNKNOWN:           7
            LOG:           0
           STOP:           0
          BLOCK:           0
         REJECT:           0
        PENDING:           0
   STOP CAPTURE:           0
          Total:           7

File signature verdicts:
        UNKNOWN:           7
            LOG:           0
           STOP:           0
          BLOCK:           0
         REJECT:           0
        PENDING:           0
   STOP CAPTURE:           0
          Total:           7

Total files processed:             10
Total files data processed:        47473897  bytes
Total files buffered:              7
Total files released:              7
Total files freed:                 0
Total files captured:              7
Total files within one packet:     1
Total buffers allocated:           1452
Total buffers freed:               0
Total buffers released:            1452
Maximum file buffers used:         759
Total buffers free errors:         0
Total buffers release errors:      0
Total memcap failures:             0
Total memcap failures at reserve:  0
Total reserve failures:            0
Total file capture size min:       0
Total file capture size max:       0
Total capture max before reserve:  0
Total file signature max:          0
Maximum buffers can allocate:      3196
Number of buffers in use:          0
Number of buffers in free list:    1744
Number of buffers in release list: 1452
===============================================================================
Snort exiting

This time the captured files have changed:

# ls -lS
-rw------- 1 root root 24211979 May  8 21:20
8452B621DC334D1FD44470A80540CBEF2F6869AF851B9E8C684EF9402016F692
-rw------- 1 root root 13045613 May  8 21:20
5CF142947C2957EE648457A91B69FB82F088F31205030F9A77B2AD827228C6E9
-rw------- 1 root root  6352738 May  8 21:20
DB57C532919D9ABABAC127F29DBDC05ED832394880E46CAD81A5DDE713CCB4BE
-rw------- 1 root root  2936119 May  8 21:20
B4127F43A3F455523B81179CC11AA4F28FC27F4C041D20E28AA08A32D85CB757
-rw------- 1 root root   495316 May  8 21:20
A294AA3D01CD8902BF842D320E7F2C043AF9EAD95D0E7198C3B71A0DBC9D253C
-rw------- 1 root root   424526 May  8 21:20
8863DB1EC4B02D5BCC1FB4BD03D220F7458136342CDD47CE507A5B886C6BB56C
-rw------- 1 root root      446 May  8 21:20 8D490C71A27631CF6A476F68C40965
5CB63BF32C17846A3C3C125A79046DB2C1

But they are still different from the original ones:

# ls -lS
-rw-r--r-- 1 root root 1044381696 Feb 18 20:12
ubuntu-14.04.2-desktop-amd64.iso
-rw-r--r-- 1 root root  375187792 May  8 21:07 VMware-viclient.exe
-rw-r--r-- 1 root root  101688487 Jul 10  2014 oversize_pdf_test_0.pdf
-rw-r--r-- 1 root root   14955972 May  8 21:07 MakeUp.mov
-rw-r--r-- 1 root root    6094376 May  8 21:07 video1.avi
-rw-r--r-- 1 root root    2187725 May  8 21:07 Fighter.mpg
-rw-r--r-- 1 root root        446 Mar 22  2013 tux-sw.bmp

##########################################################################
##########################################################################

Exit stats when listening from interface and type_id enabled:

===============================================================================
Run time for packet processing was 108.388974 seconds
Snort processed 256250 packets.
Snort ran for 0 days 0 hours 1 minutes 48 seconds
   Pkts/min:       256250
   Pkts/sec:         2372
===============================================================================
Memory usage summary:
  Total non-mmapped bytes (arena):       10100736
  Bytes in mapped regions (hblkhd):      122081280
  Total allocated space (uordblks):      8073952
  Total free space (fordblks):           2026784
  Topmost releasable block (keepcost):   108544
===============================================================================
Packet I/O Totals:
   Received:       256250
   Analyzed:       256250 (100.000%)
    Dropped:            0 (  0.000%)
   Filtered:            0 (  0.000%)
Outstanding:            0 (  0.000%)
   Injected:            0
===============================================================================
Breakdown by protocol (includes rebuilt packets):
        Eth:       256255 (100.000%)
       VLAN:            0 (  0.000%)
        IP4:       256130 ( 99.951%)
       Frag:            0 (  0.000%)
       ICMP:            0 (  0.000%)
        UDP:           24 (  0.009%)
        TCP:       132229 ( 51.601%)
        IP6:            0 (  0.000%)
    IP6 Ext:            0 (  0.000%)
   IP6 Opts:            0 (  0.000%)
      Frag6:            0 (  0.000%)
      ICMP6:            0 (  0.000%)
       UDP6:            0 (  0.000%)
       TCP6:            0 (  0.000%)
     Teredo:            0 (  0.000%)
    ICMP-IP:            0 (  0.000%)
    IP4/IP4:            0 (  0.000%)
    IP4/IP6:            0 (  0.000%)
    IP6/IP4:            0 (  0.000%)
    IP6/IP6:            0 (  0.000%)
        GRE:            0 (  0.000%)
    GRE Eth:            0 (  0.000%)
   GRE VLAN:            0 (  0.000%)
    GRE IP4:            0 (  0.000%)
    GRE IP6:            0 (  0.000%)
GRE IP6 Ext:            0 (  0.000%)
   GRE PPTP:            0 (  0.000%)
    GRE ARP:            0 (  0.000%)
    GRE IPX:            0 (  0.000%)
   GRE Loop:            0 (  0.000%)
       MPLS:            0 (  0.000%)
        ARP:          125 (  0.049%)
        IPX:            0 (  0.000%)
   Eth Loop:            0 (  0.000%)
   Eth Disc:            0 (  0.000%)
   IP4 Disc:       123866 ( 48.337%)
   IP6 Disc:            0 (  0.000%)
   TCP Disc:            0 (  0.000%)
   UDP Disc:            0 (  0.000%)
  ICMP Disc:            0 (  0.000%)
All Discard:       123866 ( 48.337%)
      Other:           11 (  0.004%)
Bad Chk Sum:          362 (  0.141%)
    Bad TTL:            0 (  0.000%)
     S5 G 1:            2 (  0.001%)
     S5 G 2:            3 (  0.001%)
      Total:       256255
===============================================================================
Action Stats:
     Alerts:            0 (  0.000%)
     Logged:            0 (  0.000%)
     Passed:            0 (  0.000%)
Limits:
      Match:            0
      Queue:            0
        Log:            0
      Event:            0
      Alert:            0
Verdicts:
      Allow:       228770 ( 89.276%)
      Block:            0 (  0.000%)
    Replace:            0 (  0.000%)
  Whitelist:        27480 ( 10.724%)
  Blacklist:            0 (  0.000%)
     Ignore:            0 (  0.000%)
===============================================================================
Frag3 statistics:
        Total Fragments: 0
      Frags Reassembled: 0
               Discards: 0
          Memory Faults: 0
               Timeouts: 0
               Overlaps: 0
              Anomalies: 0
                 Alerts: 0
                  Drops: 0
     FragTrackers Added: 0
    FragTrackers Dumped: 0
FragTrackers Auto Freed: 0
    Frag Nodes Inserted: 0
     Frag Nodes Deleted: 0
===============================================================================
===============================================================================
Stream statistics:
            Total sessions: 20
              TCP sessions: 14
              UDP sessions: 6
             ICMP sessions: 0
               IP sessions: 0
                TCP Prunes: 0
                UDP Prunes: 0
               ICMP Prunes: 0
                 IP Prunes: 0
TCP StreamTrackers Created: 14
TCP StreamTrackers Deleted: 14
              TCP Timeouts: 0
              TCP Overlaps: 0
       TCP Segments Queued: 6930
     TCP Segments Released: 6930
       TCP Rebuilt Packets: 6331
         TCP Segments Used: 6903
              TCP Discards: 7
                  TCP Gaps: 6570
      UDP Sessions Created: 6
      UDP Sessions Deleted: 6
              UDP Timeouts: 0
              UDP Discards: 0
                    Events: 16
           Internal Events: 0
           TCP Port Filter
                  Filtered: 0
                 Inspected: 0
                   Tracked: 131874
           UDP Port Filter
                  Filtered: 0
                 Inspected: 0
                   Tracked: 6
===============================================================================
HTTP Inspect - encodings (Note: stream-reassembled packets included):
    POST methods:                         0
    GET methods:                          0
    HTTP Request Headers extracted:       0
    HTTP Request Cookies extracted:       0
    Post parameters extracted:            0
    HTTP response Headers extracted:      2
    HTTP Response Cookies extracted:      0
    Unicode:                              0
    Double unicode:                       0
    Non-ASCII representable:              0
    Directory traversals:                 0
    Extra slashes ("//"):                 0
    Self-referencing paths ("./"):        0
    HTTP Response Gzip packets extracted: 0
    Gzip Compressed Data Processed:       n/a
    Gzip Decompressed Data Processed:     n/a
    Total packets processed:              13165
===============================================================================
SMTP Preprocessor Statistics
  Total sessions                                    : 0
  Max concurrent sessions                           : 0
===============================================================================
dcerpc2 Preprocessor Statistics
  Total sessions: 0
===============================================================================
SSL Preprocessor:
   SSL packets decoded: 68
          Client Hello: 0
          Server Hello: 2
           Certificate: 2
           Server Done: 3
   Client Key Exchange: 0
   Server Key Exchange: 0
         Change Cipher: 2
              Finished: 0
    Client Application: 0
    Server Application: 1
                 Alert: 0
  Unrecognized records: 64
  Completed handshakes: 0
        Bad handshakes: 0
      Sessions ignored: 1
    Detection disabled: 1
===============================================================================
SIP Preprocessor Statistics
  Total sessions: 0
===============================================================================
File Preprocessor Statistics
  Total file type callbacks:            2
  Total file signature callbacks:       1
  Total files would saved to disk:      1
  Total files saved to disk:            1
  Total file data saved to disk:        446       bytes
  Total files duplicated:               0
  Total files reserving failed:         0
  Total file capture min:               0
  Total file capture max:               0
  Total file capture memcap:            0
  Total files reading failed:           0
  Total file agent memcap failures:     0
  Total files sent:                     0
  Total file data sent:                 0
  Total file transfer failures:         0
===============================================================================
File type stats:
         Type              Download   (Bytes)      Upload     (Bytes)
         BMP(148)          1          446          0          0
         PDF(288)          1          3057259      0          0
            Total          2          3057705      0          0

File signature stats:
         Type              Download   Upload
         BMP(148)          1          0
            Total          1          0

File type verdicts:
        UNKNOWN:           2
            LOG:           0
           STOP:           0
          BLOCK:           0
         REJECT:           0
        PENDING:           0
   STOP CAPTURE:           0
          Total:           2

File signature verdicts:
        UNKNOWN:           1
            LOG:           0
           STOP:           0
          BLOCK:           0
         REJECT:           0
        PENDING:           0
   STOP CAPTURE:           0
          Total:           1

Total files processed:             2
Total files data processed:        3057705   bytes
Total files buffered:              2
Total files released:              1
Total files freed:                 1
Total files captured:              1
Total files within one packet:     1
Total buffers allocated:           95
Total buffers freed:               94
Total buffers released:            1
Maximum file buffers used:         94
Total buffers free errors:         0
Total buffers release errors:      0
Total memcap failures:             0
Total memcap failures at reserve:  0
Total reserve failures:            0
Total file capture size min:       0
Total file capture size max:       0
Total capture max before reserve:  0
Total file signature max:          0
Maximum buffers can allocate:      3196
Number of buffers in use:          0
Number of buffers in free list:    3195
Number of buffers in release list: 1
===============================================================================
Snort exiting

This time the captured files haven't changed:

# ls -lS
-rw------- 1 root root 446 May  8 21:33
8D490C71A27631CF6A476F68C409655CB63BF32C17846A3C3C125A79046DB2C1

Best Regards,


Pablo Cantos
redborder.org / pcantos at ...16842...

2015-05-08 21:49 GMT+02:00 Hui Cao (huica) <huica at ...589...>:

>  Hi Pablo,
>
>  When listening from interfaces, you have lots of discards. Because file
> processing relies on data that are reassembled correctly, it won’t be
> called for those sessions that miss file data.
>
>  In the case of PCAP, no sure why file type is not identified. It is
> interesting to see 47M file data for only 3326 packets. That is 24K per
> packet. I guess in this case, it will always hit PAF_MAX for each packet
> which might set each packet as single PDU(file). Can you try this setting?
>
>  config paf_max: 60000
>
>  Best,
> Hui.
>
>   From: Pablo Cantos Polaino <pcantos at ...16842...>
> Date: Friday, May 8, 2015 at 3:29 PM
> To: Hui Cao <huica at ...589...>
> Cc: "snort-users at lists.sourceforge.net" <snort-users at lists.sourceforge.net
> >
> Subject: Re: [Snort-users] File preprocessor fails to capture files
>
>     IP4 Disc:       122145 ( 49.331%)
>     IP6 Disc:            0 (  0.000%)
>     TCP Disc:            0 (  0.000%)
>     UDP Disc:            0 (  0.000%)
>    ICMP Disc:            0 (  0.000%)
>  All Discard:       122145 ( 49.331%)
>
>           TCP Segments Used: 6919
>               TCP Discards: 48
>                   TCP Gaps: 6459
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-users/attachments/20150508/146a0715/attachment.html>


More information about the Snort-users mailing list