[Snort-devel] Clarification upon stats

Todd Wease twease at ...402...
Wed Jul 31 11:43:30 EDT 2013


Hello Reinoud,

On Wed, Jul 31, 2013 at 4:38 AM, Reinoud Koornstra <sockstat at ...445...>wrote:

> Hi Todd,
>
> Thinking about this, there might be more adjustments needed.
> For each rule, meaning OTN, ticks are being maintained as well.
> Would these ticks need to be adjusted as well?
> I mean only while busy processing (can also mean only reading) an OTN,
> we'll get to an RTN by calling getRtnFromOtn.
>

ruleOTNEvalPerfStats is the cumulation of the all of the time spent
evaluating rules, particularly the rule options.  When a leaf node is
reached, it means there is a match on all of the rule options of an OTN and
then the RTN is evaluated.  detection_option_node_update_otn_stats()
updates the OTN times based on the time it took to evaluate rule options
associated with the OTN.  The fix we are going to go with is to not include
the RTN time so the ticks won't need to be adjusted:

Index: src/detection-plugins/detection_options.c
===================================================================
RCS file:
/usr/cvsroot/sfeng/ims/sfsnort/snort/src/detection-plugins/detection_options.c,v
retrieving revision 1.88
diff -p -u -r1.88 detection_options.c
--- src/detection-plugins/detection_options.c   3 Jul 2013 12:57:58
-0000       1.88
+++ src/detection-plugins/detection_options.c   30 Jul 2013 19:15:27 -0000
@@ -949,6 +949,7 @@ int detection_option_node_evaluate(detec
                     PatternMatchData *pmd = (PatternMatchData
*)eval_data->pmd;
                     int pattern_size = 0;
                     int check_ports = 1;
+                    int eval_rtn_result;
 #ifdef TARGET_BASED
                     unsigned int svc_idx;
 #endif
@@ -983,7 +984,12 @@ int detection_option_node_evaluate(detec
                         }
                     }
 #endif
-                    if (fpEvalRTN(getRuntimeRtnFromOtn(otn), eval_data->p,
check_ports))
+                    // Don't include RTN time
+                    NODE_PROFILE_TMPEND(node);
+                    eval_rtn_result = fpEvalRTN(getRuntimeRtnFromOtn(otn),
eval_data->p, check_ports);
+                    NODE_PROFILE_TMPSTART(node);
+
+                    if (eval_rtn_result)
                     {
                         if ( !otn->detection_filter ||
                              !detection_filter_test(


>
> I noticed you use this function multiple times in profiler.c and assuming
> a pointer to a RTN is returned.
> Are you sure that every time an valid pointer will be returned?
> I mean in the function detection_option_node_evaluate in case of a leaf
> node it contains an OTN.
> In which case you call getRuntimeRtnFromOtn(otn) which calls getRtnfromOtn
> and you check wheter the pointer isn't NULL.
>

There may not be an RTN if you are using multiple policies via "config
binding".  Looks like there is one call in profiler.c in
ResetRuleProfiling() that should check for a NULL RTN.  This function would
only be called when reading pcaps from the command line and using the
--pcap-reset option.  I'll bug this as well.


>
> Thinking about it.......
> Should ruleOTNEvalPerfStats be changed to ruleNODEEvalPerfStats ?
> Because only when the node is a leaf node it contains an OTN.
> Actually this is how i read it, because only then you cast the pointer to
> the option_data of the node to a pointer to the OTN.
>

I think the name is fine - it's the time spent evaluating rules,
specifically the rule options.  Even if a leaf node isn't reached, meaning
a match on all of the rule options, there was still time spent evaluating
the OTN.


> Thanks,
>
> Reinoud.
>
>
>  ------------------------------
> From: sockstat at ...445...
> To: twease at ...402...
> Date: Wed, 31 Jul 2013 01:08:58 +0000
> CC: snort-devel at lists.sourceforge.net
>
> Subject: Re: [Snort-devel] Clarification upon stats
>
> Hi Todd,
>
> Thanks for your reply.
> May I propose a patch to fi the OTN stats?
>
> --- src/profiler.c.orig 2013-07-30 18:03:56.595012379 -0700
> +++ src/profiler.c      2013-07-30 18:05:35.779014374 -0700
> @@ -85,7 +85,7 @@
>
>
>  /* Externs
> ********************************************************************/
> -extern PreprocStats mpsePerfStats, rulePerfStats, ncrulePerfStats;
> +extern PreprocStats mpsePerfStats, rulePerfStats, ncrulePerfStats,
> ruleOTNEvalPerfStats, ruleRTNEvalPerfStats;
>
>
>  void getTicksPerMicrosec(void)
> @@ -926,6 +926,8 @@
>      mpsePerfStats.ticks -= rulePerfStats.ticks;
>      /* And adjust the rules to include the NC rules */
>      rulePerfStats.ticks += ncrulePerfStats.ticks;
> +   /* Also adjust the semi otn stats to not include the rtn stats */
> +    ruleOTNEvalPerfStats.ticks -= ruleRTNEvalPerfStats.ticks;
>
>      for (layer=0;layer<=max_layers;layer++)
>      {
>
>
> Kind Regards,
>
> Reinoud.
>
> ------------------------------
> Date: Tue, 30 Jul 2013 09:30:33 -0400
> Subject: Re: [Snort-devel] Clarification upon stats
> From: twease at ...402...
> To: sockstat at ...445...
> CC: snort-devel at lists.sourceforge.net
>
>
>  Hello Reinoud,
>
> On Wed, Jul 24, 2013 at 6:01 AM, sockstat <sockstat at ...445...> wrote:
>
>  Hi Everyone,
>
> I wanted to ask about the performance counters that snort maintains. In
> the function detection_option_tree_evaluate you start maintaining
> ruleOTNEvalPerfStats using a macro. But this actually kicks in for any type
> of node while only the leaf nodes seem to be an OTN. It also adds up when a
> node type is Flow or pcre etc so maybe it should better be called
> ruleNODEEvalPerfStats? In detection_option_tree_evaluate you call
> detection_option_node_evaluate which in case of a leaf node gets the RTN
> from the OTN and fpEvalRTN is called where you maintain
> ruleRTNEvalPerfStats. So in the end screen where you print the stats out it
> seems to me that the microseconds for rule tree eval contain the rtn eval
> microseconds, because fpEvalRTN is called while counting the clock ticks
> for ruleOTNEvalPerfStats. In profile.c you print the amount of microseconds
> out. So rule tree eval is not just the ticks spent in
> detection_option_tree_evaluate, It has the clock ticks spent in fpEvalRTN
> included, while rtn eval is printed out sperately.
>
>
> Yes, it looks like the RTN stats are counted in the OTN stats.  I'll
> create a bug to see if we can get this fixed.
>
>
>
> Same thing holds for the mpse stats when __process_queue calls
> rule_tree_match, then the ticks spent in detection_oprion_tree_evaluate and
> fpEvalRTN are included in the ticks spent in mpseSearch. Hence it seems
> there is some overlap in the stats, because I don't see the ticks spent in
> fpEvalRTN subtracted from the amount of ticks spent in
> detection_option_tree_evaluate for example or did I miss this?
>
>
> In rule_tree_match(), rulePerfStats are started and in
> ShowPreprocProfiles() in profile.c it is subtracted from mpsePerfStats.
>
>
>
> Also in case of a core 2duo CPU the tsc isn't stable and hence the
> calculation of how many clock ticks occur per microsecond might not be
> accurate?
> Thanks, Reinoud.
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Snort-devel mailing list
> Snort-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/snort-devel
> Archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>
> Please visit http://blog.snort.org for the latest news about Snort!
>
>
>
> ------------------------------------------------------------------------------
> Get your SQL database under version control now! Version control is
> standard for application code, but databases havent caught up. So what
> steps can you take to put your SQL databases under version control? Why
> should you start doing it? Read more to find out.
> http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
>
> _______________________________________________ Snort-devel mailing list
> Snort-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/snort-devel Archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-develPlease visit
> http://blog.snort.org for the latest news about Snort!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.snort.org/pipermail/snort-devel/attachments/20130731/23ae8104/attachment.html>


More information about the Snort-devel mailing list