You are correct, unfortunately the perl module that does the extracting is slow when it comes to that specific tarball, has to do with reading the index that contains a metric ton of .txt files and then pulling them out... you might be better off just scripting a cron run of tar against that file....<div>
<br></div><div>JJC<br><br><div class="gmail_quote">On Wed, Sep 19, 2012 at 1:06 PM, Michael Steele <span dir="ltr"><<a href="mailto:michaels@...9077..." target="_blank">michaels@...9077...</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">That fixed it.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Not sure what the final solution is here, as its painfully slow processing.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">There isn’t any real processing of the </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">opensource.gz</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> file, other the extracting the signatures and moving them to the designated folder?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1f497d">Kindest regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1f497d">Michael...<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> JJC [mailto:<a href="mailto:cummingsj@...11827..." target="_blank">cummingsj@...11827...</a>] <br>
<b>Sent:</b> Wednesday, September 19, 2012 2:09 PM</span></p><div><div class="h5"><br><b>To:</b> Michael Steele<br><b>Cc:</b> <<a href="mailto:snort-users@lists.sourceforge.net" target="_blank">snort-users@...4626...ceforge.net</a>><br>
<b>Subject:</b> Re: [Snort-users] Updating Rules with PulledPork and no outside connection<u></u><u></u></div></div><p></p><div><div class="h5"><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p><p class="MsoNormal" style="margin-left:.5in">
Ok, so around line 1787, in the condition ($NoDownload && !$grabonly) there should be a chunk that reads:<u></u><u></u></p><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-left:.5in"><i>unless ( $rule_file =~ /snortrules-snapshot-\d{4}\.tar\.gz/ ) {</i><u></u><u></u></p></blockquote><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div>
<div><p class="MsoNormal" style="margin-left:.5in">you will want to change it to the following and see what happens:<u></u><u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"><p class="MsoNormal" style="margin-left:.5in"><i>unless ( $rule_file =~ /snortrules-snapshot-\d{4}\.tar\.gz/<br>
                    || $rule_file =~ /opensource\.gz/ )<br>{</i><u></u><u></u></p></blockquote><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p><div><p class="MsoNormal" style="margin-left:.5in">On Wed, Sep 19, 2012 at 8:30 AM, JJ Cummings <<a href="mailto:cummingsj@...11827..." target="_blank">cummingsj@...11827...</a>> wrote:<u></u><u></u></p>
<div><div><p class="MsoNormal" style="margin-left:.5in">Im gonna have a look shortly<br><br>Sent from the iRoad<u></u><u></u></p></div><div><div><div><p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>On Sep 19, 2012, at 6:50, "Michael Steele" <<a href="mailto:michaels@...9077..." target="_blank">michaels@...9077...</a>> wrote:<u></u><u></u></p></div><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div><div><p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’m no expert here at all, but is there a chance that there is NO code even built into PulledPork that deals with the  ‘opensource.gz’ file in ‘NoDownload’ routine?</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Also is the NoDownload routine processing the rules twice?</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><div><p class="MsoNormal" style="margin-left:.5in">
<span style="font-size:10.5pt;font-family:Consolas;color:#1f497d">Kindest regards,</span><u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:Consolas;color:#1f497d">Michael...</span><u></u><u></u></p>
</div><p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.0in"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Michael Steele [mailto:<a href="mailto:michaels@...9640...77..." target="_blank">michaels@...9077...</a>] <br>
<b>Sent:</b> Tuesday, September 18, 2012 6:30 PM<br><b>To:</b> 'JJC'<br><b>Cc:</b> <a href="mailto:snort-users@lists.sourceforge.net" target="_blank">snort-users@lists.sourceforge.net</a><br><b>Subject:</b> Re: [Snort-users] Updating Rules with PulledPork and no outside connection</span><u></u><u></u></p>
</div></div><p class="MsoNormal" style="margin-left:1.0in"> <u></u><u></u></p><p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Here is another run using –vvnT</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Looking at the log it seems to be processing the rules twice.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p><p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.5pt;font-family:Consolas;color:#1f497d">Kindest regards,</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.5pt;font-family:Consolas;color:#1f497d">Michael...</span><u></u><u></u></p><p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> JJC <a href="mailto:[mailto:cummingsj@...11827...]" target="_blank">[mailto:cummingsj@...11827...]</a> <br>
<b>Sent:</b> Tuesday, September 18, 2012 1:47 PM<br><b>To:</b> Michael Steele<br><b>Cc:</b> <a href="mailto:snort-users@lists.sourceforge.net" target="_blank">snort-users@lists.sourceforge.net</a><br><b>Subject:</b> Re: [Snort-users] Updating Rules with PulledPork and no outside connection</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in"> <u></u><u></u></p><p class="MsoNormal" style="margin-left:1.5in">Interesting, can you do a run with -vv and send the results?<u></u><u></u></p><div><p class="MsoNormal" style="margin-left:1.5in">
 <u></u><u></u></p></div><div><p class="MsoNormal" style="margin-bottom:12.0pt;margin-left:1.5in">JJC<u></u><u></u></p><div><p class="MsoNormal" style="margin-left:1.5in">On Tue, Sep 18, 2012 at 6:19 AM, Michael Steele <<a href="mailto:michaels@...9077..." target="_blank">michaels@...843.....9077...</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">Attached is a log of the run. Attached is my pulledpork.conf and I'm looking<br>for something that is causing PulledPork to not process the opensource.gz<br>file in offline mode.<br>
<br>It appears to be a problem with PulledPork only processing the<br>snortrules-snapshot-2931.tar.gz  in offline mode as PulledPork processes<br>both files (opensource.gz and snortrules-snapshot-2931.tar.gz ) if you are<br>
processing in online mode.<br><br>My run line includes switches: -nvT<br><br>Both files have been place in the: temp_path=<br><br>I'm assuming that PulledPork should process both files exactly as it does in<br>offline mode as it does in online mode, minus the file downloading, and as<br>
long as the two files reside in the designated temp folder.<br><br>I'm not sure about checksums in offline mode as  PulledPork seems to process<br>the snortrules-snapshot-2931.tar.gz every time its ran in offline mode,<br>
regardless of any checksum. I believe it does the same thing in in online<br>mode. The checksum only prevents PullePork  from downloading the file/s<br>again in online mode.<u></u><u></u></p><div><p class="MsoNormal" style="margin-bottom:12.0pt;margin-left:1.5in">
<br>Kindest regards,<br>Michael...<br><br>-----Original Message-----<br>From: JJ Cummings [mailto:<a href="mailto:cummingsj@...11827..." target="_blank">cummingsj@...11827...</a>]<br>Sent: Monday, September 17, 2012 12:48 PM<br>
To: Michael Steele<br>Cc: <<a href="mailto:snort-users@...3471...ge.net" target="_blank">snort-users@lists.sourceforge.net</a>><br>Subject: Re: [Snort-users] Updating Rules with PulledPork and no outside<br>connection<u></u><u></u></p>
</div><div><div><p class="MsoNormal" style="margin-bottom:12.0pt;margin-left:1.5in">Place the tarballs in the defined temp path that you have in your<br>pulledpork.conf.. You will want to tell pp not to download and not to<br>
validate checksums...<br><br>JJC<br><br>Sent from the iRoad<br><br>On Sep 17, 2012, at 7:02, "Michael Steele" <<a href="mailto:michaels@...9077..." target="_blank">michaels@...9077...</a>> wrote:<br><br>
> I've looked through the list archive and was unable to find any<br>> specifics on how to do this.<br>><br>> I need to run PulledPork on a closed network.<br>><br>> The run line I have is:<br>> 'perl d:\winids\pulledpork\<a href="http://pulledpork.pl" target="_blank">pulledpork.pl</a> -c<br>
> d:\winids\pulledpork\etc\pulledpork.conf -v -T -n'<br>><br>> I'm pretty sure the -n tells PulledPork to process locally?<br>><br>> There are two files that need to be used and I'm not sure what to do<br>
> with them?<br>> 1) snortrules-snapshot-2931.tar.gz<br>> 2) opensource.gz<br>><br>><br>> Do these lines need to be hashed out?<br>> rule_url=<a href="https://www.snort.org/reg-rules/%7Csnortrules-snapshot.tar.gz%7C" target="_blank">https://www.snort.org/reg-rules/|snortrules-snapshot.tar.gz|</a><<br>
> oinkco<br>> de><br>> rule_url=<a href="https://www.snort.org/reg-rules/%7Copensource.gz%7C" target="_blank">https://www.snort.org/reg-rules/|opensource.gz|</a><oinkcode><br>><br>><br>> Just to verify; using the -T in the run line means I don't have to<br>
> hash out the so_rules section below?<br>><br>> sorule_path=/usr/local/lib/snort_dynamicrules/<br>> snort_path=/usr/local/bin/snort<br>> config_path=/usr/local/etc/snort/snort.conf<br>> sostub_path=/usr/local/etc/snort/rules/so_rules.rules<br>
> distro=FreeBSD-8.1<br>><br>> Kindest regards,<br>> Michael...<br>><br>><br>><br>><br>><br>><br>> ----------------------------------------------------------------------<br>> --------<br>
> Live Security Virtual Conference<br>> Exclusive live event will cover all the ways today's security and<br>> threat landscape has changed and how IT managers can respond.<br>> Discussions will include endpoint security, mobile security and the<br>
> latest in malware threats.<br>> <a href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/" target="_blank">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a><br>> _______________________________________________<br>
> Snort-users mailing list<br>> <a href="mailto:Snort-users@...1844...ourceforge.net" target="_blank">Snort-users@lists.sourceforge.net</a><br>> Go to this URL to change user options or unsubscribe:<br>> <a href="https://lists.sourceforge.net/lists/listinfo/snort-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/snort-users</a><br>
> Snort-users list archive:<br>> <a href="http://www.geocrawler.com/redir-sf.php3?list=snort-users" target="_blank">http://www.geocrawler.com/redir-sf.php3?list=snort-users</a><br>><br>> Please visit <a href="http://blog.snort.org" target="_blank">http://blog.snort.org</a> to stay current on all the latest Snort<br>
news!<u></u><u></u></p></div></div><p class="MsoNormal" style="margin-left:1.5in">----------------------------------------------------------------------------<u></u><u></u></p><div><div><p class="MsoNormal" style="margin-bottom:12.0pt;margin-left:1.5in">
--<br>Live Security Virtual Conference<br>Exclusive live event will cover all the ways today's security and threat<br>landscape has changed and how IT managers can respond. Discussions will<br>include endpoint security, mobile security and the latest in malware<br>
threats. <a href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/" target="_blank">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a><br>_______________________________________________<br>Snort-users mailing list<br>
<a href="mailto:Snort-users@lists.sourceforge.net" target="_blank">Snort-users@lists.sourceforge.net</a><br>Go to this URL to change user options or unsubscribe:<br><a href="https://lists.sourceforge.net/lists/listinfo/snort-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/snort-users</a><br>
Snort-users list archive:<br><a href="http://www.geocrawler.com/redir-sf.php3?list=snort-users" target="_blank">http://www.geocrawler.com/redir-sf.php3?list=snort-users</a><br><br>Please visit <a href="http://blog.snort.org" target="_blank">http://blog.snort.org</a> to stay current on all the latest Snort<br>
news!<u></u><u></u></p></div></div></div><p class="MsoNormal" style="margin-left:1.5in"> <u></u><u></u></p></div></div></div></blockquote></div></div></div></div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div></div></div></div></div></blockquote></div><br></div>