online log analysis

hotspoons · **Joined:** Mon Jun 18, 2007 7:25 pm **Posts:** 89

qoncept wrote:

My host doesn't support java servlets, just perl and php, so if we do it here it'll have to be written in one of those.

Oh...that...I was talking about including a small Java http/servlet server (there are several that are under 100K) with RomRaider and a wrapper for SQLite, and making a local web server that will have the same functionality as the script I am creating, without user authentication and multi-users, so people could point their web browser at http://localhost:9999 or whatever and analyze data on their computer locally. It shouldn't take much above/beyond what I am doing here, besides translating oo PHP to Java and adding a switch to RomRaider to start the local web server. It could be a quick/easy way to include data analysis with RomRaider, until a proper GUI could be built...that is if that was ever in the plans.

qoncept · **Posted:** Mon Aug 06, 2007 7:58 am

RomRaider.org is running php 4.4.7. I tried upgrading to 5 but it broke pmwiki. I can set up another host with 5.2.2 though, its no big deal. We could call it something like logs.RomRaider.org or something.

hotspoons · **Joined:** Mon Jun 18, 2007 7:25 pm **Posts:** 89

qoncept wrote:

RomRaider.org is running php 4.4.7. I tried upgrading to 5 but it broke pmwiki. I can set up another host with 5.2.2 though, its no big deal. We could call it something like logs.RomRaider.org or something.

Right on. That will make it a bit easier for me as I don't have to rethink my coding for an older version of PHP. Once I get the front end built and I know the application is working, I'll throw it up on a test server in my house and have a couple of people test it to make sure it won't break your server, then I'll integrate it with PHPBB user control (could you send me a script or code chunk from PMWiki that has the integration with PHPBB so I can rip it off?) and we can put it live. Thanks!

-Rich

hotspoons · **Joined:** Mon Jun 18, 2007 7:25 pm **Posts:** 89

Just an update...I have the CRUD (create, read, update, delete for non-nerds) functions done for users and logs, and here is my first test reading a log from an uploaded file and parsing it to a table:

and the output:

Doing SQL insert and select statements should be cake, but I wanted to make sure I had the data right first.

I still have to do CRUD functions for car definitions (though I may just finish the rest of the basic graphing app first as I'm not going to attempt a dyno graph until I finish and polish the basics, and that would be what the car definitions would be for), build an interface, and make it graph (now that I have the data going in and out, it should be pretty easy).

And this is how to parse a CSV file:

Code:

   public function UploadCSVData(){
      $this->file = $this->path.basename($_FILES['CSVlog']['name']);
      if (!move_uploaded_file($_FILES['CSVlog']['tmp_name'], $this->file)){
         $this->proceed = FALSE;
         $this->returnStr = 'Uploaded file is invalid';
      }
      else{
         $this->flines = file($this->file);
         if (!$this->flines){
            $this->proceed = FALSE;
            $this->returnStr = 'Unable to open file. Please try again.';
         }
         else{
            $this->fileLen = count($this->flines);
            $this->fields = explode(',', trim($this->flines[0]));
            for ($i=1;$i<=$this->fileLen;$i++){
               $this->values[$i-1] = (explode(',', trim($this->flines[$i])));
            }
         }

      }
   }

And how to turn it into a table:

Code:

public function OutputCSVDataFromFile(){
      echo "<table><tr>";
      for ($i=0;$i<=count($this->fields);$i++){
         echo "<td>".$this->fields[$i]."</td>";
      }
      echo "</tr>\n";
      for ($i=0;$i<=count($this->values);$i++){
         echo "<tr>";
         for ($j=0;$j<=count($this->values[$i]);$j++){            
            echo "<td>".$this->values[$i][$j]."</td>";
         }
         echo "</tr>";
      }
      echo "</table>";
}

Freon · **Posted:** Tue Aug 07, 2007 9:08 am

Very cool stuff. Might I suggest a link feature that lets you post a link (say here on the forum) with a data range.

I.e.
"Hey look at this spot in my log WTF! http://enguinity.org/log.cgi?file=myfil ... ndrange=50"
or whatever.

qoncept · **Posted:** Tue Aug 07, 2007 9:23 am

Steve posted this in the pmwiki site about integration with phpbb, which will probably describe it was well as anything:

http://www.pmwiki.org/wiki/Cookbook/AuthPhpBB2
http://www.pmwiki.org/wiki/Cookbook/PhpBB2Integration

hotspoons · **Joined:** Mon Jun 18, 2007 7:25 pm **Posts:** 89

Freon wrote:

Very cool stuff. Might I suggest a link feature that lets you post a link (say here on the forum) with a data range.

I.e.
"Hey look at this spot in my log WTF! http://enguinity.org/log.cgi?file=myfil ... ndrange=50"
or whatever.

Fear not, you will be able to select data by time range initially, then after I get some free time, I'll write some queries to search logs for specific events based on fields, i.e. throttle position >80% for more than 1 second, boost pressure >5PSI for more than 1 second, knock events, etc., and include the ability to upload a long log (perhaps something 120 seconds long with 4 WOT pulls in it), then run it through some filters to pull out the interesting or useful data and break it apart into several runs and dump the uninteresting data.

Links will appear like 'http://logs.RomRaider.org/showlog.php?action=graph&target=external&style=4&engine=1&uid=144&runid=3342&f=1&f=2&f=3&f=4&start=49995&end=50124'

With f's being selected fields, action being graph/table/etc., target being the type of page to be displayed (external link, so just show the graph, not any other data), style being graph style (I will have different color combinations to choose from), engine being the graph rendering engine (just SVG for right now. With a raster (i.e. png file output), you would be able to embed it with an img tag, and so long as the person viewing is logged into this site (this can be changed, but for now, we'll keep it to RomRaider.org members only), the url will show up as a picture. RunID is the name for the entire run, and start and end are the ID's for the start and end points in the raw data table.

qoncept wrote:

Steve posted this in the pmwiki site about integration with phpbb, which will probably describe it was well as anything:

http://www.pmwiki.org/wiki/Cookbook/AuthPhpBB2
http://www.pmwiki.org/wiki/Cookbook/PhpBB2Integration

Looking over that, I may have to downgrade to PHP4 so it can be on the same host/root folder as the rest of your site. No biggie though...just removing member protection should make it PHP4 compatible. I'll test it out integration once I get the rest of the application built. Thanks!

NSFW · **Joined:** Wed Nov 22, 2006 10:23 pm **Posts:** 397

Nice work!

qoncept · **Posted:** Wed Aug 08, 2007 9:11 am

hotspoons wrote:

Looking over that, I may have to downgrade to PHP4 so it can be on the same host/root folder as the rest of your site. No biggie though...just removing member protection should make it PHP4 compatible. I'll test it out integration once I get the rest of the application built. Thanks!

Ah, yeah.. When I suggested that I was thinking all you'd need was the database. No big deal though.

hotspoons · **Joined:** Mon Jun 18, 2007 7:25 pm **Posts:** 89

NSFW wrote:

Nice work!

Thanks! I didn't get a chance to work on this last night, but I should have some time tonight. Hopefully I'll get something showable by the end of the week.

qoncept wrote:

hotspoons wrote:

Looking over that, I may have to downgrade to PHP4 so it can be on the same host/root folder as the rest of your site. No biggie though...just removing member protection should make it PHP4 compatible. I'll test it out integration once I get the rest of the application built. Thanks!

Ah, yeah.. When I suggested that I was thinking all you'd need was the database. No big deal though.

Shouldn't be a big deal. I think I have PHP4 on one of my LAMP servers, so I can easily test it there.

hotspoons · **Joined:** Mon Jun 18, 2007 7:25 pm **Posts:** 89

I have been insanely busy recently so I haven't had much time to work on this, but I have it reading .csv files, writing them to the database (per user, per log), and reading them out in tabular form. I added color coding for each 10 percent, with the formula '((current_value - min_value)/(max-value - min_value) * 100)'. I swear it has been so long since I did any heady math that it took me 45 minutes to figure out something so simple. I suck. Oh, and so do the colors I used. I just picked them at random. Any suggestions on the colors are welcome (in #ff00cc hex style). Currently I have them across 10% ranges, but that is easily increased. Here's a shot:

I'm going to try to get the front end/user interface built this week/weekend, then I can experiment with the graph (you need to be able to see values and select what you want to graph before you make a graph), but actually, graphing will be the easiest part! I'll keep you updated and let you know when I make some progress. Once I get the application built and working, I will focus on making it work with PHPBB access control. Before then, I would like a few people to hammer some logs at this thing and let me know if it works, or if they run into any trouble...kind of a private beta kind of thing...probably a couple weeks away as I'm going on vacation in a couple of days. Anyhow, keep an eye out. Thanks!

NSFW · **Joined:** Wed Nov 22, 2006 10:23 pm **Posts:** 397

That's pretty sweet!

I have no idea about the coloring, graphics ain't my strong suit. I drive a black car for that reason.

hotspoons · **Joined:** Mon Jun 18, 2007 7:25 pm **Posts:** 89

Alright. I finally have some more time to work on this thing. I moved my development environment to php4. Getting the application to work in php4 involved the following:

-find and replace 'public $', 'private $', 'protected $' with 'var $'.
-find and replace 'public function', 'private function', 'protected function' with 'function'

Thats it. Pretty easy. It's not like this is going to be a large application where class protection makes sense anyhow. I'm working on the graphing functionality now, which is dependent on the user interface being created for selecting columns to graph, so I imagine I'll do both in parallel. Hopefully I'll have something to show soon, and I can get a few people to test for me.

BTW, it would be dead easy to modify this script to work locally without multi-user support or authentication, and it could be modified to automatically import csv files dumped from RomRaider every time the script is run. You would just have to install WAMP for Windows (or any apache/mysql/php installation for any OS), change one configuration file, run an install script, direct RomRaider to save log files to the upload directory of this script and you could have it on the road. That would of course depend if this thing proves useful or not. Stay tuned...

javanaut · **Joined:** Tue Jan 02, 2007 7:19 pm **Posts:** 4

I just read through this thread, and I have to say that it's an awesome idea. I'm relatively new to tuning/logging, but having a place to publicly archive my logs (in an automated fashion no less) would be awesome as well. I really like the automated tuning bit where out-of-bounds figures are flagged for review.

If RomRaider could automagically publish my logs when I pull into my driveway and a wifi connection is made, that would be even better. I could see a use for statistical analysis spread out over time. It would be neat to graph across multiple logs to see if you are generally doing better or worse than before. Additionally, you could compare your results with others having similar setup.

I'm a PHP developer myself. If you need any help, feel free to ask, though it doesn't look like you really need any.

Thanks!

Toolzzz · **Posted:** Sun Nov 04, 2007 8:32 pm

Great project guys...keep up the great work!

RomRaider

online log analysis

Who is online