Another webstats alternative

[B&T]

I tried to add this to the sticky thread with the same topic . . . but the thread is closed. I suppose they don't want any more alternatives?

Here is another alternative to the webstats problems that does not involve any more analysis scripts (webalizer, awstats, etc.).
I see 2 problems with those server-based script approaches.

1) The are hard to get working and keep working.
2) They do not save the raw data so when/if there are problems you are hosed if you want to go back more than 8 days (the 8 days of logs in your logs directory).

So here is another alternative.

You have cron kick off a script each night (at 1:00 am after the log processing has run) that copies the access_log.1 file to a file in another directory that is dated (I use 11_05.log for example). Then you have ALL the logs for history anaylsis.

Then, as a part of my normal daily site backup those files are downloaded to my PC.

Now you can use a PC based log analyzer and you have all your log files (by date) on your PC. You can analyze as much as you want.

BUT . . . this still depends on Powweb to create the access_log.1 file each night correctly. In analyzing my logs this is not always the case. But when they double up or skip (as they sometimes do) at least you have the raw files and can just delete one or fix it if you want to have accurate stats (one more advantage of keeping the raw files).

The PHP script I use to make new "date named" copies (compressed to save space) of the log files each night is as follows (simple . . . only 6 lines):

<?php
putenv('TZ=HST10HDT'); // move clock back 2 hours
$day = (date("m_d"));
$filename1="/www/u/user/logs/access_log.1";
$filename2="/www/u/user/$day.log";
copy ($filename1,$filename2);
system( "gzip $filename2" );
?>

Why did I move the clock back 2 hours? The log file contains yesterday's data. So I moved the clock back two hours so the date() function would give yesterday's date. That way the date in the file name to matches the log data.

And the PC program I use to analyze the logs is:

http://www.weblogexpert.com/

the free version does all I need, and more than webstats did. You can pay for one that does even more.

I hope this helps. Works great for me.

[RocketJeff]

An excellent solution B&T!

There are other advantages to doing this besides just being able to get current stats:

1) The server load is reduced - remember when the servers all slowed after Midnight (PST) when processing stats?

2) It's more secure - if you don't want people seeing your stats, you probably shouldn't have them anywhere on your website.

3) It gives you the choice of stats programs to use. You've found a good one, but there are plenty more out there with varying capabilities.

4) By dating/keeping the log files, you can always re-run the stats again if you decide to change stat programs (see #3)

[B&T]

As I posted above, I am using

http://www.weblogexpert.com/

Works well for me. They have a free version (which I am using) and a pay version with more options.

If anyone likes some other PC based analyzer better . . . please post it here. Let's come up with the best solution we can. Or at least more alternatives.

[Nino]

Originally posted by B&T
So maybe I can't add this alternative to the sticky alternatives thread in this forum . . . or make this thread sticky . . . but I can keep bumping it to the top, making it "almost sticky" :)

Why don't you ask PowWeb to make you a moderator and you won't have that problem? Or is there some arcane, obscure, and dangerous ritual you must go through before becoming a moderator on this forum? :)
(If it involves touching your toes, I say don't do it!!! :D )

[Bannerzap.com]

This whole webstats thing is so very confusing. Here are members, many of which are very good with computers, and the stats software just doesn't seem to work. I intalled it (Awstats & Webalizer) with my own Cron jobs and they both run just fine. Yet for many others they do not run at all. Could it possibly be different servers are not making the logs properly? I am on www08 and both run for me.

[Nino]

Originally posted by Bannerzap.com
This whole webstats thing is so very confusing. Here are members, many of which are very good with computers, and the stats software just doesn't seem to work. I intalled it (Awstats & Webalizer) with my own Cron jobs and they both run just fine. Yet for many others they do not run at all. Could it possibly be different servers are not making the logs properly? I am on www08 and both run for me.

Must be a metaphysical thing . . . .:confused:

[NMS]

Originally posted by Bannerzap.com
This whole webstats thing is so very confusing. Here are members, many of which are very good with computers, and the stats software just doesn't seem to work. I intalled it (Awstats & Webalizer) with my own Cron jobs and they both run just fine. Yet for many others they do not run at all. Could it possibly be different servers are not making the logs properly? I am on www08 and both run for me.

Setting up is not always straight forward since not everyone follows the steps to the letter.

[B&T]

nmsupplies -

Not very nice . . . you deleted one of my posts in this thread.

But my post is still quoted in Nimo's post.

Why did you do that? There was nothing wrong with my post.

[(jj)]

Originally posted by B&T
As I posted above, I am using

http://www.weblogexpert.com/

Works well for me. They have a free version (which I am using) and a pay version with more options.

If anyone likes some other PC based analyzer better . . . please post it here. Let's come up with the best solution we can. Or at least more alternatives.

At one point I was running AnaLog (http://www.analog.cx/) as a PC based webstats system. But since the logs are no longer zipped each week, it's too much of a hassle for a dialup user (like myself) to download the log files.

Analog is a good program, just not too easy to configure for PC use. At one time I'd posted on the forum the steps for installing and using it, but a search did not find that post.

[B&T]

Originally posted by (jj)
. . . since the logs are no longer zipped each week, it's too much of a hassle for a dialup user (like myself) to download the log files.

You could easily modify the script I posted to compress the copied file. The command is:

system( "gzip $filename" );

Edit >> just tested that . . . works great. I will go back and add another comment about it to the first post. And weblogexpert will read the compressed files. No fuss. Thanks for the idea. I should have had compression in there in the first place.

[(jj)]

B&T,

Since I'm so PHP challenged, what would it take to make a script to just zip all of the past log files into one zip file?

Could a variable be set to take the *log.1 - *log.2 - *log.3 - etc and zip them all into one file each week? Set so that it only zips the past logs and not the active one?

[B&T]

Compressing them together does not save anything. And you have to be careful about when you would do the "rotation" (unless I am missing what you mean) - when do you know you have 8 new files. Powweb has been know to skip. I bet that's why Powweb quit doing it.

It seem to me you would be better off compressing (and dating) yesterday's log each night (as with the script in this thread). Then all the daily logs are dated and compressed just sitting the the specified directory waiting for you to download when ever you want. Why would you NOT want to do it that way?

But to answer your question, the script could be easily modified to run weekly and read files 1-7, write a single dated weekly version and compress it. Right now it has one read, just loop through that seven times each time concatenating the data, then do the write and compression.

[svsamana]

For those people that wish to know who's online "live", I would recommend www.usersonline.net.

It uses sql and you can see "live" the path visitors take through your website. And (with some modification) it logs sub-domains as part of the whole picture.

In our case sub domains are NOT set up as a stand-alone website but an integrated part of our main website. What I mean is that caribbean.sailingschool.com has only one index.html file and nothing else and that page links back to other subdomains or just files in the htdocs folder. (BTW subdomains seem to have improved our rankings in Google and other search engines)

Usersonline is not free but it is pretty nifty.

Hope this helps some of you (and yes, I also have raw data (as backup) dating back to the day we started our website back in '96 with the exception of one week where Powweb changed their logfile rotation - this still hurts a bit)

[B&T]

For anyone who has used this thread . . . I just updated the first post with a simpler version of the script.

[NMS]

Originally posted by B&T
nmsupplies -

Not very nice . . . you deleted one of my posts in this thread.

But my post is still quoted in Nimo's post.

Why did you do that? There was nothing wrong with my post.

could not send u a pm (disabled) ... nothing wrong... wanted to keep the thread short .. there was no info in that particular thread. Long threads tend to scare users. :)

[B&T]

Originally posted by nmsupplies
... wanted to keep the thread short .. long threads tend to scare users.
Too late for that . . . but you could make this thread "sticky" so I won't have to add more posts to bump it to the top. Or you could open up the current "sticky" alternatives thread so other alternatives could be added to that one.

I think this alternative is the best for Powweb and us customers.

[NMS]

Check sticky

[frogmn78]

how do you implement the php script? php is a coding for webpages, so do you load that webpage each week, or is is set into a cron entry?

[B&T]

php is not coding for web pages. but php can send a web page. in this case it does not. it just makes the backup.

re-read the post and you will see it tells you to kick off the script with cron and tells you what time to do that.

[frogmn78]

http://www.frogmn78.net/crhistmas.php

sure it is, ok I understand now.. never heard it used that way.

Thanks

[frogmn78]

Is there someway to put the clock back? or does it reset all by itself?

[frogmn78]

Sorry for the repeted messages by myself...

I entered the script exactly as it is shown, but the date on the archive file is one day ahead of the date inside the logs, is that normal, and how would I fix the date on the archive?

Thanks

[B&T]

The date situation is explained clearly in the instructions in the first post. The timezone shift in the script should take care of that. Did you remove that line from the script?

Note that the log can also contain dates from the next day as the log is rolled usually after midnight. If you left the date shift line in the script . . . look further in the file and make sure you are not just looking at the last few entries that actually were postred the next day. You may have everything right . . . but you will always have some log entries from the next day in each file because Powweb does not roll at midnight on the dot.

This should not cause a problem with anaylsis, because as you read mutliple files in to the analyzer (say for this month) they will be fine. You could have an issue if you want all your data for one day only . . . then you may need to read in two log files . . . the one for that day and the one for the day before to catch these few transactions.

[frogmn78]

here is the cron.php

<?php

putenv('TZ=HST10HDT'); // move clock back 2 hours
$day = (date("m_d"));
$filename1="/www/f/frogmn78/logs/access_log.1";
$filename2="/www/f/frogmn78/$day.log";
copy ($filename1,$filename2);
system( "gzip $filename2" );


$emailaddress = "frogmn78@frogmn78.net";

# Build the date/time
$runtime = (date("d M Y H:i"));

# Create the body of the email message
$message .= "Cron has executed the log.\n\n";
$message .= "Time of the message: $runtime (server time zone)\n\n";

# Send the email message
mail($emailaddress, "Cron Message" , $message, "From: frogmn78.net <cron update>");

?>

and the crontab entry
45 * * * * /usr/local/bin/webalizer -c /www/f/frogmn78/etc/webalizer.conf -p >>/www/f/frogmn78/logs/cron.log 2>>/www/f/frogmn78/logs/cron.err
00 2 * * * /usr/local/bin/php /www/f/frogmn78/etc/cron.php >>/www/f/frogmn78/logs/cron.log 2>>/www/f/frogmn78/logs/cron.err


And this is the log
http://frogmn78.net/logs/12_13.log

Any further help/explanation would be appricated, or if i'm just being dumb let me know

Thanks

[B&T]

Maybe not dumb . . . but clearly not thinking :)

The script sets the clock back two hours to fix the date issue. The instructions tell you this and suggest you run the script about 1:00am. This gives Powweb an hour to roll the logs and gets you within the 2 hour window.

You are running the script at 2:00am . . . so turning the clock back two hours did nothing for the date issue. You waited until the date moved again.

You will either need to turn the clock back further in the script, or run the script around 1:00am like the instructions say.

[frogmn78]

ohhhh, now it clicks. how would I change the clock back an extra hour?

[B&T]

Pick a time zone three hours before Pacific and put that in the script instead. Not sure what that would be. What is one time zone West of Hawaii, and find that code. It would be easier just to change your crontab so the script runs at 1:00 or 1:30am, like the instructions say.

But now that you get the idea . . . handle it how ever you want.

[frogmn78]

need help again...

It was working but now it stopped and I get this error in my cron.log

Status: 404

Content-type: text/html

X-Powered-By: PHP/4.3.3



No input file specified.

does anyone know what can be causing that? everything is the exact same as in the example (i changed the document paths to my home dir)

[B&T]

Did something happen to your log files? Did you have an access_log.1 file when the script ran? I can only guess. I have never seen that message.

[frogmn78]

yes, had log.1 and all, nothing new happened, I missed the e-mail went into the cron.log and saw that, tried some stuff but it still came back with that error

[B&T]

What if you run the script manually (not using cron) so you can see any messages. Does it work then?

I am having no problems, except a week or so ago when the logs did not roll correctly.