Webalizer custom web stats - how to

[njscuba]

see here:

http://www.njscuba.net/powwebstats

[Jess]

Nice write-up! Very informative. Thanks! :)

I still can't get the manual update to work, but the automatic update through the cronjob did work. Only problem with that, though, is that it wiped out all my stats for January up until the 26th. Previous months are still there. Odd. :o

[njscuba]

I updated the perl script to run Webalizer. It now no longer generates an error message when you run it.

-- rich

[omatic]

Your little tutorial on running webstats and doing crontabs answered a bunch of question I had (and haven't yet even asked anybody!). Thanks a lot! WELL DONE!

[Jess]

Originally posted by njscuba
I updated the perl script to run Webalizer.
-- rich

Nice, rich! The updated script worked like a charm. And my cron is still working fine and dandy, too. I even figured out how to set it to execute at two specific times every day. I'm a happy kid. Thank you so much! :p

[barrikady]

Thank you very much for your tutorial. I applied the changes, and I am very pleased with the new updated information.

[somersetbrad]

I realy dig the Doc you wrote.
One thing about losing the days.
I was able to keep the data from the other 26 days by simply
creating another dir in /htdocs, then changing the webilizer.conf file to post to that directory. You still lose the days in the new directory but they are also nice and neat for you in the old.
Hope that helps for other people going to try this :)

[njscuba]

I have updated the webstats tutorial with a command to run AWStats automatically without hitting the memory limit that is present on most powweb servers ( mine is venus. )

once again:

http://www.njscuba.net/powwebstats

Also added error logging instructions ( credit is due to neville )

-- r
http://www.njscuba.net
whaddaya think ?

[sailfast]

Thanks for great help. I have .htaccess on my awstats so I added the following to the url:
http://username:secretpassword\@www.yoursite.com/cgi-bin/awstats.pl?update=1
Maybe our servers will be upgraded someday to get past the 2mb barrier.

[Jess]

Originally posted by somersetbrad

One thing about losing the days.
I was able to keep the data from the other 26 days by simply
creating another dir in /htdocs, then changing the webilizer.conf file to post to that directory. You still lose the days in the new directory but they are also nice and neat for you in the old.


Could you explain how to do this in more explicit detail, please? I'm running Webalizer through my own cron, and it's been working well - but after every 7 days, I lose all the previous days. :(

[sandy]

Originally posted by sailfast
Thanks for great help. I have .htaccess on my awstats so I added the following to the url:
http://username:secretpassword\@www.yoursite.com/cgi-bin/awstats.pl?update=1
Maybe our servers will be upgraded someday to get past the 2mb barrier.

Sailfast .. I updated my crontab based on the tutorial an it works fine. This is from the tutorial. Follow it verbatim and it will work fine.. Just set you update time >15 minutes past the hour.. or else you can miss the zipfile create Sunday just after midnight (that happened to me last night) but it has now updated my stats and kept all previous stats):

Setting up and Configuring Webalizer

Make sure you switch off the Daily update from within the OPS.
Download the latest version of Webalizer from here.
Extract webalizer.conf (and edit any changes as you wish. Some tutorials are also available.)
Upload this edited file to a folder called etc situated in the same level of htdocs and cgi-bin. Make sure you chmod this file to 755. You can find some help here.
Open Notepad (or equivalent) and make sure that word wrap is switched off. Insert this cron as a single line:

0 0 * * * /usr/local/bin/webalizer -c /www/u/username/etc/webalizer.conf -p >>/www/u/username/logs/cron.log 2>>/www/u/username/logs/cron.err

Note: u = username initial (the first letter). Make sure to make 1 or 2 extra empty fresh lines underneath this code. To read more about cron you may want to visit the cron tutorial. The log and error part is important especially if you have some problems.


Save this file to crontab.txt
Upload this file to the same folder mentioned above - etc and rename file from crontab.txt to crontab.
Crontab updates every hour.
Customise this part '0 0 * * *' depends on when you wish to update your statistics. At the moment it is set to update daily at midnight. Enjoy!

[sailfast]

Originally posted by njscuba
I have updated the webstats tutorial with a command to run AWStats automatically without hitting the memory limit that is present on most powweb servers ( mine is venus. )

Originally posted by sandy
Sailfast .. I updated my crontab based on the tutorial an it works fine. This is from the tutorial. Follow it verbatim and it will work fine.. Just set you update time >15 minutes past the hour.. or else you can miss the zipfile create Sunday just after midnight (that happened to me last night) but it has now updated my stats and kept all previous stats):

Njscuba's great tutorial only works for me on neptune since I do get an out of memory error based on the preferred tutorial (4 day's of runs).
Thanks for the heads up on the log rotation.

[njscuba]

Are you sure about the 15 minute delay , or is that just a guess ?

My Webalizer just lost stats for the first week of February, which is suspiciously coincident with archiving the log files. AWStats is ok, although prior to this it had stopped working ( log file too big again )

Is it goint to be necessary to manage the log files manually to avoid this happening ? If so, how can you get POWWEB to stop doing it automatically ?

As far as AWStats is concerned, blowing away the log file after each run would seem to be perfectly acceptable, and easy to do. However, Webalizer would have a fit.

Just when everything seemed to be running smoothly.

-- rich

[sandy]

Originally posted by sailfast
Njscuba's great tutorial only works for me on neptune since I do get an out of memory error based on the preferred tutorial (4 day's of runs).
Thanks for the heads up on the log rotation.

NJ.. I used this format and mine updated fine, and, I have last week's stas as well.. I changed the timne when to run past 15 minutesd past the hour, and it did update.

23 *12 * * * /usr/local/bin/webalizer -c /www/u/username/etc/webalizer.conf -p >>/www/u/username/logs/cron.log 2>>/www/u/username/logs/cron.err

Good luck..

Sandy

[Jess]

Originally posted by sandy
I and, I have last week's stas as well.. I changed the timne when to run past 15 minutesd past the hour, and it did update.

Sandy

I have my Webalizer cron set to run at 4:10 a.m. PST and 4:10 p.m. PST. It's been updating just fine, except that when it updated on the 9th, I lost all the previous week's stats. :(

[sandy]

Originally posted by Jess
I have my Webalizer cron set to run at 4:10 a.m. PST and 4:10 p.m. PST. It's been updating just fine, except that when it updated on the 9th, I lost all the previous week's stats. :(

Jess,

What is in your crontab file? I am following the Powweb tutorial and ine updated and didn't lose last week's stats.

Sandy

[Jess]

Originally posted by sandy
Jess,

What is in your crontab file? I am following the Powweb tutorial and ine updated and didn't lose last week's stats.

Sandy

Sandy, I'm using the single line specified in Rich's tutorial that he posted in the beginning of this thread:

22 */12 * * * /usr/local/bin/webalizer -c /www/f/foobar/etc/webalizer.conf

(with it customized, of course, to my own runtime and server path.)

[sandy]

Originally posted by Jess
Sandy, I'm using the single line specified in Rich's tutorial that he posted in the beginning of this thread:

22 */12 * * * /usr/local/bin/webalizer -c /www/f/foobar/etc/webalizer.conf

(with it customized, of course, to my own runtime and server path.)

Change it by adding the -p at the end:

22 */12 * * * /usr/local/bin/webalizer -c /www/f/foobar/etc/webalizer.conf -p

As I found out the -c means use current.. the -p means look at the history file.

Mine updates 2X per day and isn't losing any history. Unfortunately you can't get back what you lost, but you won't lose anymore after this Sunday's roll-over.

Sandy

[Jess]

Originally posted by sandy
Change it by adding the -p at the end:

22 */12 * * * /usr/local/bin/webalizer -c /www/f/foobar/etc/webalizer.conf -p

As I found out the -c means use current.. the -p means look at the history file.

Sandy

Thanks, Sandy. Coincidentally, I just made that change. After I posted my previous message to you and noticed the -p in your line that was not in mine, I did a little more reading on command line options and found this

-p Incremental. Preserve internal data between runs.

http://www.panix.com/web/faq/logs/webalizer.html

So we'll see what happens this weekend. But I am feeling more confident now about not losing previous days. :)

[njscuba]

-p

oops, missed that ...

thanks

[G-Spot]

22 */12 * * * /usr/local/bin/webalizer -c /www/f/foobar/etc/webalizer.conf -p


Question1- Read above and: Whit what do I have to replace the stars in order to get the generation:

In every Minute, in every hour, in every day, and in every month

(so I can test it immediately, whether or not it is working, without vasting time!?)

Question2- Read above and: there is a switch "-p" what is the meaning of it, and are there other switches, as I've read someting here at the forum, but I couldn't find it again, as it was said, with another switch was working much better, what kind of swith was it?

Thank you in advance,
G-Spot

[sandy]

Originally posted by G-Spot
22 */12 * * * /usr/local/bin/webalizer -c /www/f/foobar/etc/webalizer.conf -p


Question1- Read above and: Whit what do I have to replace the stars in order to get the generation:

If you want to test this immediately, to give the system time to set-up, assume it's 10AM .. and you want this to run at 10:40 AM today... you set the crontab file as 40 10 * * * the first is the minutes, the 2nd is the hour .. you can keep changing this..

Question2- Read above and: there is a switch "-p" what is the meaning of it, and are there other switches, as I've read someting here at the forum, but I couldn't find it again, as it was said, with another switch was working much better, what kind of swith was it?

The -p says use the history file, which means that you don't lose the stats after the Sunday zip file is created, and your access_log is re-set. Without the -p, you get only the current weeks's stats.
G-Spot

Sandy

[sandy]

G-Spot,

This will run every hour at the precise minute:

22 * * * *

22 minutes after each hour.. For the Sunday roll-over, zip file create,,, set the time at least 15 miutes past the hour,,, that is why mine is set as 22 */12 * * * which runs at 22 minutes after midnight and noon.

Sandy

[G-Spot]

Thanks, Sandy for the quick response, I 've tried your advice, tested it etc...and now my webstats are working, eventually, including access file and Password...whooow...and NOT in default mode, but with at least 200 showings...
However, one word I'd loose, I wonder why is the whole concept so difficult and complex: conf file, crontab, passw file, acces file, etc, etc....it reminds me to the DOS operation system, which is comprehensible to the experts only....!

Cheers
G-Spot

[G-Spot]

Actually one more question:

How can I see the DATE and the TIME of the visitors/hits at the webstats?
Is it possible, to get the date and time, and if so, how?
Any help would be greatly appreciated,

G-Spot

[sandy]

Originally posted by G-Spot
Actually one more question:

How can I see the DATE and the TIME of the visitors/hits at the webstats?
Is it possible, to get the date and time, and if so, how?
Any help would be greatly appreciated,

G-Spot

I don't know if this is possible. You might want to look at the weablizer configuration file http://www.mrunix.net/webalizer/

and see if there is this set-up.

I use Addfreestats for all of that info, wherre each page is tracked:

http://www1.addfreestats.com/index.html

Sandy

[Dees-Nuts]

[QUOTE]Originally posted by njscuba
[B]I have updated the webstats tutorial with a command to run AWStats automatically without hitting the memory limit that is present on most powweb servers ( mine is venus. )

once again:

http://www.njscuba.net/miscellaneous/powwebstats.html

Also added error logging instructions ( credit is due to neville )


Thanks - worked like a charm.. Much appreciated. Anyway to get this to recognize php files? I have no idea where people are coming from an hitting them, would just like to have an idea which SE.

[njscuba]

Anyway to get this to recognize php files? I have no idea where people are coming from an hitting them, would just like to have an idea which SE.

--- directly from the webalizer sample config file:


# PageType lets you tell the Webalizer what types of URL's you

# consider a 'page'. Most people consider html and cgi documents

# as pages, while not images and audio files. If no types are

# specified, defaults will be used ('htm*', 'cgi' and HTMLExtension

# if different for web logs, 'txt' for ftp logs).



PageType htm*

PageType cgi

#PageType phtml

#PageType php3

#PageType pl

[Dees-Nuts]

I'll give that a shot, but should have been more specific..
I can get the .php - just anything after ? is chopped off, even after .cgi. I'll see if that'll make it pick up the variables and give a complete URL.

Thanks.

[njscuba]

To make it easier to get the necessary Webalizer and AWStats config files, I have added links to them. I don't think POWWeb is supplying these files to new users anymore.

Also, I moved everything to:

http://njscuba.net/powwebstats

-- r