Awstats LOG size issues, anyone?

[cinenganos]

Hi there. I've tried to awstats my site in auto-mode via cron job, and I have the following:

I wrote 2 cron lines (one for the perl command line, other for the php code), and since the .log for error and success seems to exist (though empty) I assume my cron is OK and works... the problem is this: even in manual mode, when clicking the "update now" link, I get a 500 error (ant of course is NOT a permission issue, since I can red the page that has the "update now")....

I've tried by splitting the daily log file by half, and with this it DOES work OK, but the problem is that obviously I do not want to do this on a daily basis...

My daily files (access_log.1) are around 12 MB each... when I limit to 5 or 6 MB, the update works ok in manual....

Obbviosuly, via cron, I get nothing, since the problem seems to be in the time the process is taking..

any clues? ideas?

Thanks a lot in advance...

[glampe]

I'm having problems with files 5-6 Megs not working.

[cinenganos]

It seems like the process is being killed due to "too many" resources consumed, i.e. time, and that would obviously depend on the server, the time the update is done, the number of websites hosted, that moment's real load in the server, ...

Actually, I've been able to run (in Sundays, for example) the manual update with 12 MB log file, but that's not always and is not usual...

I've managed to make a php script to copy the logs to my htdocs, so I can have extra backup in case I forget to update, but I still would like to have it done entirely in "auto" mode...

Anyone else has more clues?

[transgenik]

Yes, that looks like the same problem someone described here and that I'm having too (with 5-6 mb logs):
http://forum.powweb.com/showthread.php?s=&threadid=28896

This, plus the fact that some days are sometimes completely missing from my logs makes PowWeb's service very unsatisfying for me right now. My stats are totally unreliable :(

[NMS]

Originally posted by cinenganos
I wrote 2 cron lines (one for the perl command line, other for the php code), and since the .log for error and success seems to exist (though empty) I assume my cron is OK and works... the problem is this: even in manual mode, when clicking the "update now" link, I get a 500 error (ant of course is NOT a permission issue, since I can red the page that has the "update now")....

A 500 error is usually a problem with chmod. Have checked that all settings are correct?

[cinenganos]

Hi there.

Thanks, I'm aware of the CHMOD issues on all perl scripts, but I'm afraid is NOT a chmod problem, since I can see the page displayed, and it's only SOME TIMES (MOST of the times, actually) when I cannot do the manual update (with the link) and the process gets killed. If I split the daily log in 2 or 3 pieces, and do the manual update by handling my *.1 log files (split, rename, etc...), I can get it work (so, again is *not* a chmod)...

Problem is: LOG size and time required to process with command line (unix/linux cron) and/or http (php/manual)...

[NMS]

Then why don;t you try and run the cron more often (e.g. hourly) on the access_log

[cinenganos]

Thanks.

I've seen you use that technique. I'm also in Europe, where at least 50% of my visitors are from, and hence I'd like to know if that's working for you... the 00:30 hrs the log gets rotated is actually 9:30 here in Western Europe, so the stats would lose some info but not (hopefully) crucial... Is that working for you?

I might as well have two awstats: one for the manual upadates and one for the auto with the "live" log files...

Just some questions:

1) isn't messing with live logs some bit of a hazard?
2) have you compared the % of loss, I mean, the % of hits you miss in the stats when using the live when it's gettin' rotated?

Cheers, and thanks...

[NMS]

The maximum I can loose before rotation is 30-40 minutes which are not important for me.

I have been using this system since September 2002 (coz I never liked webstats) with no problem at all.

[cinenganos]

2 more questions if you may:

(1) Is your live log file 'large' enough to cause problems with manual update?

(2) Are you using cron with perl call or with php call to do the awstats?

Thanks.

[NMS]

1) no idea ... probably not

2) cron with php (bypassing htaccess) as per help.powweb.com/tutorials/