Powweb server changes timestamps and thus breaks automatic site updates

[malex]

The problem:
Powweb server that hosts my files changes timestamps on some of them at random, which breaks my automated site updates. The problem is amplified by the fact that only FTP is available for site updates, so my software (sitecopy) has to rely on timestamps.

No recourse:
I am appalled by the reaction of the Powweb support staff in handling a support request about this problem. I opened a ticket and their only action was to ask if the file contents get modified. When I said that only timestamps were changing they just closed the ticket.

Grudge:
It looks to me that they are turning their backs on a valid customer problem that a few people have encountered in the last few years as exemplified by a number of posts in this forum. Of course, it is easier to shrug it off and let a customer manually ftp scores of files on each site update then to provide access to a more robust transfer protocol or to fix the timestamp modification problem on the server(s).

Alex

[stevel]

Well, I'll agree that the problem exists. Frequently, Dreamweaver will tell me that I need to update files which haven't changed in months if not years. Sometimes it wants to re-upload my whole site. For me, the annoyance level hasn't risen enough for me to complain, though.

My suggestion would be to open a new ticket and ask that it be escalated. The problem I see is that it's not deterministic - so it may be hard to track down.

[rbradscott]

Sounds like a perfectly valid complaint, malex. It's stories like this that start to make me 2nd guess my intentions to bring some of my clients over to PW.

So we know the problem exists; does anyone have a clue as to the cause, or more importantly, the cure?

[James]

Though I am not positive, Dreamweaver probalby does not take into account the fact that your files are accessed through multiple systems. The internal clocks on all servers are not perfect, and therefore thats why we sync them with an atomic clock nightly. Dreamweaver accesses your files through an ftp server that has one clock, over nfs to the fileserver with another. If these two get out of sync even for a second its possible that is whats causing your problem with Dream weaver. There is no setup or configuration we can do to make sure they remain in perfect sync at all times... this isn't a problem with our servers. If it is a problem its a limitation with Dreamweaver. As far as using a more robust protocol, what would you suggest? FTP is for transferring files... thats why its called file transport protocol. No matter what service you use you will probalby see the same problem since its not a FTP issue.

[extras]

I know that the server time can be a little off,
but time stamp of files are not really influenced by it.
I've never seen strange change of time stamp so far.

So, unless the original poster provides more info, screen shots,
whatever to prove the claim, I'm skeptical about time stamp change.

The problem is amplified by the fact that only FTP is available for site updatesThis isn't true.
I use upload script of my own, and there are people who uses FrontPage to uploads.


Although it's true that it may take long time for PowWeb to correct some problems,
I'm not convinced if "the timestamp modification problem on the server(s)" exists.

[rbradscott]

I'm not convinced if "the timestamp modification problem on the server(s)" exists.

Fair enough; lets see how it shakes out. You're right - we don't yet know for a fact the problem actually exists as described.

[stevel]

The next time this happens to me I'll stop and look at what the timestamps are. I'll agree with James that there is likely an assumption that the server the FTP client is talking to is the same server that sets and reports the timestamp, and I can imagine a scenario where if the two have a clock skew, this might cause a problem. A lot depends on exactly how the timestamp info gets read and then transmitted to the client. The issue did start appearing when PW moved to the NetApps.

[malex]

deleting because I submitted a double reply...

[malex]

I know that the server time can be a little off,
but time stamp of files are not really influenced by it.
I've never seen strange change of time stamp so far.


Lucky you! If the problem is not widespread it doesn't mean it should be ignored, does it?


So, unless the original poster provides more info, screen shots,
whatever to prove the claim, I'm skeptical about time stamp change.


Below is the excerpt from a ticket I submitted to Powweb support. If necessary for those who can assist in solving this problem as opposed to outright dismissing it I can provide screenshots.

>> Example "sitecopy" session:
>>
>> Uploading dists/breezy/main/source/Sources.gz: [] failed:
>> Remote file has been modified - not overwriting with local changes
>> Uploading dists/breezy/Contents-i386: [] failed:
>> Remote file has been modified - not overwriting with local changes
>> Uploading dists/breezy/Contents-i386.bz2: [] failed:
>> Remote file has been modified - not overwriting with local changes
>> Uploading dists/breezy/Release: [] failed:
>> Remote file has been modified - not overwriting with local changes
>> Uploading dists/breezy/Contents-i386.gz: [] failed:
>> Remote file has been modified - not overwriting with local changes
>> Uploading packages-i386.db: [] failed:
>> Remote file has been modified - not overwriting with local changes
>> sitecopy: Errors occurred while updating the remote site
>>
>> Here is the listing of the files on the server:
>>
>> -rw-r--r-- 23744 Jul 7 15:14 Contents-i386
>> -rw-r--r-- 2310 Jul 7 15:14 Contents-i386.bz2
>> -rw-r--r-- 2241 Jul 7 15:14 Contents-i386.gz
>> drwxr-xr-x Jul 7 15:12 main
>> -rw-r--r-- 2453 Jul 12 18:20 Release
>> drwxr-xr-x Jul 7 15:12 restricted
>>
>> As you can see only the "Release" file has a correct timestamp. The rest
>> of the files have their timestamps reverted to Jul 7 xx:xx.


This isn't true.
I use upload script of my own, and there are people who uses FrontPage to uploads.


And what transfer method does your own script use? Using FrontPage extension is out of question for me. As far as the Powweb documentation states FTP and FP extensions are the only available methods for publishing a site.


Although it's true that it may take long time for PowWeb to correct some problems,
I'm not convinced if "the timestamp modification problem on the server(s)" exists.

I don't have to convince you. So far you have not even said anything except "I don't belive you just because I don't".

[extras]

I didn't take your post so seriously because you were trying to show the validity of claim too hard.
You said there is "No recourse" while there could be some.
And "Grudge" section is simply baseless.
You didn't present your case with hard data or way to verify by other members.

In short, your 1st post wasn't very convincing to me because of several factors including above.

Having said that, I know that direcotry cache and/or attribute cache for the NFS file system
has been showing strange behavior, for sure.
But I've seen these only as a differences among http servers on the given cluster.

So, by using https instead of http, I think we can avoid problems related ot this issue
because we tend to get the same webserver for a while when https is used.
(With http, we get different server, randomly, each time.)

But this is applicable when you are using a script through HTTP and not FTP.
If you insist that FTP server is showing wrong time stamp, I'm not yet conviced.
As you don't even say what software is having the problem,
it's hard to determine and give further suggestions.


Other than that, don't push too hard in the efffort to get attention or assistance,
as it may have opposite effect.
I think showing hard technical evidence is the best way to convince other members and PowWeb.
(It doesn't necessarily mean that the problem will be reloved quickly, though ....)

[diwakergupta]

I agree that there can possibly be a problem with timestamps. As others have pointed out, keeping the _clock_ synced up has *nothing* to do with timestamps of files getting modified.

The easiest solution is to not use FTP, but something smarter like rsync, which can do a content comparison of files and not rely on timestamps at all. I mean, really, timestamps suck for measuring file modifications.

However, since PowWeb simply DOES NOT provide any other mechanism (shell access, rsync/unison support, see my rant elsewhere: http://forums.powweb.com/showthread.php?t=53891) users have no option but to fall back to FTP.

Some clients (lftp, krusader) have the ability to a content hash comparison even over FTP so you might be better off using them.

[malex]

Yeah, it's childish to show the emotions, but after manually reuploading the files for the n-th time it can get under the skin. As I showed in the example I am using "sitecopy" to keep my local tree and the website synchronized. A curios observation is that I don't usually get that problem when uploading html files or binary files such as images to the server. It seems to concentrate on the resource files for a Debian repository I maintain for a small group of people, such as Packages, Packages.gz, Packages.bz2, Sources, ... and so on. These are small text files with gzipped and bzip2 compressed copies.

[stevel]

It's usually .jpg files for me.

[extras]

I wrote a tiny python script to do similar job, a couple of years ago.
As it doesn't bother to check the time stamp of remote file,
it shouldn't fail even if PowWeb's FTP server is truely giving wrong time stamp, somehow.

It uses a file to remeber the time of last update, and any file newer than that will be uploaded.
It doesn't delete remote file. Only updating. :)
Sometime, very primitive script like this encounters less problem than sophisticated one.
And it's easy to debug/modify, too. ;)

If you use it, do "touch time.file", initially.
Otherwise everything under "localdir" will be uploaded.

#!/usr/local/bin/python
#
# Ftp update
#

import os
import ftplib

localdir = '/whatever' # Local root
ftpserver = 'ftp.somewhaere.com' # the name of remote ftp server
user = 'username'
password = 'password'
remote_dir = '' # remote_root, if any
timefile = 'time.file' # time stamp file. Any file newer than it will be uploaded

if 1:
print "Creating ftp.object", ftpserver
f = ftplib.FTP(ftpserver)
print "Log in prcess"
f.login(user, password)
print "chdir to document root"
if remote_dir: f.cwd(remote_dir)
#print f.dir('')

lasttime = os.stat(localdir+timefile)[8] # ST_MTIME
print "lasttime=",lasttime

done = []
def uploadr(topdir):
print "\n## uploadr ##", topdir
dirl = os.listdir(topdir)
for pn in dirl:
pnf = os.path.join(topdir,pn)
pst = os.stat(pnf)
if(pst[0] & 0170000) == 0040000: # MODE S_ISDIR
f.cwd(os.path.basename(pnf))
uploadr(pnf)
f.cwd('../')
elif pst[8] > lasttime:
print "\n===>",
f.put(pnf)
#print pnf,"==>", os.path.basename(pnf),
done.append(pnf)
else:
#print "Skipping", pnf
print "*",
#print pn+",",


uploadr(localdir)
print "\n==== Finished ==="
tf = open(localdir+timefile,'wb')
tf.write(str(lasttime)+"\n")
tf.close()
print done