Poof, all my video content disappeared yesterday.

[Daemon]

(Typing this up while on hold to support, so it's a long story.)

So last night I go to refer someone to one of the videos on my site, and
discover that they're *all* gone. Virtually every .AVI, .WMV, .MPG, .MPEG, .MOV.
file I've ever created was deleted. All html and php files remain unmollested,
all image files are still there, but several gigabyte of original content
videos (videos of auto racing, R/C hobby related and misc stuff, all copyright *me* )
all gone.

If my site was hacked, it's certainly not your typical hack, where
they mess with a few things, and replace the home page. No this was
a very specific deletion of all the largest files on the site. To do this
through the ops panel File Manager would have taken forever, because there's no
way to filter on only the video files and leave all the rest. You'd have
to select all, and then unselect the htm/html/php/jpg/gif.. etc..
delete, move on to the next dir.. etc. Seems a heck of a lot more like
something someone would do from a unix shell.

I did a full recursive directory listing of my site from the home dir
with a php script, and then began the restore process from the last NetApp
snapshot in Backup/Restore manager in ops panel. After it was finished (took at
least an hour), I did another recursive directory listing and did a detailed unix diff to
see if I could find any signs of modified files that might indicate someone
hacking through a php script or something. Nothing. Aside from several
hundred deleted videos and some Gallery1.x dat files which store hit counters, no
other file has been modified anywhere else on my site. I changed all my
passwords, and finally today got around to downloading the access log.

One thing dominates most of the access log. About a zillion 404 errors (46,050 to
be exact) from a single web spider trying to download all my videos. I go back to the
very first instance of that spider's IP in the logs and see this.
125.236.157.191 - - [28/Mar/2008:09:40:32 -0400] "GET /videos/misc/ HTTP/1.1" 20
0 45273 "http://www.google.co.nz/search?q=%22intitle:index+of%22+avi+avi&hl=en&s
tart=80&sa=N" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.8.1.13) Geck
o/20080311 Firefox/2.0.0.13"

So this is clearly a private individual (not a robot, so wouldn't respect robots file if I
had one). That Google search brought them to my site, and then the logs show them
going back to my top level page, indexing every page and file on the site
and then it proceeds to try to download every avi, wmv, mpg, mov, etc
simultaneously.

Roughly 13,000 successful hits later in the access log (I don't have 13,000 videos.
Big files are usually downloaded in many pieces), a strange thing happens.
Everything goes from normal 200 responses, to 404's (file not found).
Something/someone has deleted all the video files that this spider is trying to
download at virtually the same moment. 46050 more 404's from this one spider
follow, all throughout the day and it gave up around 6pm. I performed the
restore some time later that night, and see normal traffic in the access log.
I scoured the log looking for any abnormal hits to .php scripts that one would
have to use to hack into the account, and there are none. One moment
the files were there, the next they weren't. There is nothing in
the access log during that interval, other than the spider trying to download
all the files.

Here's my hypothesis. An admin at Powweb saw this huge number of
simultaneous connections inbound to my web site, and a largish amount of data going
out to a single IP. But instead of doing what I'd consider the most logical thing (blocking
that one IP either at the firewall, or by throwing that IP into my .htaccess file
in a Deny rule), instead they simply deleted all the files that they were trying
to access. But I received no email notifications from support or admins.
I've violated no terms of my user agreement with Powweb. Haven't exceeded my
file storage or bandwidth. All content is original to me.

I just now finally got through to support, explained what I've written above,
showed them the access log, and they said (paraphrased) "There are no auto generated
notices attached to your account. We have no record of anyone doing anything.
Your account has not been suspended. There's nothing else we can do."

Great, swell.

Anyone else ever see anything like this? If it weren't for the log of the
web spider trying to download all the files at once, I'd write this off as a
random "account hacked" incident, and continue to scratch my head as to how
they got in, but I have to think that 13253 hits on couple hundred large files
simultaneously over a 2.5 hour period *had* to throw up a flag at Powweb
operations and that it's no coincidence that the files that all disappeared at the same
moment, are also the same files that the spider was trying to download.

ian

[Doc C]

Wouldn't it have been easier for the hypothetical admin at PowWeb to just lock down the account rather than searching through directories to delete files?

[Croc Hunter]

I'm guessing the snapshot restore in OPS was successful? Shell don't work here as far as I know. I'd be updating any scripts you can and delete any you don't use. Look for files at 777 666 etc and change the permissions. Change all user passwords you can. I agree with Doc, Admins don't delete files they shutdown the entire account.

[Daemon]

Wouldn't it have been easier for the hypothetical admin at PowWeb to just lock down the account rather than searching through directories to delete files?


You would think so, although since my site was violating no terms of use,
there's no legit reason for it to be disabled. It'd be a heck of a lot easier
to just reject that IP (I've added a deny rule to my .htaccess)

Of course the same argument about "wouldn't it have been easier" could be made for the
theoretical hacker. Why go through the hassle of targetting specific file types,
when
rm -r *
would have been just as effective.

It would take ages to do this through file manager in ops control panel,
and the access log shows that it happened across all directories
that the spider was downloading from within a few seconds.

It would be moderately painful to do this from a php script, and the
access logs show that there were no PHPs or other CGIs executed around the
time that the files all suddenly disappeared. (99% of my site is static
content with only php directory indexes, and not even any of the indexes were
fetched around that time) Given the way the app servers manage CGI processes,
it'd have to show up somewhere in the log within 30-60 seconds or so of the time the
404 errors started occurring. Any process that's run for longer, will automatically
get killed.

If someone can think of a way to delete 450 odd files with specific file
extensions spread across roughly 20 directories, in less than 30 seconds without using
a php script (given that there were none executed within 10 minutes of when
the files disappeared), then by all means, let me know. All the evidence
is pointing in one direction so far, towards a non-web related process doing the
deletions.

ian

[Daemon]

I'm guessing the snapshot restore in OPS was successful? Shell don't work here as far as I know. I'd be updating any scripts you can and delete any you don't use. Look for files at 777 666 etc and change the permissions. Change all user passwords you can.

Ya, the restore was successful. And yes, I know that users have no shell access
which is one reason the evidence points to other than users.

I was a unix sysadmin for several years, so I know what to look for. I don't
have any 777 or 666 files, and if I did, there'd be changed stuff in them
but my detailed comparison every single dir/file before and after the restore
shows no modified files. The timestamps changed on all the directories
that had files deleted from them, but that's *it*. And if it was a php script
that someone used, they'd still have to execute it, and that'd be recorded in the
access log, and there's no trace of it. I know that some people's sites are so
cluttered with php and mysql dependencies that they'd never
be able to tell the difference, but mine is not. I can look at my logs and see that
*zero* php or other cgi scripts were executed for more than 10 minutes before
the files were deleted. And before that, just simple bulletproof php directory
indexes.

ian

[Croc Hunter]

It has happened here before, files magically disappear but it's very rare. You won't ever find out exactly how it happened. All you can do is continue to monitor them.

[Daemon]

A few months back, Powweb had an outage due to a file server problem and
while most people came through that ok, I had a bunch of files magically disappear
a day or two before the outtage and subsequent move (even posted it about it in the active outtage thread at the
time). I was able to restore them. I just chalked it up to the impending failure of the file server,
but changed all my passwords anyway.

As for "All you can do is".. Well, there is something else I can do. ;)

ian

[IanS]

Maybe Croc meant "All you can do at Powweb is...." ;)

[Neat Pete]

Mate, you've been hacked. Let's not have any more talk about files magically disappearing.

Just search Google for "wget buffer overflow" or "wget attacker exploits" and stuff like that. You won't be pleased at what you find. Your <<site>> has been compromised. He is emailing all his geeky-criminal friends with this discovery. They are probably trying to find other sites hosted on the same server.

Those responsible for the security of your <<site>> should be concerned.

[Builder]

Just search Google for "wget buffer overflow" or "wget attacker exploits" and stuff like that. You won't be pleased at what you find.

The only return from Google on "wget attacker exploits" (http://www.google.com/search?client=opera&rls=en&q=%22wget+attacker+exploits%22&sourceid=opera&ie=utf-8&oe=utf-8) is... umm... this thread. More specifically, your post. Actually pretty impressive that Google has crawled and indexed this forum in just 6-7 hours.

As for the search term "wget buffer overflow", the returns for that are articles that are 3, 4, and 5 years old which mention vulnerabilities in Red Hat, Fedora, SuSE, and other flavors of Linux. Surely the vaunted open source community has dealt with those vulnerabilities by now? ;) Hmm. Well, yes I see they have: http://securitytracker.com/alerts/2005/Oct/1015056.html

[Daemon]

Mate, you've been hacked. Let's not have any more talk about files magically disappearing.

Just search Google for "wget buffer overflow" or "wget attacker exploits" and stuff like that. You won't be pleased at what you find. Your <<site>> has been compromised. He is emailing all his geeky-criminal friends with this discovery. They are probably trying to find other sites hosted on the same server.

Those responsible for the security of your <<site>> should be concerned.

For someone to exploit that particular form of attack, I would have to be executing
wget from within a script on my site, and any php or cgi script that did so, would show
in the access log. There were no PHP or CGI scripts in the access log within
10 minutes of the time the files disappeared, and those in the log before that, are
simple php directory indexes which all show as being unmodified.

ian

[patr547]

A few months back, Powweb had an outage due to a file server problem and while most people came through that ok, I had a bunch of files magically disappear a day or two before the outtage and subsequent move (even posted it about it in the active outtage thread at the time). I was able to restore them. I just chalked it up to the impending failure of the file server, but changed all my passwords anyway.

If this was the first/only time this has happened to you, I would say maybe it was just a weird malfunction..and its a good thing the backups were there. However, it seems a little weird that you would have files "disappearing" in this manner more than once, in such a short time frame. The last time this happened, was it the same types of files which were removed, or was there any sort of other correlation with what happened this time?

[dmacminn]

When you say the files disappeared - are they physically not present (not showing in FTP connection) or just not showing in the FileManager/dir listing? FM has a limitation in showing files over 5MB ... it uses a control that chokes if it finds large files ....

[Daemon]

Nah, they were gone gone. Not in FTP, not available via their URLs (obviously
given all the sudden errors in the access log), and didn't show in the latest archive
snapshot. Just 450 files across 20 disparate directories all gone at the same time,
with no trace of any PHP/CGI used anywhere in the access log within 10 minutes of when
it occurred.

ian