The PowWeb Apology...

[KenCarlson]

We believe the worst is behind us. We have been in business long enough to know that unexpected problems will arise and we will tackle them as they come. Although some of you believe we did not handle the move correctly, I can promise you that we had all the best intentions. First of all. I don't think anyone feels that PowWeb had anything but good intentions. No company would intentionally inflict such pain on its paying customers. No, we all believe that intentions were not part of the problem.

The part in the above quote that I have real issues with is the sentence that reads "We have been in business long enough to know that unexpected problems will arise and we will tackle them as they come." This is where I believe that PowWeb just doesn't get it.

I've been a customer of PowWeb's since February, and I have, along with my fellow customers, endured several periods of instability due to upgrades. Just like PowWeb's apology implied, the normal mode of operations for PowWeb is to boldly charge forth with new things, and then deal with the problems as they arise. What PowWeb doesn't seem to get is that this way of doing things is NOT a good business practice.

Every time this happens, the PowWeb techs work their butts off to make things right. They spend countless hours performing personal acts of heroism to put out the fires that always arise. And while I appreciate their hurculean efforts, there should be nobody that gets a pat on the back. Instead, there should be a big boot in the *** to the guy that continues to plan upgrades under the philosophy that "We'll just fix the problems as they arise".

PowWeb, you've grown too big to rely on personal heroics. You've got to institute some PROCESS into what you do there. Make changes slowly - incrementally. DOCUMENT standard configurations and procedures so that mistakes aren't repeated. Hire some professional Systems Engineers that are formally trained in System Process. Adopt zero tolerance policies. ANTICIPATE problems before they become problems and then avoid them.

If you're running a small business out of your basement and only have a handful of clients, then it's appropriate to charge forward and put out fires as they arise. However, PowWeb is not a rinky-dink business. Your client list is too large to operate that way.

It has been claimed that you were forced to leave Level3 before you planned to because of "unforeseen problems". Based on my personal experience, I doubt the transition would have been any smoother had you had more time (because you don't seem to take a systematic approach to system migration EVER). However, what I want to know is the nature of the problem. My guess is that it was a contract dispute... I'm purely speculating, but I'm guessing that the problem with Level3 was not technical, but that the company would have been forced to pay extra or something, to give you more time to transition. The "unforeseen problem" with Level3 was financial, not technical. It could have been avoided.

Andrew, have you fired the guy that was responsible for monitoring the relationship with Level3? Have you fired your Chief Engineer responsible for the fiasco that we've all been going through? Are you going to hire some serious professional Systems Engineers? Have you at least documented the details of all the mistakes so that they never happen again? Or are you too busy putting out fires?!!

All your customers are pissed off. Many of us have professional reputations that suffer because of PowWeb's bungling during upgrades. Given the circumstances, the apology you've issued is appropriate, but I'm getting less and less satisfied with the same old apologies over and over.

We are passionate about what we do and our work has become our lives.This is fine. I don't doubt your sincerety. If you are truly committed to this company and this business, you'll realize that a lot of what I'm saying is right. You'll realize that this company that you and your friends built from scratch has grown, and that perhaps it's time to bring in some outside talent. Nobody questions your intentions. Nobody is saying that any of the PowWeb staff is incompetent or unmotivated. But when Charlie-Foxtrots like this happen with every upgrade, it IS apparent that you may lack the personnel with the right skill set.

Your claim is that you are the perfect hosting solution. Based on my experience, I'm not sure you all realize how close to that you really are! You've got a great suite of services at a great price, and a stable of trained "firefighters" that is probably the envy of many companies. Bring in some outside talent and make him/her responsible for planning upgrades and migrations. Make him/her responible for implementing some formal process into what you do there, and I'm sure you'll be an even STRONGER company.

----------------------------------------------

As I reread what I've written, it sounds like a bunch of sanctimonious crap that is sure to piss of a lot of the PowWeb staff. But hey, I'm pissed off over what's gone down, and writing this has been theraputic!

Good luck to all!

Ken Carlson

[the_radix]

Yeah well said Ken..

Thats what forums where invented for..

To vent rage, anger, love, hatred, opinion, belief, religion, grace, happiness and the millions of other emotions we all go through each day...


You bought up some excellent points which I would also like to fortify with my beliefs.


As Ken has pointed out, it seems every time you do an upgrade there is a related downtime due to it..

It's not that the upgrades are not appreciated or unwanted, but perhaps with more skilled planning and better organisation there would be only the bare minimal negative affect on clients if any at all..


I understand that in this case the actual physical moving of server's makes it much more difficult to keep the sites online as while the boxes are being transported across town they are obviously "dead" to the world..

Perhaps though a slower more thought out transistion could have been planned though.. Such as starting up one server in the new datacenter and then progressively moving websites from the old datacenter servers to the new server, then move that old server into the new home, and repeat the same process with the other machines.. That way you can virtually guarantee a 0% downtime during the process if everything transfers ok..



Anyway.. Enough said on the topic.. I still love your service and the features of your package and I have very little other choices anyway considering i live in Australia and I'd rather have a few days downtime each year and be on powweb, than pay the hundreds of AUD I would if I hosted locally in Australia..


Peace out..

[John_paaw.com]

I agree Ken, and I am the process of moving the Paaw.com domain to another provider who provides the exact same mb space and bandwidth, and price. But as the Paaw.com domain grows, it will eventaully need its own server anywho.



I wish them luck and they'll be losing my business. Hopefully they'll fix their business "thinking" and correct this and end up a really good web hosting company. Maybe.

[James]

I would like to point out that this does not happen on every upgrade. There are a number of changes EVERY DAY to our system to help improve service and/or offer more services or options. Many of the changes go by without a hitch. Many of the changes that do happen that cause problems are forced upon us by need. PowWeb has grown 200% since November 25 of last year!! With our ever expanding customer base, we simply out grow things and need to make changes.

Ken, you have only been with us since Feb. then you were not with us from before Sept 25 2001, we were using a completely different system that offered very little control over packages (The only think you could do was manage email aliases and pop accounts, or add DNS info). There was no accounting, no domain registration control. The web based mail system was POP based only and didn't have any features it has today. We didn't have a forum. The list of changes/improvements goes on and on.

Customer who were with us from the beginning might remember the speed improvements and greater stability of our network the first time we moved over 2 years ago. We cannot control everything, like how quickly your ISP fixes their broken name servers or dumps your cached zone info. We had all our customers zones set to expire in 2 days, then 3 days before the move we made them an hour. For most people this worked fine because we did ANTICIPATE the DNS changes and made them. We had to move our network and re-ip our machines. There was no other way to do this without an outage, and I don't see how any company could have moved their data center and machines without the same result. To say ALL our customers are pissed is wrong. I sent the emails out and I watched the replies come in all morning.... I finally composed two long emails to send to all powweb's staff because of the great things they were saying and thought all our staff should hear them.

Personally, as an employee and customer of PowWeb... I would rather have to deal with an outage once every 6 months or so for better service, more options, and a company thats not afraid to do the work it requires to do such upgrades, then to stay with a company that never offers anything more or improves its service. Companies like that tend to end up with a BIG outage... they usually go out of business. Many of our customers have already replied saying great things and how they hardly were affected by the move. To say they are all pissed off is WAY out of line. I sent out them email and went through the replies. I made two long emails about the customers congrats and thanks and sent it to our staff just because whey they said was so nice and I thought all of our employees should hear it.

[stevel]

A word or two if I may..

I am a "newbie" to PowWeb, having my first domain moved here in June, and a second in August. I'm also webmaster for a small company and recommended that they move to PowWeb, which they also did in August. I've also recommended PowWeb to three others to date, two of which have set up shop here.

Since then there have been numerous outages and times of great frustration, some of which seemed to be due to PowWeb's difficulty in thinking ahead. But when all is said and done, I'm still happy with the service, and would still recommend it to others. Why? Because I truly believe that the staff of PowWeb is really trying to do right by their customers and that they care about each and every piddling little $7.77 "package". I moved here from Interland, about as big as hosts get, and while I hardly ever had an outage, I felt that they focused only on the large corporate accounts and weren't interested in "the little guy".

Yes, I came in at a particularly painful time to be a PowWeb customer, and I can think of many opportunities where they could have done things better, but the spirit is there and that's what counts by me. I do believe that things will keep getting better from now on.

So, while I will continue to point out problems and flaws, I plan on remaining a PowWeb customer and will continue to recommend PowWeb to others.

[JoeP]

Powweb just does not get it....

How much money has been lost because of this to site owners?

I've been a customer for over 1.5 years and I've noted a decrease in service not an increase!

Each time the problem is multiplied because of the lack of action to correct problems.

I've just transferred I think 6 domains from their domain registrar because of poor service.

I'm just about ready to move on again, but I'm afraid I was too stupid to back up site prior to this latest fiasco.

And to all those who claim "you get what you pay for" 99.7% of the time you do.

I just thought Powweb was different.

Fool on me.

[mapleone]

Chill out guys,

Notice the fine speed today!!!!!

[cjdj]

Mr Paaw.com guy,

You mentioned that you were moving servers because of the dissastisfaction over Powweb's recent upgrade. You mentioned that you are looking for more stability.

I looked at your website, and you currently have it down for 2 days during the transition to the new host.

Now I wonder, couldn't you have done this move with no downtime? Surely you didn't shut your site down for 2 days because powweb was down over the weekend are you?

Well, its ok really. I had a chuckle when I saw that. But anyway, good luck with your move, and such.

[iisven]

here's my 2 cents:

I joined up with powweb back in June looking for a web host that would be affordable for a 15 year old. I was impressed with what I saw, and I still am. This is my 3rd foray ever into the world of web publishing, and I can only say that it has been a good experience for me. I have learned a lot, and accomplished a lot. Maybe this is due to the fact that I have little other experience to go by, but I still think I'm getting more than I paid for (that's not to say you should raise the price, though :cool: )

Granted, mine is only a hobby site (although I was still bugged by people over the weekend), but over the past 3 months (and beyond), I've had a good experience with powweb. I would still recommend it to anyone interested, and intend to stay here as long as I can.

I know about growing pains, I've experienced a few with my own community. Despite the recent issues, I will stick with powweb and continue to support it.

[mapleone]

Hello iisven

When you get the urge, let us know your site address,

I too do it for a hobby and work etc.

Maybe we can help you learn

[KenCarlson]

To James and the other defenders of PowWeb:

If you read my post carefully (sorry, I know it was LONG), I never said I didn't like the upgrades. In fact, I complimented PowWeb on a "great suite of services for a great price". When things are stable, I am as happy as a clam.

Upgrades and improvements are great! No sane person would argue with that. My point was that upgrades and improvements are possible without the pain we've all endured if they are planned appropriately and implemented methodically.

Given a choice between enduring PowWeb's bungling in exchange for a vastly upgraded system, or not getting any improvements or upgrades at all, I think I'd take the "pain" for the improvement (that, incidentally, is why I'm still a customer). But the "perfect hosting solution" would deliver the same improvements without the meltdowns. Do you all see the difference?!!!

The goal at PowWeb (and any truly professional organization) should be to deliver the goods seamlessly and painlessly... NO, it is NOT impossible to conduct a major system migration without problems!! The problems everyone's been experiencing were NOT inevitable. The mistakes should not be dismissed or forgiven just because they led to improvements.

Heck, I LOVE the improvements... KEEP 'EM COMING, PowWeb... Just don't keep stepping on it while giving them to us, okay?

And stevel, I agree that PowWeb's techs are as dedicated as they come. In fact, in my original post I call them "heroes". Again, my point was that their heroics should NOT be necessary if system migrations are done with professionalism. The heroics should be in the planning, not in the cleaning up of the mess that improperly planned and executed system changes create.

I just can't say it any simpler than that.

Ken

[sandy]

I looked at your website, and you currently have it down for 2 days during the transition to the new host.

CJDJ's comment is petty funny when you think about it. Yes, John was quite angry, and he stated that quite eloquently. But now that Powweb appears to be running at warp speed, John takes his system down for up to 2 days (and that assumes that the new host functions as planned). I think the whole thing is pretty ironic!

Sandy

[MarkHutch]

When I was a kid my dad said to count to 100 before I hit some other kid that made me mad. I was pretty hot headed as a kid so I had to count to 1000, but the advice is still good. Don't make decision when you're angry. Whether it involves hitting someone or switching hosting providers.

[James]

Okay fine,..... I still would like to know how you are going to physically move a machine without powering it down, take it of a wired network and re-ip it on another network with out an outage.

[KenCarlson]

James, actually, another poster already suggested an idea that would work.

1) Take a new server, not already in use. Surely an organization as large as PowWeb has at least one "float" server for emergencies.

2) Set this server up at the NEW data center.

3) After informing all the clients on one server only, make a tape backup of the entire system, and then freeze it (temporarily no more updates on the server at Level3).

4) With the original server at Level3 still running (but with updates temporarily frozen), run the tape of the file system over to the new data center and upload the content onto the float server.

5) Run through all the testing protocols using the IP address of the new computer.

6) Once you are SURE that everything is working on the new server, make all your DNS changes so that propagation/cache refresh can occur, leaving BOTH sites live for about 3-4 days.

7) Once DNS caching is no longer an issue, and 99% of all ISP's are pointing the domains involved to the NEW IP, take the old one offline. Power it down and move it to the new data center.

8) This machine that you've just taken down and brought to the new data center is your new "float" server.

9) Record all the problems/challenges/issues with the first server migration and make a point to mitigate these as you repeat the first 8 steps with subsequent servers. Give the guinea pig group (the first group) a free 6 months or something for their trouble.

Go through this process iteratively until you no longer need the space at Level3. THEN terminate your contract with them.

This process would take several months instead on one Sunday, and would be far more expensive because it involves paying for service on both Level3 and the new data center during the overlap. Also, it involves the use of float hardware that has to be purchased. Cheap and easy? No. Better for your customers? Yes.

Is what I've suggested possible given "unforeseen problems" with Level3 that FORCE you outa there in three days time? Of course not, but had this thing been properly planned (including the additional expense of doing it RIGHT), it wouldn't have been an issue.

I'm no expert, so I'm sure that there are some holes in the process I've just described, but a fuly qualified and formally educated Systems Engineer (another expense that PowWeb doesn't seem to want to swallow) coulda led this transition with much less (or even ZERO) impact to the majority of your customers.

So again, congrats and thank you to PowWeb for your upgrades and improvements... But shame on PowWeb for taking your bull-in-a-china-shop approach of just pulling the plug all at once at Level3 and then frantically putting out the fires when things went badly in the new data center.

James, I've got no beef with you. You are one of the heroes I've been talking about. You are the one working to put out the fires. It's the PowWeb policy makers (your boss or your boss's boss) that really screwed the pooch on this one (and on just about every other "upgrade" period us customers have endured).

Ken

P.S. It's now well over 100 hours since the plug was pulled at Level3, and I still have several domains that are not working with my particular ISP. I've never had such a hard time with DNS caching/propagation as with this transition... Still very frustrating.

[KenCarlson]

By the way, a similar approach could have been used to transfer your primary and secondary name server hardware. That's why we all identify a primary and a secondary with our Registrar...

Leave the secondary in place until the primary has moved and is working from the new location. THEN mess with the secondary.

This way, I'll bet you avoid half the problems you faced when the name servers were completely down!

Plan carefully. Execute slowly and methodically. THAT's good systems engineering.

Ken

[RSaucier]

Hey Ken,

My understanding of the problem is that they didn't have time to do the move in any manner but quickly.

From http://forum.powweb.com/showthread.php?s=&threadid=9315 , I gather that it went from a methodical plan to a "gotta do it NOW" kinda thing.

As is usual, reality steps in to screw up even the best-laid plan...

The DNS issue, though...that's got me stymied. Like everyone else, I've not seen this level of trouble with propagation. Maybe due to Wednesday's problems? <shrug>

BTW, good post -- well thought out!

[James]

The nameserver was moved first.. in fact we had two of each running, one with the old IP and one with the new. We then made the change to InterNIC. Both servers were left running during the move. By notifying one server at a time and then making a tape backup and restoring in... yada yada yada... it would have take more than a month!! In a perfect world yes Ken your method sounds great, but we had 3 days.

I am done debating this with you Ken, if you feel we could have done a better job, fine. Personally I think we did an excellent job with the situation we were given. Had we not been making plans already to move before we were made to we would have been in a very bad situation... we were prepared, and did the best we could.

[KenCarlson]

As I said before, if you you are faced with the ultimatum from Level3 that you only have 3 days to be moved, PowWeb did the best they could. James, I'll certainly grant the fact that given three days to take action, you guys pulled off a miracle.

Has Level3 completely gone out of business? No, I don't think so. My guess is that the short-fuse deadline with Level3 was not an absolute. If PowWeb management went to Level3 and said, "We need 4 months for a transition, how much will that cost us?", and then were willing to make that investment, then there would not have been a short-fuse situation at all.

The official PowWeb apology made reference to "unforeseen problems" at Level3 which led to a short-fuse situation. My question is why were the problems unforeseen? Whose job is it at PowWeb to monitor the relationship with Level3? Again, poor foresight and management forced the techies at PowWeb to be heroes...

Would the "unforeseen problems" at Level3 have gone away if PowWeb mangement had offered the $$$$$ (whatever it took) to get the extra time to do a proper transition? It is true, a proper transition WOULD have taken not just one but many months. Ask your management, James-my-man, why you and the rest of the technical staff at PowWeb were handcuffed with only three days to pull off a job that should have been carefully scheduled over many months.

No, there is no way that you will ever convince me (or any business savvy customer) that the problems at Level3 were unavoidable. "Unforeseen" okay - I believe that. "Unsolvable" with the right amount of investment - you'll never convince me.

The short-fuse situation was caused by a business decision made by PowWeb. There were other options. Perhaps they were expensive ones. Perhaps they would force PowWeb to incur unwanted debt or even lose money this fiscal year. But you'd better believe that for the right price, there were other options than scrambling to pull the plug at Level3 in three days time.

I guess I'm done debating this as well (I'm glad James used the term "debating" and not "arguing" - because this really is a discussion/debate and NOT an argument). I'm happy that all this is almost over (I'm still experiencing difficulties associated with the DNS caching problem). I look forward to a long time of stability, and based on the advertised advantages of Internap, I have even more confidense in the reliability of PowWeb as a provider.

I guess that the reason I've taken so much time to express/vent my frustration is that I see the PowWeb explanation and apology as hollow. In the same breath that they apologize and restate their commitment to great service, they try to put this whole ordeal off as something that was unavoidable - or even worse, not their fault. I just don't buy it.

Again, I LOVE the improvement... The PowWeb goal of "'complete redundancy' to eliminate all 'single points' of failure between our web servers and the Internet cloud" is right on target and well on its way to being realized. PowWeb is moving in the right direction, and I will continue to be a customer. However, you'll never convince me that the "perfect hosting solution" couldn't have been a whole lot more perfect during this transition.

Have a great day, and to all you technical heroes at PowWeb, get some well deserved rest!

Ken

[Roy]

We are still waiting for the answer, James. Why did you have just three days? I would suggest you and the others at PowWeb reread Ken's excellent posts in this thread. Start yourselves a "how to do it" manual for the next time. We all know, there will be a next time.

Roy Buckman
Former Cutover Manager for the US DOD Overseas AUTODIN

[sommert]

Well folks, this is my last day at PowWeb. I lasted just short of 30 days. I am just waiting on my domain transfer to finish and I am off with my host. I find these debates interesting on the site between the tech folks and the customers. I have been professionally employed in the IT field 12 years and have worked with a mail, DNS, proxy and other internet services during my career. I understand “exactly” what they went through during the data center move.

I am not leaving just because there was an outage. I am leaving mostly due to the communication I have seen between PowWeb and their clients. I can’t even believe techs are arguing with their customers. Guys, a little customer service tip ->. It doesn’t matter how good of a job you think you did if your customers are not happy. I think you upset more people by your lack of communication and trying to push the blame than you did with your outage. If you guys want to call this move a success in your own minds that’s up to you. The fact is your customers experienced an extended outage that was unacceptable. Customers want regular status updates in a professional manner and the feeling that you are doing everything possible to get them back up. You did not do well in that area. I can’t believe you are going to let your technical pride get in the way of good customer service and communication. I wish I had the motivation to start a Web Host. I envy you guys because you are sitting on a potential goldmine and have a pretty cool job. However, I can’t believe you are going to loose business because you are too stubborn to realize this latest change was a disaster.


Originally posted by James
The nameserver was moved first.. in fact we had two of each running, one with the old IP and one with the new. We then made the change to InterNIC. Both servers were left running during the move. By notifying one server at a time and then making a tape backup and restoring in... yada yada yada... it would have take more than a month!! In a perfect world yes Ken your method sounds great, but we had 3 days.

I am done debating this with you Ken, if you feel we could have done a better job, fine. Personally I think we did an excellent job with the situation we were given. Had we not been making plans already to move before we were made to we would have been in a very bad situation... we were prepared, and did the best we could.

[LHS1969]

Exactly, sommert - and it's this undercurrent of arrogance that bothers me the most. E-mail inquiries to tech support during the "2 to 3 hour" outage that lasted 2 to 3 days were either ignored or answered with a pre-written "remain calm, we know what we're doing" response. Reading through the threads on this forum, it's difficult to believe that PowWeb is convinced that this move was handled correctly.

I'm paid until mid-November - I have until then to find a provider that places customers first.

[laynetworks]

LHS1969 and sommert well said.

Had they were noted my complain seriously the outage might take little or planned time.

http://forum.powweb.com/showthread.php?threadid=8185&goto=newpost

There was a DNS problem since about 1st of Sept. 2002.

Kartik

[Denise]

This has been an interesting thread to read and I've learned a lot by reading the debate/discussion. Thank you posters.

Excuse the long post but I feel a need to say this.
I do have have some non-techie advice (since I wouldn't know DNS from PMS)

I would have liked to have gotten an email letting me know this was going to take place. I know you said you sent them - but I'm one of the people who didn't get one.

I then made 7 phone calls (that got through) and got tape recordings switching me to other tape recordings which then cut me off (call charged to me).

I live in NY - the calls were very long distance. I switched to PowWeb after reccommendation from friends who are very techie. I asked them about a service that I could afford - that would also provide forums like this one where a clueless person could actually interact with other people willing to lend an ear with good solid advice.

When I first left the shelter of my VAX system at the university many years ago - I joined Pipeline ...which was small and friendly...then it was eaten by Mindspring and it was not so small but user friendly for a while and then it became Earthlink and became as much of a monster as AOH**L and other cybergiants.

I became more and more frustrated - since not all of us can become puter wizzes - but many of us folks out here in cyberspace actually need to use the services and connectivity it provides - and we need helping hands from time to time who don't patronizingly write us off.

I am happy with the cost - and with the advice.

I just want to say that as a person who has run several large organizations in the past - that even a simple recorded message would have been extremely helpful to me - letting me know that it wasn't a problem on my end. It would of saved me the time making calls to my ISP, additional time spent trying to reconfigure my browser (thought maybe my husband messed it up- or the cat) and more time that I was glued to a phone set to redial getting busy signals.

When you were down - I couldn't access PowWeb, so couldn't read the announcement - couldn't you have made arrangements for an alternate site where an announcement could have been posted?

Just suggestions for the future. I plan to stay here, for a long time, and will gladly reccommend you to other economically and technically challenged artists and educators (as long as you don't become a corporate megolith).


Thanks for (hopefully) listening.


Denise

[Andrew]

Ken,

From what you know about PowWeb, you've made some valid points. There are also a lot of things you do NOT know. We are NOT here to explain the full details of our internal operations. James is the CTO of PowWeb and I stand 100% behind his actions. He has helped brought PowWeb to where it is today.

Please let US run the company and YOU just sit back and enjoy the service. We have a clear direction for PowWeb.

Please use this community forum to help each other with problems. Let us handle all the business related issues.