Search:  

 
 
   News
newer
story category Data Center Fire at The Planet Disrupts Popular Blogging Service
9,000 servers temporarily affected but should be up and running soon
(old news - 09:49AM Sunday Jun 01 2008)
tags: business · hardware · trouble · Oddities
The Planet is a dedicated server hosting provider which operates six large data centers in Texas. One of their Houston-based data centers suffered from a fire last night which has left all servers there temporarily down which means that approximately 7,500 customers can’t access their websites. One of those customers is Entrecard, a popular social networking and advertising service for bloggers. The fire apparently didn’t do any actual damage to the servers but caused the power to go out which took out the service. As of this morning, they don’t have an estimated time of repair but are providing hourly updates to customers here.

Related:
  1. AT&T VRAD Shot, Killed
  2. Startup Offers Smart Phone Rentals
  3. PA Man Charged With Selling Hacked Cable Modems
  4. Nortel Files For Bankruptcy Protection
  5. Belkin Busted Paying For Positive Amazon Reviews
  6. New Verizon Femtocell Service Is A Bad Joke
  7. New Docs Show FCC Glossed Over BPL Flaws
  8. The NSA Doesn't Take Kindly To Fiber Cuts
Forums » Data Center Fire at The Planet Disrupts Popular Blogging Service
view: topics flat text 
Post a:
Jonbo298

join:2004-01-12
Council Bluffs, IA

1 edit

Howardforums.com also

But not completely. Disrupted any images hosted on the site. (caused it to be egads slow).

sansri88
Go digtal you analog laggards
Premium
join:2005-12-17
Iselin, NJ
clubs:

Proboards affected too

Somewhere around 27 Proboards servers are affected by this as well.

frankenfeet
Premium
join:2001-10-14
Smiths Grove, KY

VCD Quality as well.

Just checked out VCDQ and it's down too.

Nsane_iceman
Workaholic
Premium
join:2001-02-26
Bedford, TX
clubs:

Re: VCD Quality as well.

said by frankenfeet See Profile :

Just checked out VCDQ and it's down too.
Wow, sucks.
I also have a MPLS issue in Houston.
The internet ends in Houston?

Gerard780
Premium
join:2002-01-25
Simpsonville, KY
Of all the thousands of sites down I'm surprised someone mentioned mine.. I'm honored..

-Gerard
wolffer

join:2002-01-31
MS
Its back up now. Not updated since Saturday.
ross

join:2000-08-16

digizip.com is also down

Digizip's servers are hosted in Texas and are also down due to power outage.

joako
Premium
join:2000-09-07
/dev/null

Re: digizip.com is also down

Good. They don't deserve to be online.
ross

join:2000-08-16

Re: digizip.com is also down

said by joako See Profile :

Good. They don't deserve to be online.
?

joako
Premium
join:2000-09-07
/dev/null
·AT&T U-Verse

Re: digizip.com is also down

said by ross See Profile :

said by joako See Profile :

Good. They don't deserve to be online.
?
They're a bunch of scumbags. I had a DSL line with them and one day it just stopped working (it would still sync) I told them to fix it and a week later they still had no clue, so I told them to cancel. I kept on getting bills, despite countless calls to them.
--
09:F9:11:02:9D:74:E3:5B:D8:41:56:C5:63:56:88:C0
ross

join:2000-08-16
·Digizip

Re: digizip.com is also down

said by joako See Profile :

said by ross See Profile :

said by joako See Profile :

Good. They don't deserve to be online.
?
They're a bunch of scumbags. I had a DSL line with them and one day it just stopped working (it would still sync) I told them to fix it and a week later they still had no clue, so I told them to cancel. I kept on getting bills, despite countless calls to them.
Gee, at least they were communicating with you...*snicker*

joako
Premium
join:2000-09-07
/dev/null

Re: digizip.com is also down

IIRC it was mainly their voicemail.
Surfinusa
Premium
join:2001-02-08

Re: digizip.com is also down

How depressing.

pokesph
It Is Almost Fast
Premium
join:2001-06-25
Sacramento, CA
clubs:

The Planet

Nice to know that, as a The Planet customer, I have to read / learn about this major outage issue from DSLR.

Thanks guys!

phattieg

join:2001-04-29
Winter Park, FL
·Verizon Wireless B..
·Sprint Mobile Broa..


1 edit

Re: The Planet

Well, I'll be honest about it. If I ran a company, I would be more concerned about damage control than telling you, the customer, what is going on. I mean think about it, a FIRE occured, the first thing on the owners mind was "whats down, whats up, who should I call, how many people are affected, what can we do to get back up ASAP before people sue". So before you get upset that DSLR "reported" it before the company did, you have to think, the company has to prepare a press release about how the fire got started, what they've done to make sure there is a backup in place if it happens again, etc, etc, and it all has to be worded in a long winded, lawyer friendly fashion. So I can safely say I am grateful it didn't cause the affected customers downtime resulting in days, and I happened to used quite a few of the affected sites myself within the last 24 and 48 hours reprogramming my old Verizon post paid phone to a pre-paid using QPST and the help of howardforums.com. Thanks to them, ironically, the internet on my cell phone started working properly after I seen a screen shot in the "how to" section, and found the hex code for the internet service in the actual shot that was hosted by this provider. So kudos to them for damage control and getting the site back up before I noticed it was down!

EDIT: I am actually posting this message on my PC using the same cell phone I was programming because ironically the speeds are faster than the apartments owned cable internet service. I get about 300 kpbs more download, and about 40 kbps faster upload. And although Verizon's prepaid plans suck for prices (99 cent a day to use it for that day, 10 cent a min, 10 cent per text, unlimited mobile web for 99 cent a day, thats what made me do it.). Besides, my Sprint Moto Q just died, and I have to save up for a replacement, so I am using the prepaid till I get my Mogul.

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN
·Verizon FIOS

said by pokesph See Profile :

Nice to know that, as a The Planet customer, I have to read / learn about this major outage issue from DSLR.

Thanks guys!
Maybe you could check in the System and Network Status forum at ThePlanet, and in particular this thread.

iv81

@mindspring.com

this is obviosly due to power grid modifications performed

This is obviosly due to power grid modifications performed recently to reduce power consumption with installed air conditioners, read the article here »www.thehostingnews.com/news-dedi···306.html

leibold
Premium,MVM
join:2002-07-09
Sunnyvale, CA
clubs:

Re: this is obviosly due to power grid modifications performed

That article doesn't really mention anything about changing the power grid, it does talk about improvements to the cooling of the datacenters and associated energy savings.
Reducing the power consumption shouldn't cause a transformer explosion, on the contrary it should make it less likely.
The only item in the article that raised a warning sign in my mind was when they mention "Sealed power distribution units to reduce bypass airflow" since the PDUs need cooling too.
--
Got some spare cpu cycles ? Join Team Helix or Team Starfire!

Markcausa

@sbcglobal.net

What a crazy couple of days...

I have hosted with The Planet for about 3 years and run a hosting company.

Within two hours of the outage, we had new servers provisioned in their Dallas data center using their Automated Provisioning system.

It has been a very long night, but their provisioning system saved my company tonight.

Defiance82
Computer Elite
Premium
join:2002-09-11
Reeds Spring, MO
clubs:

sucks

This outage is going onto 24 hours almost. The Planet continues to not get an ETA which is very unprofessional IMO. I have 3 servers with them that are currently down due to this.
--
Programming is like sex. One mistake - endless support.

nightwalker
Nightwalker

join:1999-08-07
Appleton, WI


2 edits

cover story or something fishy at theplanet.

I just chatted with a buddy who was onsite during the supposedly 'electrical' fire, and said everything was business as normal.

Chat Log:
[Sunday June 01 2008 - 14:48:47] the whole thing with TP
[Sunday June 01 2008 - 14:48:51] is really fishy
[Sunday June 01 2008 - 14:49:12] i was there last night
[Sunday June 01 2008 - 14:49:17] and there was no trucks
[Sunday June 01 2008 - 14:49:21] no nothing
[Sunday June 01 2008 - 14:49:30] no electricians
[Sunday June 01 2008 - 14:49:58] so either they renamed the spring tx location tohouston 1
[Sunday June 01 2008 - 14:50:03] or nothing happened
[Sunday June 01 2008 - 14:55:07] the msg from Doug came at 11PM
[Sunday June 01 2008 - 14:55:18] i was there at 2 AM and everything was fine
[Sunday June 01 2008 - 14:55:23] and stuff is still not back.

He thinks there might have been a search warrant for servers, and they are using an electrical outage as a 'cover' story to hide the real truth of what happened.

--
»www.reverse.net

deaninnc

@rr.com

Re: cover story or something fishy at theplanet.

I dont know if people realize how much of an outage this is causing. There are hundreds of thousands of sites if not more down because of this.

I have no idea about the fire being a cover. All I know is there are people losing a ton of money every minute and I imagine their will be lawsuits and lots of lost business.

Lets not forget a lot of DNS runs through theplanet.com also.

I am interested to hear more details. I cannot get to the page with the status as I assume it is getting a lot of hits.

Dean
wvcaver
Premium
join:2005-04-17
Millersburg, OH

Re: cover story or something fishy at theplanet.

wonder if this is why tigerdirect.com is down ?

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN

Re: cover story or something fishy at theplanet.

TigerDirect routes to servers down in Miami that appear to be part of Internap. Unless they quickly switched over, I think it's probably just a coincidence.

FastiBook

join:2003-01-08
Newtown, PA
My first thought is an electronic terrorist attack. How would a fire in any way disable a datacenter unless they had the suppression systems turned off in the whole building? I agree. Fishy.

- Andy
patcat88

join:2002-04-05
Jamaica, NY

Re: cover story or something fishy at theplanet.

Perhaps the boys in blue forced everyone out, then the boys in red came with fire hoses (with water ).

Guspaz
Guspaz
Premium,MVM
join:2001-11-05
Montreal, QC
·Colbanet
·TekSavvy Solutions..

Transformer exploded with enough force to take out three of the electrical room's walls. Fire department wouldn't let them turn on their backup generators. Fire suppression systems are designed to limit damage, not keep you running even though your datacentre is on fire.

kruser
Premium
join:2002-06-01
Saint Louis, MO
clubs:
·AT&T Southwest


1 edit

Re: cover story or something fishy at theplanet.

said by Guspaz See Profile :

Transformer exploded with enough force to take out three of the electrical room's walls. Fire department wouldn't let them turn on their backup generators. Fire suppression systems are designed to limit damage, not keep you running even though your datacentre is on fire.
I've seen a lot of backup generator installs and many have the transfer switch(s) located very near the transformer vault.
Generally this is fine but in this case I'd imagine the fact that the walls blew several feet also caused the power cables and transfer switches to move as well causing an unsafe condition.
I've seen transformer room explosions both on the primary and secondary side and they tend to be extremely powerful

edit: The CEO has a voice message explaining that the transfer switch did fail as a result of the explosion. »service-update.theplanet.com/
lujohnson
N 0 E Q P

join:2002-08-08
Agoura Hills, CA
·Sprint Mobile Broa..

Ripples

A website I have share-hosted at 3ix went down for over 2 hours yesterday, beginning at around 11:45 AM PDT. A traceroute shows that it's now being hosted at Hostdime in Orlando FL. I'm not sure if 3ix normally hosts from there or not. They only refer to their Texas data centers on their website. Maybe they transferred some hosting? maybe unrelated?

I probably wouldn't have caught the outage on my site, except that I happened to be running a trial of host-tracker.com monitoring... funny thing is, as of now, THEIR website is down too!

ripples....
wcnghj

join:2008-05-01

Re: Ripples

It's not 3ix's datacenter, they resell from other hosts.
lujohnson
N 0 E Q P

join:2002-08-08
Agoura Hills, CA

Re: Ripples

not surprised.

ebb0ts
Premium
join:2008-01-14
Tucson, AZ
·Sprint Mobile Broa..

DC1 and the planet

»www.thehostingnews.com/news-dedi···306.html

lots of good stuff done here..

150 - 30 ton air conditioners!!! that is enough to chill Dodger stadium.

Glad no one was injured.

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN
·Verizon FIOS

Re: DC1 and the planet

said by ebb0ts See Profile :

150 - 30 ton air conditioners!!! that is enough to chill Dodger stadium.
That's across their 6 data centers though. It's amazing the amount of heat that servers can put off and thanks to blades and other high density equipment, it only gets worse...

DMS

@frontiernet.net

1 up...1 down

I have two servers over there - one is up and one is still down... 7:15pm - were supposed to have an update at 7:30...

ebb0ts
Premium
join:2008-01-14
Tucson, AZ
·Sprint Mobile Broa..

What more can he say.........

from the Planet forum :
From Doug Erwin:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact.

--------------------
DSHIELD

join:2006-05-27
Micmac, NS

StatCounter

StatCounter Free invisible Web tracker, Hit counter and Web stats is down
lujohnson
N 0 E Q P

join:2002-08-08
Agoura Hills, CA
·Sprint Mobile Broa..

Pictures of the Houston Data Center

Found this a few minutes ago... they don't show any pictures of the recent damage of course, but a pretty good set of pics of their Houston center ....

»blog.theplanet.com/2007/08/07/data-centric/
Madtown

join:2008-04-26
Madera, CA

Data Center Fire

This morning my DSL went down, AT&T HSI for several hours and couldn't get online from 6:30am til after 2pm today, not sure if this has anything to do with it, but thought it would be useful to put in this thread.
voogru

join:2001-07-22

Re: Data Center Fire

I'm all the way in Miami FL and I experienced an outage of my AT&T DSL during those same hours.
flycuban

join:2005-04-25
Homestead, FL

Re: Data Center Fire

we have four OC-3's at my work place here in Miami, and at&t told us last week that they were performing "major upgrades" to their network here in Miami. Found out this morning that they are migrating parts of the old bellsouth.net network onto the at&t network.
patcat88

join:2002-04-05
Jamaica, NY


1 edit

fire dept

The local Fire Dept and cops and their "emergency response team" is probably looting the datacenter as we speak. Remember, if you go back into the building while its still "dangerous" you will be arreasted, and you can't stop the Fire Dept from water hosing all the server "just to be safe", even if the fire was many floors away.

The racks will also require buzz saws and the jaws of life to make sure there is no fire inside the cabinets since they have no clue how to pull out 19 inch rack systems.

So much for Halon when the Fire Dept will water it down anyways.

So what does happen when the Fire Dept demands to buzz saw and soak servers needlessly in a datacenter, or the police dept seals the building for a few days to do an investigation? Hope your insurance is paid up and you have instant offsite failover and continuous realtime DB mirroring offsite? Or this stuff doesn't happen/there are laws to protect datacenters from the demo and reclamation crew?

AGBell

@cox.net

picture worth a thousand words

lujohnson's link for photos of one of the Houston centers tells the story. I was trying to understand why the explosion with three walls demolished. Battery acid fumes ignited by fire from a shorted conduit wire (some electrician's ass is in a serious sling).
Forums » Data Center Fire at The Planet Disrupts Popular Blogging Service


Saturday, 04-Jul 04:20:41 Terms of Use | Privacy Policy | Hosting by www.nac.net - DSL,Hosting & Co-lo | feedback | contact
over 9.5 years online! © 1999-2009 dslreports.com.