site Search:


 
   
story category
Shaw HQ Explosion Disrupts Service
911, Broadband, Radio, Hospital Services Disrupted
by Karl Bode Thursday 12-Jul-2012 tags: business · bandwidth · cable · trouble
Tipped by state See Profile
Canadian cable operator Shaw Communications this week suffered a significant disruption in service, courtesy of an explosion at their corporate headquarters. The explosion originated in an electrical room inside the company's headquarters, resulting in an outage for radio stations, broadband service, hospital computers, government networks and regional 911 service. A transformer fire began the explosion, with backup systems then being taken offline as sprinklers kicked in to control the fire. Fortunately it appears that nobody was injured by the fire or explosion. "All I can tell you right now is that it's all hands on deck as we try and resolve this situation as soon as possible," a Shaw employee posts in our forums. Shaw's Facebook page and Twitter account are providing updates.

view: topics flat text 
Post a:
elray

join:2000-12-16
Santa Monica, CA

Boom!

Transformers explode.

Those who depend on a single pipe from a cable company WILL experience massive failure(s) at some point in their lives.

But hospitals and 911 service?

How far we've regressed.
bt

join:2009-02-26
canada
kudos:1
Reviews:
·Start Communicat..

Re: Boom!

said by elray:

But hospitals and 911 service?

How far we've regressed.

911 was down because the phones were down.
MattAKM

join:2002-08-04
Anoka, MN
From my experience most managers, telcos, and exec's cannot distinguish the difference between virtual and physical connections.
BiggA

join:2005-11-23
EARTH
Yeah, that's pretty bad to not have any sort of backup for something that critical. It's really surprising how many systems and pieces of infrastructure aren't properly backed up or backed up at all.

battleop

join:2005-09-28
00000
Cheap, Fast, Reliable. Pick two.
elray

join:2000-12-16
Santa Monica, CA

Re: Boom!

said by battleop:

Cheap, Fast, Reliable. Pick two.

I doubt that the failed systems met any of the above.

Raptor
Not a Dumptruck

join:2001-10-21
London, ON

Fire Supression System

"Water and electricity aren't a good combination," Bissonnette said.

So, the backups came online as intended only to be shorted by the sprinklers. Am I missing something, why would they not have an inert suppression agent (e.g. Halon, etc.) in the data center assuming the important gear is in its own room....

Seems like something one would do with such important data being handled.
--
....where's my fiber?
flashcore

join:2007-01-23
united state

Re: Fire Supression System

Based on experience, electrical transformers are NOT inside the datacenter, they are in a utility riser in large buildings like that and they usually do have sprinklers to cover it. As for the backup systems, typically when the sprinklers go on, the fire systems shut off any and all electrical in the facility, backup or otherwise to prevent electrocutions.

That all being said, when a transformer on the 13th floor explodes everything on every floor below it where the sprinklers go off will be soaked as well preventing an easy restoration of the other floors. In this case most likely every other utility from ground to 13 is soaked.

Raptor
Not a Dumptruck

join:2001-10-21
London, ON

Re: Fire Supression System

Thanks.
Kearnstd
Elf Wizard
Premium
join:2002-01-22
Mullica Hill, NJ
there has also been major pushes to get Halon out of the data centers for personnel safety and I think its also as bad as old freon on the ozone layer.
--
[65 Arcanist]Filan(High Elf) Zone: Broadband Reports

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Re: Fire Supression System

said by Kearnstd:

there has also been major pushes to get Halon out of the data centers

..and replace them with what? Sprinklers?

Sure, you won't die from having the oxygen sucked out, but you'll die by electrocution.
Kearnstd
Elf Wizard
Premium
join:2002-01-22
Mullica Hill, NJ

Re: Fire Supression System

I can imagine that any good fire system using water around servers would automatically cut power to the room when the fire alarms go off.
--
[65 Arcanist]Filan(High Elf) Zone: Broadband Reports

Mike Wolf

join:2009-05-24
Beachwood, NJ
kudos:3
Halon has really been mostly banned in most data centers which are being retrofitted with a more enviornmentally safe alternative. Data centers which cannot be retrofitted are still allowed to use halon.

bbeesley
VIP
join:2003-08-07
Richardson, TX
kudos:5
said by Simba7:

..and replace them with what? Sprinklers?

FM-200 »www2.dupont.com/FE/en_US/products/FM200.html

Juggernaut
Irreverent or irrelevant?
Premium
join:2006-09-05
Kelowna, BC
kudos:2
Halon is deadly, that's why. Full masks are required.

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Re: Fire Supression System

There is a reason it is used, though.

Sure, I'd love my $50-100k server to be soaked by water. Are you going to replace it and the data?

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN
kudos:7

Re: Fire Supression System

said by Simba7:

Sure, I'd love my $50-100k server to be soaked by water. Are you going to replace it and the data?

Are you going to pay the multi-million dollar settlement for wrongful death when someone suffocates when the system discharges?

If you have $50k-$100k tied up in a server, buy some insurance. And hopefully a portion of that money also went into some type of a backup. Even if water doesn't destroy your equipment, what would happen if a fire did?

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Re: Fire Supression System

You do realize you have a minute or two before it releases on top of a loud alarm going off?

If you don't get the clue of "Get the fsck out of there", you deserve to be removed from the gene pool.
kamin

join:2003-03-30
Puyallup, WA

Re: Fire Supression System

Hahaha. That made my night.

When I worked for an ISP many years ago, that was the first thing the owner taught me when we went to a data center where we have servers: "If that alarm and flashing lights go off, GET THE HELL OUT NOW OR YOU WILL DIE."
Joe12345678

join:2003-07-22
Des Plaines, IL
some fire codes don't let them use halon others are duel systems.

Draco

@184.16.42.x
some of these telco buildings are old and it takes a long slow time to modernize. There are modern replacements to halon that are many factors safer for people and equipment. Novec is one, it looks and acts like water, but does not react with electronic components in any way, you can actually fully submerge computer systems in it if they do not depend on hard drives or moving fans for cooling as well. but this type of event will give them the opportunity and the insurance money to "update" their systems

»www.periphman.com/Fire-Suppressi···ions.htm


Raptor
Not a Dumptruck

join:2001-10-21
London, ON
Very much aware Halon is bad news and all but banned in Canada. We have boat loads of gov't paperwork to fill out should a release occur. I used the term "etc." for a reason . And I don't have enough familiarity with data centers to know if alternatives (FM-200, etc.) can be used.

Lots of good information in this discussion though from everyones different experience!

I just figured that if the engineering was done right, emergency response systems shouldn't cause a semi-unrelated cascade failure in ones backups. The backup location/setup doesn't seem ideal to achieve redundancy if the main systems gets hit and the response is to cripple the backups; ESPECIALLY in a 911 data routing location.
--
....where's my fiber?
eqshadimar
Premium
join:2004-10-20
Plano, TX

No Disaster Recovery Plan?

Humm the company I work for is pretty small (couple hundred of people) and even we know that your backup recovery data center needs to be located at least 20 miles from your main location. At least have it in a different building. I guess they do not get audited by SOX 404 auditors like we do. We could not have passed like that.

Talk about having all your eggs in one basket. Just another example of not spending the money on infrastructure.

Laters,
Jeff

cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN
kudos:7

Re: No Disaster Recovery Plan?

said by eqshadimar:

Humm the company I work for is pretty small (couple hundred of people) and even we know that your backup recovery data center needs to be located at least 20 miles from your main location.

This isn't just a data center. It's a central office where thousands of lines are terminated. You don't just switch to an off site backup center. Some functionality may be able to be moved offsite or otherwise routed around. But others may not be able to just have a switch thrown and it all magically start to work again.

I guess they do not get audited by SOX 404 auditors like we do.

Yeah, I'm going to bet that SOX 404 doesn't apply to a Canadian company doing business in Canada.

Talk about having all your eggs in one basket. Just another example of not spending the money on infrastructure.

I don't know too many major central offices that can suffer from an electrical explosion followed by fire/water damage and just keep chugging along like nothing happened.

Killa200
Premium
join:2005-12-02
Southeast TN

Ah...

Its The Planet all over again....
horseathalt7

join:2012-06-11

Re: Ah...

You don't need halon anymore most advanced fire suppression systems that are around commercial electronics use carbon dioxide now. Depends on the fire codes in various territories.
jdmm72

join:2002-02-12
Nitro, WV

Fibernet/Frontier/Verizon

In my area, in WV, when any of these entities fail 911 centers also fail.

It has happened multiple times. The last time was blamed on a switch owned by Fibernet/nTelos/Lumos. 911 service was out for something like 24 hours.

antdude
A Ninja Ant
Premium,VIP
join:2001-03-25
United State
kudos:4

One place?

Again, not a good idea to put everything in one place/basket.

boogi man

join:2001-11-13
Jacksonville, FL
Reviews:
·Comcast

Re: One place?

said by antdude:

Again, not a good idea to put everything in one place/basket.

Like any business they play odds. Every now and then bad things happen. What about power? Should we all just run our own genset to avoid central failure?
--
my site
jst3751
Premium
join:2004-07-08
Rowland Heights, CA
What you do propose, reduntant set of phone lines from every building to 2 different COs? And just who is going to pay for that?

anonanonanon

@telus.net

Halon

Being in a building with a halon release requires evacuation. The evacuation can not be completed in time before people are suffocated.

Saturday, 25-May 22:28:19 Terms of Use & Privacy | feedback | contact | Hosting by nac.net - DSL,Hosting & Co-lo
over 13.5 years online © 1999-2013 dslreports.com.