O Canada! > Canadian > Canadian Broadband > Fire at Peer 1/151 Front St Last Night

Fire at Peer 1/151 Front St Last Night

Teksavvy went down. See their forums for more. Would not have affected Bell or Rogers (I don't believe) as they don't use Peer 1.
--
Need-based health care not greed-based health care.

Bell/Rogers might have lost some routes that went through 151, but they stayed up. A bunch of us were using Rogers (cell/tether) or backup Bell logins.

reply to DKS
Apparently it was a battery that caught fire. Some sysadmins of other ISP's are saying that the Peer 1 UPS in Suite 802 has smelled like burnt wiring for a couple of years.
--
Need-based health care not greed-based health care.

Does anyone know if any of the automated CO2 or Halon systems got discharged ? Or did they use actual fire extinguishers to focus the gases only to the fire ?

These types of buildings are quite different from a bank data centre for instance because they have multiple tennants and it is much harder to coordinate post event recovery.

In a bank data centre for instance, the bank *should* have a number of staff specially trained by the fire department so that they can re-integrate the building (with firemen) quickly after the event in order to perform certain critical activities (such as triggering shutdowns so production can go to backup centre for systems that don't have remote management ability, recover iomportant data tapes for systems whose disaster recovery consists of taking the laters batch tapes and loading them on the offside place etc etc.

reply to DKS
151 Front Street is a disaster waiting to happen. With so much critical infrastructure of multiple communications providers concentrated there, its only a matter of time before Canada suffers a wide scale communications outage.

reply to jfmezei

reply to HeadSpinning

reply to DKS

Re: 151 being single point of failure.

If you are Bell, Videotron, Rogers, 151 isn't a single point of failure.

A building that is mission critical as is 151 should have a lot of safety built in.

It is possible to build a fault tolerant building. And within the building, one could ensure that sufficient equipmeent is located at one end while the rest is at the other end on different floors.

Perhaps 151 should have some fireproof divisions within floors.

The old Molson brewery on Lakeshore was, at one point, going to be turned into a carrier 'hotel'.

BMO got a good deal on the old IBM data center at 401/DVP years ago. That's was a well designed facility in its day. When I used to go there regularly it was quite well protected and had huge backup power generation too.

I imagine the new BMO facility in south Barrie is 1st rate.

reply to DKS
Dear Valued Customer,

As you are aware, over the weekend the 8th floor of PEER 1’s Toronto data center located at 151 Front Street experienced a fire with a UPS cabinet that set off a chain of events that resulted in power outages in multiple suites and network outages in others. I have detailed the events below.

PEER 1 is extremely thankful that no one was injured during the fire. The fire itself was contained within a single UPS cabinet and the Toronto fire department was able to extinguish it with the use of dry chemicals. No water was used, which significantly reduced the damage and aided in our ability to restore services to all data centers as quickly as possible.

I was personally involved in PEER 1’s response to the situation, and would like to take a moment to apologize to you for the information contained within our first few email alerts regarding the outage. When we were first organizing the team, we received an early report from our on-site staff that all power was out at 151 Front Street. That was not accurate. The fire caused a power outage in PEER 1’s suite 802. The local fire department cut power to PEER 1’s suite 709 as a preventative and safety measure to contain and extinguish the fire. The network gear providing connectivity for Suite 816 are kept in a cage in Suite 802. As a result, Suite 816 suffered a network outage. Power in the rest of the building was unaffected throughout the incident. There is a fine balance to providing information during a crisis, and we do our best to make sure updates to our customers are, quick, transparent, and accurate. In no way would we try to misrepresent the situation.

PEER 1 manages all of the equipment it provides to customers with safety and performance in mind. The UPS mentioned above has been in service for approximately four years. Its last service inspection was on July 2nd and it was only at 65% load. We do not yet have the root cause, but there was nothing in the unit’s maintenance record to indicate there was potential for a problem. Our service contractor and manufacturer are both involved in the investigation.

As part of the process to restore power quickly, we had to bypass the affected UPS. This means that suite 802 is running without UPS power until we can schedule delivery and replacement. We will expedite the process and keep customers in this suite up-to-date with the latest information on the timeline.

Outage Details:
(Timelines represent impact to “most” clients. Some specific network devices failed due to power loss and restore took longer.)
· Suite 802 – Power and network outage. Start 23:32PT End 03:10PT
· Suite 816 – Network outage. Start 23:32PT End 03:10PT
· Suite 709 – Power and network outage. Start 23:32PT End 00:50PT
· Suite 402 – Limited network outage (some latency/packet loss issues). Start 23:32PT End 00:50PT
· Firewall Clients. Network outage. Start 23:32PT End 03:10PT

Ted Smith
SVP, Operations

PEER 1

Beat me to it. Just saw that letter this morning.

reply to DKS
151 was not built as a carrier hotel; rather it's was built as more traditional office space (initially as a teletype office for the railway) - and has been converted over time.

There are some very good rules in place, regarding fire stopping, cable routing, double-walled pipe for fuel lines, etc - and the building operators are VERY active in ensuring the rules are followed.

Batteries and UPS's have traditionally been a weak spot in network infrastructure... Anyone else remember the Simcoe St "event" back in 98?

Simcoe St.... yeah...

I was at the convention center that day (Comdex) and everyone's Internet, cell phones and even the pay phones just halted. A bit of kaos because everyone starting buzzing about this and at the same time the fire trucks and police cars were going by the front of the center... it set off a bit of a "oh my god, end of the world" for some folks.....
--
Nexicom High Speed Internet - »www.nexicom.net/

reply to LazMan

reply to pstewart