Canadian Webhosting Forums

[tony]

There was a explosion in one of the under ground power lines which caused a blackout to the colocation center. We had problems bringing up the generators this morning but they are online now.

BC Hydro still are not able to get in the affected area to do repairs. Their current ETA is sometime tomorrow morning.

We will have extra staff on hand during the night shift incase there are any problems with the generators.

I will post updates as soon I get them. I will also try to post some pictures as well.

[tony]

Here is the Timeline from our Colocenter:

• 9:00 AM - Fire started in a manhole located in the 500 block of Richards Street, Vancouver, BC.

• 10:00 AM - Utility power to most of downtown was lost due to a subsequent transformer explosion. Harbour Centre’s main building generator, Gen No.7 started and provided power to the Data Centers. No clients were affected.

• 10:20 AM – Gen No. 7 failed due to overheating as the result of a failure in the generator’s cooling system. Power went with UPS systems now running on battery.

• 10:40 AM - First alarm of batteries failing in UPS systems.

• 10:45 AM - 11:30 AM - Clients lost power due to UPS batteries running out.

• 12:30 PM – Gen No.7 is restored and providing power to the Data Centers.

• 12:30 PM - 12:45 PM – Clients’ servers are back up and running.

• 1:00 PM – Gen No.7 fails a second time due to overheating as the result of a failure in the generator’s cooling system. Clients’ lost power within minutes due to UPS system batteries being low.

• 3:00 PM - Gen No.7 is restored and providing power to the Data Centers. PEER 1 did not turn client servers on immediately; waited for power to first stabilize.

• 3:30 PM - Gen No.7 fails a third time due to overheating as the result of a failure in the generator’s cooling system.

• 3:40 PM – Gen No.7 is restored and providing power to the Data Centers.

• 4:00 PM - Power restored to UPS systems, but not to client servers; waited for power to first stabilize.

• 4:20 PM - ALL UPS systems now online and power to client servers restored.

• 5:55 PM - Utility power restored to the Data Center and Gen No.7 shut down.