by tony » Mon Jul 14, 2008 11:32 pm
Here is the Timeline from our Colocenter:
• 9:00 AM - Fire started in a manhole located in the 500 block of Richards Street, Vancouver, BC.
• 10:00 AM - Utility power to most of downtown was lost due to a subsequent transformer explosion. Harbour Centre’s main building generator, Gen No.7 started and provided power to the Data Centers. No clients were affected.
• 10:20 AM – Gen No. 7 failed due to overheating as the result of a failure in the generator’s cooling system. Power went with UPS systems now running on battery.
• 10:40 AM - First alarm of batteries failing in UPS systems.
• 10:45 AM - 11:30 AM - Clients lost power due to UPS batteries running out.
• 12:30 PM – Gen No.7 is restored and providing power to the Data Centers.
• 12:30 PM - 12:45 PM – Clients’ servers are back up and running.
• 1:00 PM – Gen No.7 fails a second time due to overheating as the result of a failure in the generator’s cooling system. Clients’ lost power within minutes due to UPS system batteries being low.
• 3:00 PM - Gen No.7 is restored and providing power to the Data Centers. PEER 1 did not turn client servers on immediately; waited for power to first stabilize.
• 3:30 PM - Gen No.7 fails a third time due to overheating as the result of a failure in the generator’s cooling system.
• 3:40 PM – Gen No.7 is restored and providing power to the Data Centers.
• 4:00 PM - Power restored to UPS systems, but not to client servers; waited for power to first stabilize.
• 4:20 PM - ALL UPS systems now online and power to client servers restored.
• 5:55 PM - Utility power restored to the Data Center and Gen No.7 shut down.