SDSC East Datacenter Cooling Outage – 03:45, 30 Sept 2016

[update 14:36] Operations has added both standby cooling units and additional alerting capabilities. UCSD FM has modified controller code to facilitate a faster recovery in the event of a problem.

[update 14:08] UCSD FM reflashed the controllers remotely at 13:50-13:58. We apologize that no advanced notice was able to be provided.

[update 13:16] UCSD Facilities Management is planning to reprogram the controller for air handlers and chilled water. Another short outage while the controller is swapped and we will put a status update before that occurs.

[update 12:21] All air handlers and chilled water pumps are online.

[update 12:07] Campus facilities management is on site investigating, but we have experienced two additional cooling outages, 10:40-11:15 and 11:55-12:02

[update 08:41] The outage start time has been revised to 03:45 based on additional logging and metrics collected.

At approximately 05:00 on the morning of 30 Sept 2016, the east datacenter air handlers went offline. Cooling was restored around 06:00. We will continue to update this page with more information.

Leave a Reply