Peak 10 Data Center Incident - 11/9/13

Here is the latest update from our testing and analysis of what transpired on November 9th 2013 in Nashville. The Peak 10 and GE Engineering Teams have completed their review of the data captured in the lab over the past week and have isolated what they believe to be the issue. According to the findings, it appears that a software configuration setting (under a specific set of circumstances) does not respond the way it is designed. The result of this issue creates the conditions we saw in Nashville, and subsequently results in a disruption of power flow. The current move forward plan is to allow GE time to write the required firmware update, then retest those conditions, and ultimately deploy that code fix to the Nashville UPS units. The estimated time frame for completion of this work is 1-31-14. While that is obviously still several weeks away, we want to assure you that the issue only manifests itself when the UPS’s are manually brought out of bypass mode and back to normal operation. That is a series of events reserved for maintenance activity and therefore, we have no reason to believe there is any risk to the Nashville Data Center right now. We stopped all work on the UPS units that afternoon and will not be performing maintenance on them again until deployment of this firmware upgrade.

The new UPS firmware version (release scheduled for 1/31/2014) implements the following enhancements:

  • Improvement of load calculation.
  • Improve the OFF time of the bypass that is not contributing with its inverter.
  • No load OFF when we have stop operation for an overload. Switch the load on bypass in case we are below a defined phase shift between inverter and bypass.
  • Implement “permissive” commutation from bypass to inverter and allow only when all the inverters are available.

Thank You,

Andrew Mametz
Peak 10
Vice President of Service Delivery

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request