The Long Road to Redundancy
By Doug Wagel, Manager, Administration & Facilities
Two years ago, we started a multi-faceted project to expand MDT’s Southfield Data Center (DC). We wanted to increase our DC rack space, add additional cooling to the DC, add Automatic Transfer Switches (ATS) and, most importantly, add generator redundancy.
All of this occurred as we were negotiating a new lease for our Southfield space and moving the Southfield staff into our corporate headquarters in Farmington Hills. Both of these big events now seem a distant memory, as we shortly thereafter entered the final stages of implementing the new 400 KW natural gas generator (for the redundancy phase of our project).
This project had its ups, downs, and delays. The longest delay was the move of the fiber cabling and in-ground termination box as this was all located under where the generator enclosure would be built.
When the work began, the fiber providers stated that there was ample cable slack in the ground and that the underground termination box could be picked up and moved away from the new generator location. However when spring came, some-thing had changed… One of the vendors declared they would need to reposition the fiber cabling path and move the underground termination box to the new location. This created another delay in this phase of the project because it meant more time spent digging and pulling new fiber and installing a new termination box. Once finalized, we would then be able to schedule the cutover to the new fiber. This was done as a scheduled maintenance outage for some of our clients whose services run through these connections. I could go on and on with more of the hurdles we had to overcome during this project, but that is not what the story is about.
Let’s fast forward to the testing phase – after the generator was in place – to make sure everything was wired and con-figured correctly.
Moving a new piece of environmental equipment to a live environment is a process that must be documented to ensure all aspects of testing are substantiated and completed. When the natural gas generator was set into place and wired, we scheduled a date to run the startup testing script. The task of completing the steps in the generator startup script, as well as the actual testing and vendor certification, takes a team of technicians who are specialists in their chosen field. For this event, the team of specialized technicians included: Access, Inc. (DC expansion architect), Vertiv (UPS/battery), ASCO (ATS), and Centerline Electric (power to the DC) and, of course, our MDT Facilities Department staff were on hand as well.
When we created the testing script, we considered many factors: how we wanted the new generator to work as the primary unit (first to run) and how the diesel generator would be the backup unit, along with the generators’ relationship to other environmental equipment within our Data Center.
We tested the startup, running and transfer of the generators from the primary natural gas generator with and without a Data Center load (or DC power usage). This step was of the utmost importance to ensure the coverage of generator redundancy. There were many things to consider when we developed the script/process and certification criteria. One of the most surprising factors was how fast the backup diesel generator would fire up to speed compared to the primary natural gas generator (which created circumstances we needed to address in order to achieve our planned sequence of events).
We successfully ran through the testing script without a hitch and every step worked as intended. When we switched back and forth between generators and ATS, battery power (UPS) levels stayed consistent. The generator redundancy project (Natural Gas and Diesel) is now complete for our Southfield Data Center.