PIERRE, S.D. (South Dakota Searchlight) — The network outage that downed a host of South Dakota state government systems in August was caused by mislabeled wiring in a room full of routers at the Department of Transportation building in Pierre, lawmakers learned Thursday.
The mislabeling ultimately led to the torching of a critical, $80,000 network traffic controller switch and its backup. State computer systems, including those used by the courts, local registers of deeds, auditors and treasurers offices and, for a short time, the state radio system, were down for nearly two business days.
Two leaders of the team that manages the data center in Pierre told members of the Legislature’s Government Operations and Audit Committee that the state plans to relabel the wiring correctly, update its redundancy plans and work with state agencies on backup power schemes for the most critical state services.
Given the disruptions this summer, lawmakers said, the state’s citizens need to be sure that work will get done.
“On the farm, we always have extra fuses. We always have extra switches. We always have extra everything,” said Rep. Julie Auch, R-Yankton. “Are you going to have extra switches on hand or extra parts on hand so this doesn’t happen again?”
Overpowered switches
The mislabeled wiring issue came to light after the scheduled weekend replacement of a piece of equipment called an uninterrupted power supply, according to Darin Seeley, commissioner of Human Resources and Administration.
As the system was being brought back online, a main breaker tripped several times, which is “not uncommon,” Seeley said.
The breaker’s job is to modulate the power coming into the data center from a transformer, reducing it to the level needed to operate the equipment inside.
The breaker was damaged so badly, however, that it eventually gave out entirely, causing the initial network outage.
The electricians who replaced that breaker turned off the main power supply to the room, Seeley said. Or at least they thought they did. Assuming that doing so would cut off power to everything else, they got to work. Unbeknownst to them, Seeley said, the two critical network switches were connected to a different breaker.
That breaker was designed to be connected to the main breaker and was labeled to say so, Bureau of Information and Telecommunications Commissioner Mark Wixon said.
“Turning off the main breaker should have turned off the power,” to the switches, Wickson said. “That wasn’t the case.”
As a result, the switches took on 10 times the power they’re meant to and burned up, one after the other.
It took a day to get a new switch from Louisville, Wixon said. Bureau staff picked it up at the airport in Pierre and had it installed around eight hours later.
Redundancies, disaster planning
Seeley told lawmakers that the building with the mislabeled wires had been “updated numerous times” over the past 50 years.
Rep. Karla Lems, R-Canton, asked Seeley what’s been done “to make sure, ‘OK, we got everything labeled correctly.’”
He said the intent is to “try and redraw all of the lines” to make sure everything’s set up as designed, but that hasn’t happened.
“It has to be done when everything’s off, because we’re talking about some pretty significant power,” Seeley said, adding that such a scheduled outage needs to be done “planfully.”
Auch wanted to know if the bureau had considered adding a third $80,000 switch to its supply, given how impactful the situation was. But Wixon said such a move had never felt necessary.
“This is really kind of the first time this type of an event, where the routers were damaged, that’s happened in over 50 years,” Wixon said. “So we have high confidence that the lessons learned around this one are being applied.”
The situation has also sparked “strategic” conversations on resilience, Wixon said.
Sen. Mark Lapka, R-Leola, wanted to know if that thinking involved the contemplation of “geo-redundancy, to have a backup in another location.”
That’s part of the discussion on disaster management, Wixon said. But building out a separate infrastructure to create what he called a “high availability level” doesn’t come cheap.
“Going through that type of exercise really does drive to ‘what is the cost of high availability?’” he said.
That expense leads most companies to prioritize which systems need to be highly available, he said, and which can survive “a three-day outage.”
The bureau plans to work with other state agencies in the coming months to determine such a hierarchy. The bureau is already working with the Department of Public Safety on a cloud-based system to make sure state radio communications, which were impacted by the August incident for approximately five hours, are protected from an outage.
Comments