US city enlists ‘threat hunters’ after ransomware attack

Since it was hit by a Conti strain ransomware attack earlier this year, the City of Gary in Indiana has appointed a private sector partner as its ‘cybersecurity department’ and doubled down on training. The city’s Chief Innovation Officer shares what he learned from the experience.

On a morning in April 2021, Lloyd Keith, the City of Gary’s Chief Innovation Officer, first knew there was a problem when several users began reporting connection errors. Endpoint security was “acting goofy” he says. The network administrator then found what every city increasingly dreads: a server message containing a ransomware notice with a demand for US$880,000 – a significant amount for the municipality which has a population of around 70,000 people.

Like many small cities, Gary didn’t have a dedicated cybersecurity department. We were “praying [a cyber-attack] would never happen to us,” Keith says, but knowing the risk was growing he had attended several conferences and seminars to inform strategies and checklists.

This paid off to an extent. The network administrator notified the FBI, the Multi-State Information Sharing and Analysis Center (MS-ISAC) and Homeland Security. They also identified that a remote desktop application was being used to encrypt files so shut the program down to prevent further proliferation. But by then, damage had already been done.

New cybersecurity department

Several critical citizen-facing systems were down for around two weeks.

“When customers would come to the city to do their various requests, pay various fees…we weren’t in a position to do that,” Keith says, noting the IT team had to help departments figure out temporary, workaround processes.

Fortuitously, though, the city had system back-ups in place – the online versions were corrupted but the offline Unitrends back-ups were still intact.

Before restoring systems, the city needed to make sure that the attacker wasn’t still in the network. UncommonX, a software-as-a-service (SaaS) cybersecurity provider, deployed its BOSS (Business Operations Security Suite) security platform and was able to map Gary’s digital environment and contain the ransomware within 72 hours. The vendor helped eradicate the malware while working on recovery and bringing the servers back online.

Gary has now signed a long-term contract with UncommonX for ongoing managed cybersecurity services, initially funded through American Rescue Plan Act (ARPA) money.

“Gary has an IT department, but we’ve never had a cybersecurity department,” Keith comments. “Now UncommonX is our 24/7 cybersecurity department.”

He said he had considered employing a Chief Information Security Officer (CISO) directly as some larger cities do, but that: “I can hire a person with all of the certificates in cybersecurity from here to the end of the alphabet, but I still need a tool for monitoring what’s happening actively on the network.”

Threat hunting

The company also provides support such as specialist public relations, which can be crucial to cities when they’re in the eye of a cybersecurity storm.

“Some of these things are not necessarily at the ready within mid-size organisations,” says Patrick Hayes, Chief Security Officer, UncommonX. “We have a number of professionals that come from varying backgrounds, from enterprise security architects all the way through to threat hunters and incident response folks.”

UncommonX monitors and proactively searches for threats on multiple fronts, using commercial and open-source tools.

“We also have our own feed from the dark web and from a number of honeypots that we have placed globally,” Hayes says. Each customer is given a unique risk profile.

“It’s hard to predict where the next attack is going to be,” says Hayes. “It is easy to predict, though, where attackers are going to come — they’re coming for the money, and they’re going to pick soft targets that they think will pay.”

Cities are seen as such ‘soft targets’ due to the critical services they provide and the fact they often lack the resources to build sophisticated cybersecurity operations like large private companies.

According to The Washington Post, more than 400 ransomware attacks have hit city and county governments in the United States since 2016. Examples include Atlanta, New Orleans, Baltimore and, earlier this year, Tulsa.

Because they had the back-ups in place, Gary did not pay the ransomware but several cities have done so, using insurance to cover most of the costs and calculating that on balance paying up is cheaper and faster in the long run.

But Keith notes: “You’d have to trust a crook,” and engaging in this way could leave cities open to repeat attacks.

Some states are considering banning local governments from paying hackers.

Human element

Since the incident, Gary has also doubled down on the human aspects of cybersecurity.

Some forensics were destroyed in the city’s early efforts to shut down the attack, but investigations did show where and when it began and that it was launched via email.

New tools now detect and block malicious emails from getting to staff to prevent people clicking in error.

Gary already had a training programme from KnowBe4 in place but it was not enforced as strictly as it could have been.

All employees now need to complete an education campaign once a quarter.

“I now have my ‘sheriff’s badge’ on,” says Keith. “And the administration is backing me 100 percent on this. I tell staff – complete your training by the deadline or you will lose your access. We are making believers of the staff and employees that we mean business.”

Keith’s message for his peers in other cities echoes that of fellow CIOs who have been hit by a cybersecurity attack: “It’s going to happen to you,” he says.

He adds that: “I am open to helping any municipality that wants to learn from our experience. We are available.”

Image:  Yanawut Suntornkij | Dreamstime.com