In the early hours of a Friday in December 2019, the team monitoring the computer network handling governmental operations for New Orleans noticed something suspicious.
"At first, it didn't seem like anything too worrying," Kim Walker LaGrue, chief information officer for the City of New Orleans, told VOA. "It looked like a user with the wrong credentials was trying to access our data center, but that could have been one of our administrators doing some early morning work. We didn't think it was anything malicious."
That was at 5 a.m. Within a few hours, similar activity was affecting multiple users, and the service desk was called to investigate.
From there, it didn't take long for LaGrue's team to figure out what was going on.
"We identified a ransomware attack was being launched against the city," she said.
Ransomware is malicious software that is planted in a computer network to seek out sensitive data. Once that information is located, hackers threaten to either publish the data or prevent it from being used until a ransom is paid.
And this type of attack was not unfamiliar to New Orleans' City Hall. A month earlier, Louisiana — the state in which New Orleans is located — had been the target of another ransomware attack.
In fact, in 2019, 106 city and county governments were targets of ransomware attacks. And the problem seems to only be getting worse. Last year, the United States suffered more than 65,000 similar attacks. Recent high-profile ransomware hacks have targeted a U.S. oil pipeline and a major meat processing outfit.
"Am I surprised? Not at all," explained Vince Gremillion, owner and founder of Restech Information Services, a cybersecurity firm based in the New Orleans area.
"Ransomware attacks can be extremely profitable for the attackers, and the victims are often ill-equipped to stop them. If I'm surprised by anything, it's that this doesn't happen even more often."
Attack started with phishing
Phishing is the practice of sending emails pretending to be from a reputable company in order to get people to reveal personal information such as passwords. This is often done by inducing victims to click a link in the email.
This is how the New Orleans attack began.
"It's something that gets ramped up over the course of several days," LaGrue explained. "Days before we detected anything, one of our employees on the network clicked on a link they thought was legitimate."
Gremillion said email is just one way criminals attempt to hack into a system.
"You might think I'm exaggerating, but every time a new internet connection is established, that connection is being probed for vulnerabilities," he said. "It's all automated, and they're just looking for weak passwords they can take advantage of. Unfortunately, there are many weak passwords."
While Gremillion says attacks can come from anyone and anywhere, he and experts like Andrew Wolfe, Loyola University's cybersecurity degree program director, say many come from countries such as Iran, North Korea, former members of the Soviet Union and even China.
"Attacks aren't coming directly from a foreign government, and it's not just some nasty guy in a Siberian hut," Wolfe told VOA. "There's a real blurring of the lines between individual hackers and governments."
U.S. authorities have traced several recent high-profile ransomware attacks to Russia. Russian President Vladimir Putin has not denied that ransomware attacks originate in his country. But he has steadfastly denied any Russian government involvement or coordination with hackers.
Wolfe said an entire industry has developed around these attacks.
"Some people are focused on developing the ransomware, while others are executing the attacks," he said. "Some are creating new and better ways to collect ransom and launder the money while others are providing actual customer service to the criminals. A whole dark supply chain exists now."
Prime targets
"When the employee clicked on the malicious link, it allowed attackers access to our network," LaGrue explained.
She said the attackers began uninstalling antivirus software that could detect attacks. They meticulously removed layers of security protecting the system.
Gremillion said the speed at which hackers can gain access to a vulnerable system is staggering.
"I've seen instances where Russian hackers can gain administration-level access to a system in 20 minutes," he said. "It's so fast, and that was a couple of years ago. It's probably even faster now."
After a criminal has that level of access, they set their sights on the organization's confidential data so they can use it to extract a ransom.
Local governments are frequently targeted for ransomware attacks, and Wolfe said there are several reasons for that.
"One is that they really need this data," Wolfe said." Cities do so many essential tasks — public health, public safety, taxes and so much more — that they can't afford to lose access to that data. Attackers know this, but they also know local governments don't have a great reputation for having the most competent IT staff when it comes to system security."
Lax security protecting valuable data -- plus the increased possibility that insurance companies will agree to pay the ransom on behalf of their local government clients -- are all reasons attackers focus on cities like New Orleans.
A changing situation
"Now, if we're being fair, the way New Orleans handled its ransomware attack was a near-best-case scenario," Wolfe said.
By the time system administrators realized what was going on, the criminal hackers were already well on their way to gaining control of the data they would need to demand a ransom. That's when city officials made a decision that experts celebrated as especially savvy.
"Mayor (LaToya) Cantrell made a declaration of emergency, and we instructed all our employees to shut down and unplug their computers as well as to disconnect from the internet," LaGrue said.
The mass shutdown brought many of the functions of city government to a temporary halt, but it also made it impossible for the hackers to continue their attack.
“I don’t want to understate how difficult and burdensome it was for our city’s agencies to do so much of their work manually,” LaGrue said, “but cyberattacks have become more frequent and we knew we had to prepare. Once we identified the issue, we executed our plan.”
But even a well-executed plan proved costly. LaGrue said recovering from even the unsuccessful attack they sustained had a price tag of approximately $5.2 million.
That's well under the $17 million Atlanta, Georgia, spent after the city suffered a ransomware attack in 2018, and less than the $18.2 million recovery of Baltimore, Maryland, in 2019. Still, the cost for New Orleans was substantial.
"Most of that was replacing inventory," LaGrue said. "We had to replace about 600 devices — or nearly 25% of our inventory — to ensure all of our computers were clean of the virus."
And it wasn't just a financial cost. In addition to cleaning those devices, the city assessed and cleaned more than 3,000 computers and 200 virtual servers. It also built new storage and security infrastructures. Recovery stretched on for months.
During that time, according to Wolfe, the city had to pause or delay basic municipal functions.
"They were able to still carry out essential functions like public safety," he said. "But there were stretches where things like, for example, paying parking tickets or getting a building permit were really difficult to do."
Prioritizing cybersecurity
LaGrue acknowledged there were some priorities the city had to put on hold while it recovered from the attack, but she felt City Hall is stronger having gone through this process.
"It allowed us to improve our cybersecurity infrastructure in a way we probably wouldn't have if it weren't for the attack," she said. "For example, the improvements made it possible for us to allow our employees to securely work from home far quicker than we otherwise could have."
The city also better understands the importance of ongoing cybersecurity training for its employees.
"If we have 4,000 employees, that means we have 4,000 potential cybersecurity vulnerabilities," LaGrue said. "We need to make them better aware of the threats they are likely to encounter while online."
Experts such as Gremillion say they are happy to see improvements but would like to see organizations secure their network before a crisis.
"Half of all internet traffic is malicious, but IT departments don't seem to act like that's the case," he said. "The priority seems to be to strip away things — such as making you wait to log in if you type your password incorrectly several times. IT staff get rid of those things because it's inconvenient for employees who want to get online and send emails, but those 'inconveniences' keep your network safe."
Gremillion believes security measures like this are essential to avoid painful and costly attacks down the road. They range from the complex security layers that can take a day or two to implement to the very simple.
"Don't get me started on passwords. If you're still using your initials or 'password' as your password, what are you doing? Cybercriminals are getting more advanced. The good news is the systems used to repel those criminals are getting more advanced as well. So we need to educate ourselves, and we need to do better, because the consequences if we don't are so much worse."
