Section 1: Incident Report
On 5 February 2021, unidentified hackers remotely accessed the water treatment plant of Oldsmar, a small Florida city of 15,000 people in the Tampa Bay area, and briefly changed the levels of lye sodium hydroxide (NaOH) in the drinking water. The quantity of NaOH, a chemical often used in drinking water treatment processes to adjust pH and alkalinity that can be dangerous when present in excessive quantities, was altered from 100 to 11,100 parts per million enough to seriously sicken residents had it reached homes. The attack was caught before it could cause public harm.
The plant’s computer system allows for remote access so that authorized users can troubleshoot problems from other locations. An operator machine had the vulnerable remote access software package TeamViewer installed and accessible to the Internet, which provided the means to manipulate control set points for the dosage of NaOH in the water.
Water systems are engineered with many safeguards to keep parameters within acceptable limits including process instrumentation that continuously monitors water quality parameters (e.g. pH) which provide real-time alerts when those parameters go outside acceptable limits; these critical parameters are typically monitored at multiple points throughout the treatment process and in the transmission and distribution systems. In addition to this instrumentation, trained and licensed drinking water treatment operators are also employed on site. One such operator on duty noticed that someone was controlling his computer and realized that different programs were opening and that the level of lye had been changed. The employee identified the incident and restored the normal operating parameters fast enough so that pH monitoring alarms did not detect a level beyond acceptable parameters. Had the operator not observed the attacker actively manipulating the screen, it is possible that several other mechanisms in the water treatment plant control and monitoring system could have alerted plant staff to the changing lye levels. It is also entirely possible, however, that any further delay in correcting the chemical leak could have resulted in injury or death as a result of drinking unsafe water.
Had the treatment plant safety systems been completely automated, the attack could have potentially had a disastrous effect. Many smaller water treatment plants in the United States and elsewhere do not maintain a constant supervisory staff, and these installations are even less likely to have robust cybersecurity defences in place to ward off attacks of this nature.
Between budget cuts and pandemic conditions promoting the transition to remote working, water plants – which are not known for their security resources to begin with – have become even more vulnerable. In many water facilities, as well as other local utilities and industries, the need to keep systems up and running in the midst of pandemic has resulted in the mass rollout of remote access technologies often prioritizing business continuity over safety, to the detriment of cybersecurity.
Section 2: Targeting US Water Treatment Systems
A Bluefield Research paper published in April 2020 highlighted how remote monitoring and digital asset management were already widespread across the water sector prior to the pandemic: “79% of US community water systems have SCADA systems fully implemented, while just 21% have network optimisation solutions in place that facilitate remote management.” In 2020, the number of vulnerabilities disclosed increased by 25% compared to the previous year, according to a report by industrial cybersecurity firm Claroty – and the trend is expected to continue even after the pandemic. In terms of industries, Claroty stated that the most at risk in the second half of 2020 were critical manufacturing (194 vulnerabilities), energy (186), water and wastewater (111). Water and waste water facilities operate with slimmer margins than other CNI facilities and thus struggle to invest in effective cyber security compared with other industrial sectors, as highlighted by a recent study of 15 cybersecurity incidents in the water supply industry. A 2020 survey found that only 19% of US water utilities found their profits to be sufficiently solid to cover the cost of existing services, let alone pursing infrastructure upgrades.
It is possible that the Oldsmar hack will now spur regulatory or policy scrutiny in water security from the US government. The only federal law that applies to the cybersecurity of water treatment facilities in the country is America’s Water Infrastructure Act of 2018, which requires water systems serving more than 3,300 people to have emergency response plans that account for cyber incidents. There is no federal law that requires such facilities to report cybersecurity incidents, such as the one in Oldsmar and the sector remains woefully under-regulated compared to electricity or nuclear power infrastructure, which have more stringent cybersecurity standards. Still, increasing regulations without providing water utilities more resources would most likely only exacerbate the sector’s vulnerabilities. The Oldsmar incident has illustrated the systemic weakness that characterizes the sector: the (largely under-funded) 150,000+ public water systems in the US form a heterogenous patchwork, less uniform in technology and security measures compared to other developed countries. All considered, in the industry – to use Lesley Carhart’s words, principal incident responder specializing in industrial control systems at Dragos Security – “we were all expecting this to happen”. Below, a summary of the situation characterizing much of the US water treatment plant system:
· There are approximately 151,000 public water systems in the US, of which about 54,000 are Community Water Systems – which serve the majority of the country’s population.
· Most serve less than 50,000 residents, with many serving just a few thousand.
· Virtually all of them rely on some type of remote access to monitor and access these facilities.
· Many of them are unattended, underfunded, and without a 24/7 surveillance to either IT or OT operations.
· Many facilities have not separated operational technology from safety systems that might detect and alert on intrusions or potentially dangerous changes.
Experts are urging water facilities to take a closer look at their network connections set up through software vendors and restrict remote access when possible. Oldsmar stands as a warning about how dependency on these programs can leave the door open for hackers to undermine remote surveillance tactics.
Endemic CNI vulnerabilities
While the full details are still emerging, early investigations suggests that the hackers gained remote access to the plant’s systems through a weakly protected software application called TeamViewer, a software package used by a large number of organisations to easily manage remote access to IT systems. The Oldsmar’s water treatment plant had stopped using TeamViewer six months ago, but left it installed. Moreover, TeamViewer is neither the only nor the least safe software used for remote access in ICS. Prior to the pandemic, using these tools to access critical process controls might be described as negligence in terms of security. However, today, there are legitimate reasons for needing remote access and monitoring during the pandemic – and there are ways to make TeamViewer and similar apps more secure if they are the only available option. Unfortunately, though, many municipalities simply do not have the fiscal or time budget to be more circumspect of this kind of software.
Overall, the three main vulnerabilities most often mentioned in connection to the Oldsmar cyber sabotage are: password sharing (a matter of cyber hygiene), use of beyond-end-of-life software (a patching and updating issue), and the use of TeamViewer for remote access and control system. All these vulnerabilities were present in Oldsmar and are common features across all CNI sectors in the US and elsewhere.
Section 3: Contextual Framework – Critical Water Infrastructure Vulnerabilities
Cyber attacks on critical national infrastructure (CNI) are not new. Since the Stuxnet attack on Iran’s Natanz nuclear facility in 2007, CNI installations and processes have become frequent targets for hackers, and the threat should one, such as a power grid or major water supply, be exploited could be catastrophic. Lesser known, less successfully disruptive cyber operations against water treatment or waste plants have occurred for more than two decades, often flying under the radar. Iran, for instance, has long been linked to several operations against water systems.
Cyber actors affiliated with Iran’s Islamic Revolutionary Guard Corps were allegedly behind two separate failed attempts to hack the Israel’s water treatment plants and agricultural irrigation systems to alter water chlorine levels between April and July 2020, according to Israeli officials. Israel retaliated in kind, with a disruptive cyberattack on an Iranian port. Just as in Oldsmar, these attacks took advantage of poorly secured remote access software to gain access to industrial control systems (ICS). It is also not the first time that hackers targeted infrastructure in a small town. In 2013, a hacker group in Iran claimed responsibility for gaining remote access to SCADA systems at a small dam in upstate New York. Non-state actors have also dabbled in similar types of activity, as in 2016, when a hacktivist group gained access to SCADA systems at an unnamed US water utility and manipulated the flow of chemicals, although without significant effect.
A cyber attack on a water treatment facility leading to the contamination of water supplies has long been feared by cybersecurity experts and, although no harm was done to public health as the concerned facility had safety checks in place, the level of access obtained by the attacker in the Oldsmar incident is a reminder that the growing digitalisation of CNI has rendered it vulnerable as never before. Across the US, water plant operators, plus those at dams and oil and gas pipelines, have accelerated the transformation to digital systems that allow engineers and contractors to monitor temperature, pressure and chemical levels from remote workstations. These features, while heightening the convenience of surveillance and repair, also increase the exposure of such systems to malicious cyber actors seeking remote access to enact harm to public health.
Section 4: Attack Methodology
Even among facilities that actively monitor their networks, only a subset run network anomaly detections regularly. Had such detections been run in the Oldsmar facility, however, no network anomalies (from behind the firewall) would have been generated due to the attacker leveraging an officially sanctioned remote access pathway (i.e. TeamViewer). There is a chance that, had the attacker been accessing the system from outside the area, an anomaly would have been generated – If the attacker was in close proximity to the plant, however, the connection may have masqueraded as the system viewed them as a local employee working at home. It is also possible that the attacker breached a completely unrelated device, like a personal tablet, which was connected to the same Wi-Fi network used by a remote-work engineer responsible for managing the plant. The perpetrator(s) stole the credentials for TeamViewer, and subsequently used the credentials to log into the facility and attempt a sabotage.
Due to the Human-Machine Interface (HMI) functionality existing on the same machine as TeamViewer access, no network anomalies would have been present and the attacker(s) would have already had all the necessary access to the system to manipulate lye levels. There was no need for reconnaissance activities or lateral movement within the system – which means network anomaly detection technologies are not a good fit to mitigate this specific case. The Oldsmar attack happened within the stream of data used to monitor and control the process. The attacker used a legitimate HMI to send a legitimate packet with a legitimate payload which increased the levels of lye in the water. An alert from the FBI, CISA, EPA and Multi-State Information Sharing and Analysis Center on the attack stipulates that “all computers used by water plant personnel were connected to the SCADA system and used the 32-bit version of the Windows 7 operating system,” for which Microsoft ended support in January of last year. “Further, all computers shared the same password for remote access and appeared to be connected directly to the Internet without any type of firewall protection installed.” Moreover, email addresses and passwords with the domains ci.oldsmar.fl.us and myoldsmar.com had surfaced days before the breach, with credentials belonging to Oldsmar city employees included in the leak as revealed by Cyber News and confirmed by Allan Liska, a senior security architect at Recorded Future who tracks dark web activity.
Likely attribution
The Oldsmar incident remains unattributed and, due to the basic nature of the tactics employed, the perpetrator(s) could be anyone ranging from a disgruntled insider with knowledge of the plant’s TeamViewer software, to a nation state actor. Yet, the lack of precision and the apparent failure of the attacker to conceal his presence by removing the plant operators visibility and control suggest that this is the work of an opportunistic, unskilled hacker rather than a strategic, sophisticated threat actor. Based on the information available at this moment, this attack seems to lack any sophistication that could trigger more serious additional action. Furthermore, the attacker increased the levels of sodium hydroxide by a significant amount, typically monitored by automated systems, which likely suggests that the threat actor didn’t possess any specific knowledge of the water treatment process. Sophistication is not a prerequisite for conducting this sort of attack, and this is exactly why the Oldsmar hack is so worrying. It does not take a particularly skilled actor to successfully breach a system such as the one used in Florida’s water utility network.
It is worth signalling a possible lead indicated by the cybersecurity firm Intel 471 in the aftermath of the attack with regard to its perpetrators, pointing toward cyber criminals. Researchers from the firm re-examined an incident they had reported last spring in which likely Iranian hackers were attempting to sell access to a US hydroelectric power plant. “Further investigation found that what the actor was actually advertising was access to a water treatment plant in Florida, via a virtual network computing permission that granted system access to a ‘Groundwater Recovery & Treatment System,'” reads a blog post from the cybersecurity firm. “Additionally, one screenshot showed levels and controls for a sodium hydroxide pump.” Intel 471 could not confirm or deny links between those hackers and the Oldsmar attack.
If water treatment facilities in the US as well as in the rest of the world have thus far not been a major target of profit-minded cybercriminal groups, it is certainly not because of particularly deterring security mechanisms in place. Rather, it is probably because most of these facilities have very little worth stealing and usually no resources for paying extortionists. This, of course, is not a deterrent for terrorist organizations or aggressive nation-state actors, should the perpetrators’ main goal be to cause harm to as many people as possible with the least possible costs and risk of detection.
Section 5: Further Implications
While it is perhaps inevitable that state actors with enough time and resources will be able to disrupt CNI, the inability to prevent more basic attacks from non-state actors such as insiders, hacktivists and criminals – and potentially even more dangerous groups (i.e. cyber terrorist organizations) – signals that this will continue to be a persistent, and growing, threat.
And this is not just a US problem. All countries have vulnerable Industrial Control Systems (ICS) tied to critical infrastructure. In the US, the attack sparked a wave of advisory statements from local, state and federal authorities. The US Water Information Sharing and Analysis Center, a clearing house for cyberthreats in the sector, has circulated an advisory for securing such remote connections in the aftermath of the Oldsmar attack. An alert to public water suppliers that Massachusetts’ Department of Environmental Protection released earlier this week referenced another report from the FBI, DHS, Secret Service and the Pinellas County Sheriff’s Office which provided more detail that referenced the plant’s SCADA systems. The National Security Agency (NSA) and the Cybersecurity & Infrastructure Security Agency (CISA) have issued an alert with several recommended actions to reduce external exposure and minimize risks to OT and ICS. All these alerts offer a sound and generally applicable advice on digital hygiene and best security practices. Yet, this is actually not entirely reassuring as it further suggests the extent of the security challenge and the very large number of potentially vulnerable attack surfaces.
Internationally, this case will generate dialogue around the appropriate standards and regulation for CNI, illustrating that the cyber resilience of CNI is only as strong as its weakest link. In the UK, work has been done in the telecommunications and agricultural sectors to provide cybersecurity guidelines to operators, but such guidance is still missing in other sectors. The situation is no different in the rest of Europe, says Marcin Dudek, a control systems security researcher at CERT Polska, the computer emergency response team which handles cyber incident reporting in Poland. This is mostly due to a mix of poor awareness and underfunding, which is worse in parts of Europe than in the US as a whole. Marcin says TeamViewer would actually be an improvement over the types of remote access systems he commonly finds in his own research, which involves HMI systems designed to be used via a publicly-facing website.
Finally, though the Oldsmar hack was unsuccessful, the incident is a disquieting reminder of the much worse scenario that could occur if more advanced actors launched a similar attack, such as those responsible for breaching Israel’s water treatment systems last summer. Remote access is unavoidable today – most installations are unmanned – and this will persist beyond the pandemic. In the case of very small water or sewage treatment plants, there most likely will be no people inside checking that nothing goes wrong – which is ultimately what prevented the Oldsmar hacker(s) from poisoning 15,000 people.