An initial RFI was completed for Pool Re and submitted on 15 February 2021. The following supplement provides additional insight into the attack and expands on evidence made public since the initial submission.
Section 1: Actor Identity and Motivation
Due to the seemingly unsophisticated nature of the attack, the most likely circumstance is that the Oldsmar hacker was an amateur actor, likely scrolling Shodan looking for compromised TeamViewer ports on the internet. More may be surmised on the identity of the hacker by analysing the dynamics behind the breach.
The relatively short time it took to enter the system, discover the functions necessary to control the levels of sodium hydroxide and enact those changes to potentially lethal doses, and then get out (altogether just around 3-5 minutes according to the investigators), may suggest the attacker’s familiarity with the environment, with its target or with similar SCADA software.
Were we to classify the actor behind the attack judging from the little tactical evidence available, we could do so under the Insider Attack category, as most likely only someone with access to multiple TeamViewer user accounts and an intimate knowledge of the target’s OS & SCADA software would be able to do what the hacker did in such a short time. At time of writing, too little is known to still give a definitive assessment, however.
Section 2: Counterfactuals
Several counterfactual analyses of this scenario persist. If the facility employee hadn’t noticed the intrusion, would have the attack succeeded? How does the plants safety management work? What other physical barriers are there and how secure or suitable are they? A joint advisory from the FBI, the CISA, the EPA, and the Multi-State Information Sharing and Analysis Center confirmed that the operator on duty during the attack reversed the change immediately. Still, according to local authorities, even if nobody had been watching, it would have taken 24 to 36 hours to affect the water supply and other plant safeguards would have detected and blocked changes of that magnitude anyway, thus maintaining the system’s integrity. In this sense, we may say Oldsmar is mostly a good-news case study. And yet, redundant controls do not diminish the severity of this cyber threat to public health. What if the human-machine interface (HMI) has been manipulated by malware to report ‘all clear’ during the attack as occurred, for instance, during the Stuxnet attack? Would the breach have been detected in time?
Detecting toxic water on its way to consumers requires sensors in the distribution network that must be connected in order to transmit data which then prompts preventive actions. Needless to say, anything that is connected can also be hacked and manipulated. Given that the perpetrator of the attack managed to break into a system that had reportedly been password-protected, assurances that automatic safety systems would have raised the alarm about dangerous sodium hydroxide levels before it entered the water supply are not fully convincing.
The fact that the water supply was not contaminated shouldn’t give the impression that safety measures at the Oldsmar’s water treatment facilities were adequate. The circumstances of the attack suggest that it could have been much worse. This is because, at any scale of sophistication, the Oldsmar attack had no real chance of causing damage.
There was no sign of an attempt, or even of the capability to attempt, to hide the intrusion during the attack (which occurred during far more sophisticated attacks on CNI in the past). The intruder attacked during a workday, not the middle of the night when monitoring would have been less rigorous, and made no effort to hide. Anybody looking at the screen would have seen that someone from a remote location had taken over the controls. Moreover, some analysts noticed how the level of sodium hydroxide was changed to such an extent that the system couldn’t have handled the supplies required to actually meet the input levels anyway.
More sophisticated attackers would have likely had little to no trouble navigating around the technology redundancies built into the system. In the case of Stuxnet, for instance, the attack destroyed the uranium enrichment centrifuges by tricking the system controlling them to think they were fine when in reality they were spinning out of control.
To summarise: with the kind of ‘amatorial’ actors behind this specific attack, safety measures in place at the plant would have prevented any plausible worst-case scenario. However, the Oldsmar attack remains alarming as a reminder of what could have happened had the perpetrators possessed greater motivation to cause real harm. Should more sophisticated actors decide to perpetrate a similar operation, security redundancies may not be enough to avoid physical damage.
Section 3: Cybersecurity at Similar Locations
Where digital systems integrated in the plant and critical water-treatment processes are automated, facilities require little human intervention in day-to-day operations. Thanks to remote-access technologies, many maintenance and monitoring activities are performed off-site by a third party. All this is great for efficiency, but naturally increases security risks. In a perfect world, these communications and operations would be walled off from internet-connected systems. Practical demands to monitor operations in real time and perform remote maintenance may expose vulnerable infrastructure to the other side of the firewall, which means more web-based hacks of operational technology systems occur as hacker can access to critical infrastructure facilities when corporate devices are inadvertently connected to the internet or a network administrator’s credentials are stolen.
However, the Oldsmar attack is concerning not just because there was an external access to the system, and that the plant’s security system was configured poorly and allowed someone in. Both happen far more frequently than publicly disclosed; these are the obvious control failures, and the easiest to fix as well. The real issue at play is arguably the lack of defence in depth with the HMI itself, which presented some fundamental flaws that practically opened the front door’ for a malicious insider.
As it has been widely reported, the attacker stole the credentials to TeamViewer and subsequently used them to log into the facility. Due to the HMI functionality existing on the same machine as TeamViewer access, no network anomalies would have been present. With attackers already having access to the process, there was no need for reconnaissance activities or lateral movement. Network anomaly detection technologies aren’t a good fit to mitigate this specific use case, just like system hardening operations, as vulnerabilities weren’t directly exploited. As a matter of fact, this attack highlights how no typical cyber security activity would have mitigated this risk, including vulnerability management, network segmentation, system hardening, identity and access management, firewalling, etc.
Oldsmar and many other OT-specific attacks have something in common: they all impacted the process through approved technology infrastructure. When the infrastructure is the target, attacks are detected more easily, but when the infrastructure itself is used to attack the process, things get more complicated, and more sophisticated tools are required to mitigate the threats. Zero-trust logic works in a way that, for the operator, nothing is to be trusted, so the process itself has to be monitored together with the parameters being sent from all the devices in the control room to the equipment. If a water facility has this level of cyber security, alarms would be triggered from the moment toxic substances values are set to anomalous numbers.
Most of facilities are usually only protected by wide monitoring which doesn’t penetrate deep into the industrial control protocols themselves. It would be ideal for CNI to monitor the industrial control process using artificial intelligence and anomaly detection to identify and stop anomalies within the process that aren’t a part of regular operations, so as to mitigate risks such as malicious/negligent insiders, as well as external attackers looking to commit an act of terrorism. By monitoring deep inside the process for anomalies, any attempt by attackers to ‘fool’ HMI systems would fail, as attackers would be limited by anomaly detection systems to send previously used safe values without raising red flags. However, this type of security system is far from the norm, especially among smaller facilities with lower budgets available for cybersecurity.
Although several issues were named in the initial reporting of the attack the Windows system version was outdated, TeamViewer passwords were shared among multiple (if not all) users, the plant was connected to the internet without a firewall these were perhaps not the most relevant aspects to consider in light of this attack. HMI design may be the ultimate culprit behind the relative simplicity of the breach, as outlined by Jeremy Morgan, Principal Solutions Engineer at Industrial Defender. Considering the layers of controls (with access control being the process of granting or denying specific requests from a user, program or process) one can typically assume during a 24-hour manned operations centre, the HMI is likely logged continuously and was during actual attack in Oldsmar.
In a best-case scenario, operators log out and in at shift changes but that is not necessarily the norm, since in many cases a single shared account is logged into every HMI at the OS level. One can also assume the HMI is set up with at least two levels: a default level, where screens can be seen but not manipulated and which should be the default authorization context; an operator level, where normal levels and inputs can be entered, and an engineer level where things can not only be manipulated, but also actual changes to the screen can be made or default boundaries can be overridden; finally, there is usually an administrator role that is used only for the most extreme cases. Only when action needs to be taken should the operator then elevate themselves into the actual operator level. For a systematic review of the State of Cyber-Security in Water Systems see this study.
In Oldsmar, it is likely that either proper role-based authorization was not implemented in this system at all, or that protective controls like the timed return to default level were not in use. Something as simple as validating all inputs would have prevented the attacker from being able to set the lye value so high. Or, had the operator panel had been designed with appropriate enforcement of access controls, requiring any authorization prior to modification of any kind let alone to dangerous levels, then this hacker would have been prevented from their attempted attack.
Section 4: Working From Home and CNI Security
Less than two weeks before the attack, the US Government Accountability Office released a report detailing concerns about gaps in the government’s cybersecurity approach across several sectors of CNI. The report concluded by recommending further communication between the CISA and the EPA, instructing the agencies to coordinate on how to close gaps in security at water and wastewater facilities. In this framework, remote work poses additional risks that warrant additional precautions. CISA had recommended changing default passwords, restricting network access, encrypting data, installing firewalls, maintaining antivirus software, file sharing with caution, and connecting using VPNs while, in its official alert following the Oldsmar hack, it gave recommendations to secure remote access software such as TeamViewer.
ICS vulnerabilities have been on the rise since the beginning of the pandemic as gaps in remote work expanded attack surfaces. Throughout the second half of 2020, 71% of industrial control system (ICS) vulnerabilities disclosed were remotely exploitable through network attack vectors, according to the second Biannual ICS Risk & Vulnerability Report released by Claroty. The report also revealed a 25% increase in ICS vulnerabilities disclosed compared to 2019, as well as a 33% increase from 1H 2020. During 2H 2020, 449 vulnerabilities affecting ICS products from 59 vendors were disclosed. Of those, 70% were assigned high or critical Common Vulnerability Scoring System (CVSS) scores, and 76% do not require authentication for exploitation.
The water and wastewater sectors saw the highest increase in vulnerabilities during 2H 2020:
· Critical manufacturing increased 15% from 2H 2019 and 66% from 2H 2018
· Energy increased 8% from 2H 2019 and 74% from 2H 2018
· Water and wastewater increased 54% from 2H 2019 and 63% from 2H 2018
· Commercial facilities increased 14% from 2H 2019 and 140% from 2H 2018
· Assessment of ICS vulnerabilities sees growth in third-party researchers
According to Claroty, two factors contributed to this spike in recent years: a heightened awareness of the risks posed by ICS vulnerabilities, and researchers and vendors increasingly focused on identifying and remediating security flaws as effectively and efficiently as possible. This growth does not necessarily indicate a negative trend, therefore. Third-party researchers many of whom were cybersecurity companies, were responsible for 61% of discoveries. This signals a change in focus to include ICS alongside IT security research, further evidence of the accelerated convergence between IT and OT. Among all third-party discoveries, 22 reported their first disclosures, a positive sign of growth in the ICS vulnerability research market.
The accelerated convergence of IT and OT networks due to digital transformation enhances the efficiency of ICS processes, but also increases the attack surface available to adversaries. Particularly nation-state actors are looking at the network perimeter to exploit given the strategic importance of CNI, but cybercriminals are also focusing on ICS processes as opportunities for financial gain. CNI will always need some degree of remote accessibility not just at times of global pandemic whether for operational efficiency or simply because of a lack of personnel; they cannot easily be ‘air-gapped’ and isolated.
Section 5: Why is the US Water Sector so Vulnerable?
Many of water treatment facilities in the US lack the required technical and financial capabilities to address all emerging risks, such as cyber risks, according to a 2016 National Infrastructure Advisory Council Report. The situation has not improved over the past five years. The Cyberspace Solarium Commission concluded in March 2020 that water utilities remain largely ill-prepared to defend their networks from cyber-enabled disruption. In fact, the former chief technology officer for the state of New Jersey called water and wastewater probably the least mature sector from a cybersecurity standpoint.
This is due to several reasons. For instance, the sector-specific agency (SSA) and risk manager for the water and wastewater industry, the Environmental Protection Agency (EPA) is responsible for identifying and assessing cyber risks to the industry. The EPAs cybersecurity budget, however, is a fraction of that of the Department of Energy, the SSA for the closest comparable lifeline sector. Moreover, municipal governments own more than 80% of US water systems and more than 95% of wastewater systems, but most of these local governments lack the resources to make the needed cybersecurity investments.
In addition, the water industry overall remains generally uninformed. There is little information sharing about cyber attacks among operators of critical infrastructure (like water supply) for multiple reasons: system operators don’t want to alarm the public, they want to avoid any possible liability risk, and they don’t want to publicised the vulnerability of their systems. Finally, there is no law mandating this information sharing. While the America’s Water Infrastructure Act of 2018 requires water systems serving more than 3,300 people to develop or update risk assessments and emergency response plans,there’s no requirement to report cybersecurity incidents. A report from Cyber News shows just how vulnerable US infrastructure is. The report was published in July 2020 after a lengthy investigation by researchers at the publication, before the Oldsmar hack.
Of particular interest is the following excerpt from the research blog:
By scanning IP blocks for open ports in the US IP address range as part of an internet mapping project, we found a number of unprotected and accessible Industrial Control Systems in the country. Industry, institutions, and cybersecurity experts are all aware of the dangers associated with outdated ICS systems. But as our research shows, many ICS access points in the US, particularly in water and energy sectors, are still vulnerable to attacks: i) by using search engines dedicated to scanning all open ports, or scanning the ports themselves, hackers can remotely take control of critical private and public US infrastructure ii) Unprotected ICS access points mostly include the energy and water industries: offshore and onshore oil wells, as well as public and private water distribution and treatment systems iii) These systems could be accessed by anyone with no passwords at all.
Stretched financial resources, employees performing multiple unspecified duties, and all the various other security issues and cyber hygiene problems seen in Oldsmar are fairly common across similar, smaller water treatment facilities.
Section 6: What About the UK?
As increased automation and connectivity reduces the scope for standalone or manual operation of the water supply, the UK’s Department for Environment, Food and Rural Affairs (DEFRA) had outlined its vision for 2017-2021 of a secure, effective, and confident sector, resilient to the ever-evolving cyber threat.
The Water Sector Cyber Security strategy is designed around protecting and developing strong preparedness to respond to emergencies and securing both IT and OT systems. And yet, as of today, many water companies across the UK are still yet to complete the security upgrades and processes needed across their many sites to be fully capable of responding to a cyber-attack. Standardisation across the estate is essential if the water sector is to work as one. With the UK water industry still currently seeing sites working in isolation, inadequately guarded remote water reserves and inferior consumer grade technologies are potentially left wide open to attack, according to the UK’s main digital hub for water industry professionals Water & Wastewater Treatment.
Adopting appropriate measures, such as the installation of enterprise grade security systems, and working closely with trusted partners to proactively guard against attack will be crucial. And as the Centre for the Protection of National Infrastructure (CPNI) has warned, the consequences of failing to formulate a strategic security vision and investment in appropriate measures to mitigate the risk can be costly.
Regulations such as GDPR and the NIS Directive place greater burdens on industry sectors to demonstrate security understanding and compliance and to ensure the integrity of their systems. A scalable, future-proof solution, backed by government guidance and strategy may unite the industry to create a resilient barrier against the next generation of attack. The UK government has made the security of water infrastructure services a national priority, according to the government’s annual National Cyber Security Strategy 2016-2021 (NCSS) progress report. The report outlined plans to extend the deployment of the National Cyber Security Centre’s (NCSC) Active Cyber Defence (ACD) programme beyond traditional government sectors in support of private sector CNI. The UK government has thus established strategic priorities for the period from 2020 to 2025 aimed at securing long-term resilience in the water industry and supported by major investments by water companies and providers.