Most industrial sites deploy firewalls as the first line of defense for their Operations Technology (OT) / industrial networks. However, configuring and managing these firewalls is a complex undertaking. Configuration and other mistakes are easy to make.
This article explores eight common mistakes that firewall administrators make and describes how these mistakes can compromise firewall functionality and network security. The lesson here though is not “stop making mistakes.” This article also explores unidirectional gateway technology as an alternative to our most important OT firewalls. Unidirectional gateways provide physical protection for industrial operations, rather than merely software protection. This means that with a unidirectional gateway, no mistake in configuration can impair the protection that the gateway provides to the industrial network.
Everyone makes mistakes. The secret to secure operations is not to magically avoid all mistakes, but to design our industrial networks so that even when we make mistakes configuring our defenses, there is still no threat to correct and continuous industrial operations.
#1 – Leaving the IP “any to any” access rule in place
Firewalls are inconsistent and confusing. Most consumer firewalls deny all connections inbound to the home network by default but permit all outgoing connections with an “allow any to any” rule. Commercial and industrial grade firewalls are inconsistent – some have a “deny all” default policy for all directions of traffic. Some have a default “allow all to any”. Some have a default “allow any to any” until the first non-default rule is configured, and the default then switches to “deny all.” Some show their default rules in their configuration user interfaces, and others do not.
This makes it very easy, for some models of firewalls, to mistakenly leave an “allow any to any” rule in place. The consequences of mistakenly leaving a visible default or invisible implied “allow any to any” rule in place is, of course, that very many kinds of connections into our industrial networks are left enabled – the rule permits all connections/attacks into our networks that are not explicitly forbidden by some other rule we have configured.
#2 Using the wrong order of rules
Once we have identified all the rules that need to be set up for a firewall, we must pay careful attention to the order of the rules in the rule set. Firewall rules are processed in order. Rules entered or configured in the wrong order will yield unexpected and undesirable firewall behaviors.
For example, imagine we have two rules, one saying “accept all connections from IP addresses 1-64 in a subnet” and a second one saying “deny connections from IP address 23” in that same subnet. If the accept rule is first in the ruleset, and the firewall receives a connection request from address 23, then the “accept 1-64” rule causes the connection to be allowed. The “deny” rule is never evaluated.
#3 Failing to disable unused administrative interfaces
Since firewall manufacturers want to ensure easy configuration, they enable many types of administrative interfaces by default, often including SSH, Telnet, and serial interfaces, as well as encrypted and unencrypted web interfaces. Leaving un-encrypted interfaces enabled means attackers may be able to see passwords on the network when those interfaces are used. In addition, leaving any un-necessary interfaces enabled increases the attack surface and exposure. It also enables attackers to use phishing attacks or other attacks to steal passwords for these interfaces and simply log in to reconfigure the firewall.
#4 Leaving the firewall default password unchanged
Most firewalls ship with a default administrative user name and password. These passwords are documented in the device’s user manual and so are well-known to attackers and to anyone else who searches for the information. Failing to change the default password means that any attacker who can connect to one of the administrative interfaces that we have left enabled can log into the firewall and reconfigure it.
Failing to change the default password is an especially common error when the firewall is connected to external authentication, authorization, and accounting (AAA) service such as a RADIUS server, Active Directory server, IAM infrastructure, or other password management server. Password management services generally have no control over the built-in administrator account and password. In fact, best practice holds that at least one administrator account should be beyond the reach of the identity management system, to serve as a backup in case we lose contact with the AAA system for any reason.
#5 Failing to patch the firewall
Engineers and administrators may have extensive and costly testing and software update programs to keep their industrial control systems patched. These programs often focus on difficult-to-patch operations equipment to the exclusion of network infrastructure components such as firewalls and managed switches. Failing to patch a firewall means that attackers can use well-known and widely-available exploits for old and well-known firewall vulnerabilities to compromise our firewalls.
#6 Failing to plan for additional infrastructure costs
Deploying a firewall can incur significant and unexpected costs because firewall deployments often demand significant changes to other infrastructures. These changes and costs may include:
- Renumbering IP addresses on OT and/or enterprise networks when a new firewall is used to segment what was previously a “flat” network,
- Additional network infrastructure, such as switches, routers, and authentication systems, and
- People and/or systems to gather, correlate, and analyze firewall logs, to try to detect when an attacker is guessing passwords or otherwise trying to gain access to our networks.
Even greater than these costs, there may be downtime costs for physical/industrial operations if any of the above changes require a restart of the entire control system, and such restarts are often necessary.
#7 Mismanaging rule sets over time
Firewall rules must be updated and reviewed regularly. “Temporary” rules that are introduced for short-term testing, emergency repairs, and other needs, must not be allowed to persist. Rules for obsolete equipment and software systems must be removed. Rules must be removed that provide OT access for IP addresses used by employees who have left the organization or who have changed responsibilities. All of these and many other changes must be documented, so that future reviewers know who to contact to determine whether configured rules are still necessary.
In short, if we permit un-necessary rules to accumulate, then over time the firewall looks more and more like a router – permitting far too many kinds of connections into the very networks the device was nominally deployed to protect.
#8 Believing a firewall is “outbound only”
In process control system networks, there is often a belief that we are protected because the firewall has been configured to permit only outbound connections, from the industrial network to an external network. This is a serious mistake. To quote Ed Skoudis, a respected SANS instructor, “outbound access equals inbound command and control.” All TCP connections, even those through a firewall, are two-way connections. Connections to email servers and web servers pull attacks through “outbound only” firewalls all the time. Malware connecting to command and control servers does the same. More generally, industrial clients connecting to compromised enterprise services risk propagating compromise into control systems and putting industrial operations in peril.
The unidirectional alternative
Thoroughly secured industrial networks increasingly deploy unidirectional gateway technology at IT/OT interfaces, instead of firewalls. One important advantage for unidirectional technology is that even when accidentally misconfigured, no misconfiguration can impair the protection that the gateways provide to OT networks. Unidirectional gateway hardware is physically able to send information in only one direction – from the OT network out to the enterprise. No cyber attack, however sophisticated, can change the behavior of the hardware, nor can any cyber attack information reach through the hardware to impair the protected OT network in any way.
To simplify safe IT/OT integration, unidirectional gateway software replicates servers from industrial to enterprise networks. For example, let’s say enterprise users and applications acquire their OT data from a SQL-based historian database. In this case, the unidirectional gateways query the database on the OT side, convert the received data into unidirectional formats, send the data to the enterprise side, and there insert the data into an identical SQL/historian database. Enterprise users and applications then have access to their data normally and bi-directionally from the enterprise replica database. If needed, the replica database server can have exactly the same IP address as the industrial historian database has – this because the unidirectional gateway is not a firewall or a router, and so cannot be confused by the same IP addresses in use on both sides of the device.
With unidirectional gateway hardware protecting an industrial network, no software attack from an enterprise network, or from the Internet beyond the network, can reach the protected OT network, no matter how many passwords are stolen or systems we might not be able to patch.
Summary
Firewalls are easy to misconfigure. While the security consequences of such errors may be acceptable for some firewalls, the accumulated risks of misconfigured firewalls in a defense-in-depth OT network architecture are generally unacceptable. The solution is only partly to “be more diligent” about managing our firewalls. No human process is perfect. In addition, many industrial and critical infrastructure sites replace at least one layer of firewalls in their defense-in-depth architecture with unidirectional gateways. The most common such deployment is to replace the IT/OT interface firewall with a gateway.
With at least one layer of unidirectional gateway in place, the attack path leading from the Internet into our OT networks is broken. Unidirectional gateway hardware prevents any online cyber attack, no matter how sophisticated, from reaching our vulnerable production networks. The hardware saves the OT network from interference, even if we have accidentally misconfigured the gateway software.
Free book: Unidirectional gateway technology is often misunderstood. To learn more about unidirectional IT/OT integration with disciplined control, consider visiting the Waterfall Security Solutions website at https://waterfall-security.com/sec-ot to request a free copy of the book Secure Operations Technology. The book describes a wide variety of unidirectional network architectures in more detail.
