Millions of vulnerable industrial control devices

It’s a known fact that embedded devices, especially those designed for industrial automation and that have a long lifespan, use a mix of in-house and third-party code, created in an age when software vulnerabilities were not were not as well understood as they are today. Critical flaws discovered in proprietary components that hardware vendors have been using for years have far-reaching consequences. Especially since the application of patches is not always possible. This is underscored by results obtained over the past year by researchers from Forescout Research Labs and JFrog Security Research, who have studied the TCP/IP stacks used in various IoT and other embedded systems. Their work has identified major flaws like Ripple20, NAME:WRECK, NUMBER:JACK or AMNESIA:33 that impact millions of devices.

Their latest report, released today as INFRA:HALT, covers 14 critical and high-risk vulnerabilities found in a proprietary TCP/IP stack called NicheStack, widely used in operational technology (OT) devices by nearly 200 suppliers, including programmable logic controllers (PLC), such as the Siemens S7, which make up industrial automation and are used in critical infrastructure sectors.

The huge attack surface of TCP/IP stacks

TCP/IP stacks, or Internet Protocol Suites, consist of implementations of common Internet protocols, including DNS, HTTP, FTP, ARP, and ICMP that enable operating systems and their applications to send and receive data on IP networks. Given the multitude of protocols supported by these stacks and the amount of data and packet formats they process, they expose a large attack surface, often exploitable without authentication. For a long time, industrial control devices communicated mostly via serial interfaces, but over the years they have increasingly been equipped with Ethernet interfaces and, implicitly, TCP/IP stacks, in order to be able to communicate with computers and common computing devices.

Many modern IoT devices run on Linux and therefore use the Linux TCP/IP stack, which has been closely scrutinized by security researchers and Linux kernel developers for three decades. However, industrial control devices commonly run proprietary real-time operating systems (RTOS) that use proprietary TCP/IP stacks with inconsistent versions, highly customized modifications, and ownership changes, making identification difficult. vulnerable products and, ultimately, patching.

Vulnerable to buffer overflow

Originally developed in 1996 or earlier by a company called InterNiche Technologies, the TCP/IP NicheStack was extended to support the new IPv6 technology in 2003. In 2016, InterNiche Technologies was acquired by another company called HCC Embedded which still maintains the stack. “Over the past two decades, the stack has been released and released in multiple “flavors” by OEMs like STMicroelectronics, Freescale (NXP), Altera (Intel), and Microchip for use with multiple operating systems (real-time ) or its own simple RTOS called NicheTask,” the Forescout researchers say in their report. “It also served as the basis for other TCP/IP stacks, such as SEGGER’s emNet (formerly embOS/IP).”

The majority of the 14 vulnerabilities discovered by Forescout and JFrog researchers are buffer overflows and out-of-bounds memory reads and writes that result from insecure packet scanning across various protocols. These vulnerabilities can be exploited over DNSv4, HTTP, TCP, ICMP, or TFTP and can lead to remote code execution (two vulnerabilities) and denial of service conditions (eight vulnerabilities). Other flaws result from predictable TCP Initial Sequence Number (ISN) sequence numbers, insufficiently random DNS transaction IDs, and predictable source port numbers for DNS queries, which allow for example TCP spoofing or corruption attacks caching. All vulnerabilities impact all versions of NicheStack prior to 4.3, the latest version available at the time the research was performed.

Vulnerability coordination hell

The two remote code execution flaws are located in the DNSv4 and HTTP implementation and have a respective score of 9.8 and 9.1 in the Common Vulnerability Scoring System (CVSS), which means that they are seriously critical. Severity scores for Denial of Service (DoS) issues are 7.5 or 8.2 in CVSS. However, it should be noted that in the context of industrial control systems, the potential impact of a denial of service issue can be severe, depending on the type of industrial process controlled by the affected device. For example, recovering from attacks exploiting these vulnerabilities, including DoS attacks, would require turning the affected devices on and off. “That means having physical access to it,” explained Elisa Costante, vice president of research at Forescout. “So the impact can be quite significant. Imagine that this device is at sea for substations or oil extraction”.

It took more than a year for researchers to coordinate to disclose the INFRA:HALT vulnerabilities, far longer than the usual 90 days for software vulnerabilities. Forescout and JFrog Security Research contacted HCC Embedded to alert them to vulnerabilities in September 2020 and they worked with the CERT Coordination Center (CERT/CC), the German Federal Office for Information Technology Security ( Bundesamt für Sicherheit in der Informationstechnik or BSI) and the Emergency Response Team for Industrial Control Systems Computer Emergency Readiness Team (ICS-CERT) which is part of the US Federal Cybersecurity and Infrastructure Security Agency (CISA ). Despite this, identifying potentially affected devices and vendors has been very difficult and this work is far from complete. Using queries on the SHODAN search engine, researchers found around 6,400 publicly available devices that use NicheStack. Using its own proprietary database of millions of digital device fingerprints, Forescout identified 2,500 potentially vulnerable devices from 21 vendors, with the most affected verticals being manufacturing processes, retail and unit manufacturing. Nearly half of the devices identified relate to industrial control systems for energy and electricity.

No attenuation without visibility

But the real impact of these flaws is much broader. According to the researchers, a former InterNiche website, which is no longer online, listed nearly 200 device vendors as customers, including major OT system vendors like Siemens, Emerson, Honeywell, Mitsubishi Electric, Rockwell Automation and Schneider Electric. “Only a small number of vendors will issue public advisories, but the forecast is that the actual number of affected devices is likely in the millions,” said Elisa Costante. “This is the first study that shows so significantly the wide variety of ICS vendors impacted […] but few OT devices are actually on public display, so it’s a bit more difficult for us to get information about them.” Forescout maintains a list of vendor advisories impacted by its TCP/IP stack research on GitHub and will be updated with new INFRA:HALT-related advisories as they become available.

HCC Embedded has developed patches for the vulnerabilities, but they are only available upon request from customers, who are mostly device manufacturers. End users of the affected products should wait for fixes from the manufacturers of their respective devices. The issue is further complicated by the fact that it is unlikely that all vendors, especially smaller ones, that have incorporated this TCP/IP stack into their products over the years still have active contracts with HCC Embedded. Additionally, some of the affected devices no longer receive technical support and may never receive patches. “Another problem is that the owner of these devices, even if a patch is available, does not always know that these devices are vulnerable,” said Ms. Costante. “Sometimes they don’t have a complete inventory of all their assets, so even the risk assessment is incomplete.”

Forescot has developed an open-source script that asset owners can use on their networks to find out which devices are running NicheStack or other TCP/IP stacks in which the company has discovered vulnerabilities as part of a larger investigation dubbed Project Memoria, conducted by the past. Forescot has also updated its own commercial products to find affected devices and detect exploit attempts. Another issue with patch deployment: Some of the affected devices are likely controlling critical or always-on processes in factories and industrial facilities, or they’re deployed in the field in remote locations so they can’t be turned off and updated without scheduled maintenance. “Mitigation measures will work better than patches for the majority of providers, especially for the smaller ones,” Elisa Costante said.

Tips for mitigating INFRA:HAL vulnerabilities

• Apply segmentation controls and good network hygiene to mitigate risk from vulnerable devices. Limit external communication paths and isolate or confine vulnerable devices to areas as a mitigation control if they cannot be patched or until they can.

• Monitor release of patches by affected device vendors and develop remediation plan for vulnerable asset inventory considering risk and business continuity requirements.

• Monitor all network traffic for malicious packets that attempt to exploit known vulnerabilities or possible zero-day flaws. Abnormal and malformed traffic should be blocked, or at least reported to network operators.

• Disable DNSv4 client if not needed, or block DNSv4 traffic. Since several vulnerabilities facilitate DNS spoofing attacks, the use of internal DNS servers may not be sufficient (attackers may be able to hijack request-response matching).

• Disable the HTTP protocol if it is not needed, or whitelist HTTP connections.

• Monitor traffic to detect and block malformed IPv4/TCP and ICPMv4 packets.