Having the ability to remotely manage and monitor servers even when their primary operating system becomes unresponsive is vital for enterprise IT administrators. All server manufacturers provide this functionality in firmware through a set of chips that operate independently of the rest of the server and operating system. These are known as Baseboard Management Controllers (BMCs) and if not properly secured can open the door to highly persistent and difficult to detect rootkits.
Over the years, security researchers have found and demonstrated vulnerabilities in BMC implementations from different server manufacturers and attackers have taken advantage of some of them. A recent example is iLOBleed, a malicious BMC implant found in the wild by an Iranian cybersecurity firm that targets Hewlett Packard Enterprise (HPE) Gen8 and Gen9 servers, but it’s not the only such attack found over the years.
According to an analysis by firmware security firm Eclypsium, 7,799 HPE iLO (Integrated Lights-Out) server BMCs from HPE are exposed to the internet, and most do not appear to be running the latest firmware. When more vulnerabilities were discovered in the BMC implementation of Supermicro servers in 2019, over 47,000 publicly exposed Supermicro BMCs from over 90 different countries were exposed. It’s safe to say that among all server vendors, the number of BMC interfaces that can be attacked from the Internet is in the tens or hundreds of thousands.
“BMC vulnerabilities are also incredibly common and often overlooked when it comes to updates,” Eclypsium researchers said in a new blog post following iLOBleed reports. “Vulnerabilities and misconfigurations can be introduced early in the supply chain before an organization takes ownership of a server. Supply chain issues can still exist even after deployment due to upgrades. vulnerable or if adversaries are able to compromise a vendor’s update process. Ultimately, this creates a challenge for businesses in which there are many vulnerable systems, very high impacts in the event of a attack and adversaries actively exploiting the devices in the wild.”
The iLOBleed implant
HPE iLO technology has been in HPE servers for over 15 years. It is implemented as an ARM chip which has its own dedicated network controller, RAM and flash storage. Its firmware includes a dedicated operating system that runs independently from the main server operating system. Like all BMCs, HPE iLO is essentially a small computer designed to control a larger computer, the server itself.
Administrators can access iLO through a web-based administration panel served by the BMC’s dedicated network port or through tools that communicate with the BMC through the industry-standard Intelligent Platform Management Interface (IPMI) protocol. Administrators can use iLO to power on and off the server, change various hardware and firmware settings, access the system console, reinstall the main operating system by attaching a CD/DVD image remotely, monitor hardware and software sensors and even deploy BIOS/UEFI updates. .
The iLOBleed implant is suspected to be the creation of an Advanced Persistent Threat (APT) group and has been in use since at least 2020. It is believed to exploit known vulnerabilities such as CVE-2018-7078 and CVE-2018-7113 to inject new malicious modules into iLO firmware that add disk wiping functionality.
Once installed, the rootkit also blocks firmware upgrade attempts and reports that the new version has been installed successfully to fool administrators. However, there are ways to know that the firmware has not been updated. For example, the login screen in the latest available version should be slightly different. If not, it means the update was prevented, even though the firmware is reporting the latest version.
It should also be noted that it is possible to infect iLO firmware if an attacker obtains root (administrator) privileges on the host operating system, as this allows the firmware to be flashed. If the server’s iLO firmware has no known vulnerabilities, it is possible to downgrade the firmware to a vulnerable version. On Gen10, it is possible to prevent downgrade attacks by enabling a firmware setting, but this is not enabled by default and is not possible on older generations.
“Attackers can abuse these [BMC] abilities in various ways,” the Eclypsium researchers said. “iLOBleed has demonstrated the ability to use the BMC to erase disks from a server. The attacker could just as easily steal data, install additional payloads, control the server in any way, or disable it altogether. It’s also important to note that compromising physical servers can put not only workloads at risk, but entire clouds as well. »
Past BMC Attacks
In 2016, Microsoft researchers documented the activities of an APT group called PLATINUM which used Intel’s Serial-over-LAN (SOL) Active Management Technology (AMT) to set up a secret communication channel to transfer files. AMT is a component of Intel Management Engine (Intel ME), a BMC-like solution that exists in most Intel desktop and server processors. Most firewalls and network monitoring tools are not configured to inspect AMT SOL or IPMI traffic in general, allowing attackers like PLATINUM to evade detection.
In 2018, BleepingComputer reported attacks on Linux servers with a ransomware program called JungleSec which, based on victim reports, was deployed through insecure IPMI interfaces using manufacturer default credentials.
In 2020, a security researcher show how he could leverage insecure BMC interfaces on an organization’s Openstack cloud to take control of virtualized servers during a penetration testing engagement.
“iLOBleed provides an incredibly clear case study of not only the importance of firmware security in BMCs, but also for firmware security in general,” the Eclypsium researchers said. “Today, many organizations have embraced concepts such as zero trust, which defines the need to independently assess and verify the security of every asset and action. Yet, in most cases, these ideas have not made their way to the most fundamental code of a device. .”
BMC attack mitigation
Standard security practice for IPMI interfaces, whether built-in or added via expansion cards, is not to expose them directly to the Internet or even to the main corporate network. BMCs should be placed in their own isolated network segment for management purposes. Access to this segment can be restricted using VLANs, firewalls, VPNs, and other similar security technologies.
Organizations should periodically check with their server manufacturers for BMC firmware updates and more generally track CVEs discovered in the firmware of all their critical assets. The lack of firmware version tracking and vulnerability scanning creates a large blind spot on corporate networks and low-level rootkits like iLOBleed can provide attackers with a highly persistent and powerful presence in environments.
If the BMC firmware provides the ability to block the deployment of older firmware versions – downgrades – as in the case of HPE Gen10/iLO5 servers, this option should be enabled. Other firmware security features such as digital signature verification must also be enabled.
Default administrative credentials for BMC interfaces and admin panels should be changed, and security features such as traffic encryption and authentication should still be enabled.
Finally, many BMCs have logging capabilities that allow changes to servers to be monitored and recorded through specifications such as Redfish and other XML interfaces. These logs should be audited periodically to detect any unauthorized changes.
Copyright © 2022 IDG Communications, Inc.