Researchers have identified a new malware family that was designed to backdoor and create persistence on VMware ESXi servers by leveraging legitimate functionality the hypervisor software supports. According to researchers from Mandiant who found and analyzed the backdoors, they were packaged and deployed on infected servers as vSphere Installation Bundles (VIBs). VIBs are software packages used to distribute components that extend VMware ESXi functionality. The malicious VIBs provided hackers with remote command execution and persistence capabilities on the servers and the ability to execute commands on the guest virtual machines running on the servers.
Hackers used unsigned VIBs that were hard to detect
By default, VMware ESXi is configured to accept only the installation of VIBs that are VMWareCertified, VmwareAccepted, or PartnerSupported. At these levels of acceptance, the bundles need to be digitally signed by either VMware or a partner whose signature VMware trusts.
However, there is a fourth level of acceptance called CommunitySupported and VIBs in this category do not need to be digitally signed. The downside is that these bundles need to be deployed by an administrator by intentionally using the –force flag on the installation command through the esxcli command line tool.
The malicious VIBs found by Mandiant had their manifest file modified to list “partner” as the acceptance level, but in reality had no digital signature and had been deployed using the –force command. This means the attackers already had administrative-level access to the servers before deploying them. so they were a late-stage payload.
One effect of listing “partner” as the source in the manifest of the rogue VIBs was that they appeared listed as PartnerSupported when the “esxcli software vib list” command was used when in fact they weren’t. This oversight in the command that simply displays what the manifest says, helped attackers better hide their backdoors from administrators. To discover them, admins would have had to use the command “esxcli software vib signature verify” that would have verified the digital signature of all the deployed VIBs on their servers.
Attackers deployed both hypervisor and virtual machine backdoors
In addition to a manifest file and a signature file, VIBs include a collection of files and directories that will be copied on the system. One of these files was a passive backdoor that used VMware service names to hide itself and listened to traffic on a hard-coded port number on the ESXi server. The backdoor, which was named VIRTUALPITA can perform arbitrary command execution, upload and download files, and start and stop vmsyslogd, the ESXi service that’s responsible for logging messages from the system kernel and other components.
“During arbitrary command execution, the malware also sets the environmental variable HISTFILE to 0 to further hide activity that occurred on the machine,” the Mandiant researchers said. “Variants of this malware were found to listen on a Virtual Machine Communication Interface (VMCI) and log this activity to the file sysclog.” The VMware VMCI is the high-speed communication interface through which virtual machines communicate with the host kernel.
Two samples of VIRTUALPITA found on Linux vCenter systems masqueraded as startup services in init.d – a startup mechanism in Linux – and their file name was disguised as ksmd (Kernel Same-Page Merging Daemon), a default kernel service, in the directories /usr/libexec/setconf/ and /usr/bin.
The researchers also found a secondary backdoor in the malicious VIBs that they dubbed VIRTUALPIE. This backdoor program was written in Python and listened for IPv6 traffic on port 546. Attackers could use this backdoor to execute arbitrary commands, transfer files and open a reverse shell. The communications through the port was done through a custom protocol that used RC4 encryption.
Finally, some attacks involved a third backdoor dubbed VIRTUALGATE that was written for Windows and was deployed on the guest virtual machines that ran on the compromised ESXi servers. This backdoor allows attackers to execute commands on the guest VM from the hypervisor or between different guest virtual machines running on the same host through the VMCI.
The researchers observed attackers using VIRTUALPITA to execute a shell script that launched a Python script that then executed commands on guest virtual machines. On the VMs, the commands were executed by the legitimate VMware Tools service (vmtoolsd.exe). In one instance the commands involved listing files from certain directories then packaging them as CAB archives and in another instance the attackers used the MiniDump utility to dump a process’ memory and search for plaintext credentials in it.
Mandiant hasn’t linked these attacks to any known groups, so it tracks them under a new group identifier called UNC3886. “Given the highly targeted and evasive nature of this intrusion, we suspect UNC3886 motivation to be cyber espionage related,” the researchers said. “Additionally, we assess with low confidence that UNC3886 has a China nexus.”
Even though there are not yet many incidents where VIB malware was used to compromise ESXi servers, Mandiant expects other threat groups to copy this technique in the future.
VMware recommends enabling UEFI Secure Boot
VMware has published an advisory in response to Mandiant’s findings as well as a PowerShell script that can be used to hunt for malicious VIBs in an environment. However, the primary recommendation is to enable UEFI Secure Boot on the system, which provides cryptographic attestation of components since early in the boot process.
“When Secure Boot is enabled the use of the ‘CommunitySupported’ acceptance level will be blocked, preventing attackers from installing unsigned and improperly signed VIBs (even with the –force parameter as noted in the report),” VMware states in its guidance. “vSphere 8 takes another step and prevents the execution of unsigned binaries, or binaries installed through means other than a properly signed VIB. Efforts to disable that feature by attackers generates undismissable ESXi alarms as clues that something is happening in an environment.”