How to adapt Dissect Acquire for hypervisor-based scalable evidence collection during ransomware attacks
By Matthijs Vos, Michael Schrijver, Wouter van Kranenburg, Marinus Boekelo
Ransomware attacks are an all-too-common cyber threat that can bring entire businesses to a standstill. When a Northwave response team steps in to help companies recover from these complex and potentially devastating incidents, one of the first challenges we face is gaining access to critical evidence against the attackers. This is because the evidence is often locked away in virtual machines (VMs) that the cyber criminals have taken offline.
We realised we could discreetly gather evidence, without disrupting the compromised systems, by entering the hypervisor-level data stores. These are the central hubs where VMs are managed. This method is particularly valuable when analysing large-scale environments because we can collect data from every virtual machine housed on the hypervisor.
Thanks to the advancements made by Fox-IT on their Dissect Acquire tool 1, our team can now perform targeted evidence collection directly from VM files. But we wanted to make this method even more effective. Since VMware ESXi is the hypervisor platform we most frequently encounter, running Dissect on ESXi would enable us to swiftly and efficiently gather evidence, while minimising the operational impact on the affacted systems.
So, we developed way to allow the Dissect toolchain to run self-contained on ESXi instances. The result is a scalable solution for automated, targeted evidence collection in offline virtual systems. If you are interested in how to run the Dissect toolchain on ESXi, skip to the TL;DR below. Or keep reading to for a closer look at how we addressed this challenge.
Adapting Dissect Acquire for VMware ESXi
We initially attempted to run Dissect on ESXi instances like we would do on Linux environments. Our attempts failed when we stumbled against the limitations of ESXi’s proprietary architecture (which on many levels deviate from the Linux kernel interface). We then shifted our focus to developing a way to seamlessly operate within ESXi environments while maintaining the integrity and efficiency of the evidence collection process.
To achieve this, our next approach was to run Dissect Acquire on ESXi using static builds produced by PyOxidizer, which Fox-IT2 had previously suggested. PyOxidizer provides the means to convert Python projects into self-contained binaries, leveraging Rust to support multiple platforms. These binaries are designed to contain all the necessary dependencies, essentially embedding a Python interpreter into the code itself.
A key feature of PyOxidizer is its ability to produce ELF binaries that embed Python code. In theory, these ELF binaries are compatible with ESXi. However, when we tried running a PyOxidizer build of Dissect, we and others3 encountered run-time errors:
These errors are caused because ESXi does not expose the process executable path via "/proc/self/exe", which Rust appears to expect:
Further testing revealed that ESXi, unlike Linux, does not feature the /proc virtual filesystem as an interface with the kernel, so /proc/self/exe cannot be resolved on ESXi” 😕. So leveraging PyOxidizier to build ESXi-compatible builds of Dissect Acquire resulted in run-time roadblocks.
Hot-patching Rust
After encountering the above issue, we looked into the Rust codebase and noted that there isn’t an ESXi target available in Rust.However, Rust does have an implementation for “vxworks” that resolves executable paths by using “argv[0]” instead of /proc/self/exe. This approach is compatible with ESXi and provides a good alternative to the “/proc/self/exe” approach.
Off to patching Rust… By manually changing the Linux target from the “/proc/self/exe” implementation to the “argv[0]” implementation, we can create the Rust environment that works for ESXi!
After generating Dissect Acquire builds using this environment, we confirmed that it worked:
The scripts we used to create a build environment to produce ESXi compatible builds of Dissect can be found at https://github.com/NorthwaveSecurity/Dissect-ESXi.
External Dissect Dependencies
While building Dissect for ESXi, we noticed Python libraries that use native code can cause issues in PyOxidier builds. For Dissect, we found the optional Python library PyCryptodome4 (which relies on native C code) caused issues.
We noted PyCryptodome was used in Dissect to decrypt the encrypted ‘local.tgz.ve’ on ESXi targets. This file contains the ESXi configuration files and is by default encrypted since vSphere 7.0 Update 2.
Fortunately, there is a workaround for this: ESXi provides us with the ‘crypto-util’ binary which can be used the decrypt this file. Fox-IT accepted our pull request with the changes necessary to make Dissect automatically use this workaround when the PyCryptodome package is not available5. This means a manual action is no longer required to work around this problem.
Community Sharing
The workaround described above was shared with participants of Dissect’s community day, organised by Fox-IT in July 2024. As the ESXi compatibility problems were recognised by many of the participants, the workaround was received positively.
Shortly after Dissect’s community day, Fox-IT went a step further and found a more elegant workaround to address the ESXi compatibility issues. Instead of having to patch and build Rust, Fox-IT forked and patched PyOxidizer to replace the call to the standard library function “std::env::current_exe” with functionality that looks up the current executable via “std::env::args_os”. The solution of Fox-IT is currently in review and is expected to be adopted in their CI/CD pipeline soon6.
TL;DR
In the meantime, if you would like to run Dissect Acquire directly on the ESXi hypervisor right away, take the following steps:
- Apply the Rust patch to the Rust source code
- Compile Rust
- Compile Dissect using PyOxidizer
All these steps, including the code to produce ESXi-compatible Dissect binaries, can be found on our GitHub: https://github.com/NorthwaveSecurity/Dissect-ESXi.
Conclusion
By creating a solution to run Dissect in a self-contained way, we are now able to take advantage of Dissect's capabilities directly on the ESXi hypervisors. This allowed us to build further automation to collect targeted evidence from all VMs hosted on the hypervisors, delivering the scalability we were looking for.
By making our workaround available on our GitHub, we hope to return some valuable capabilities to the DFIR community. This will hopefully allow other Dissect users to continue adding capabilities in digital forensics and incident response, rather than spending time fixing a problem that’s already been solved.
We are here for you
Need help with getting your organisation ready for DORA or wondering far along you your business currently is?
Get in touch and we will guide you with your next steps.