Investigating a possible Ivanti compromise

On 10 January 2024, Ivanti warned about two high severity zero-day vulnerabilities (CVE-2023-46805 and CVE-2024-21887¹) in Ivanti Connect Secure VPN (formerly known as Pulse Connect Secure, or simply Pulse Secure). The vulnerabilities allow for authentication bypass as well as command injection in the web component of Ivanti Connect Secure and Ivanti Policy Secure. The vulnerabilities allow a threat actor to access restricted resources by bypassing control checks, as well as allowing an authenticated administrator to send specially crafted requests and execute arbitrary commands on the appliance. See also our threat response on these two zero-days².

Already, these zero-days are reported to be actively exploited in the wild and utilised by nation-state-linked APT groups that are deploying techniques to avoid detection. Additionally, with publicly available public proof-of-concept code, several threat actors are now actively exploiting the vulnerabilities to deploy malware and cryptocurrency miners³.

Performing digital forensics on the physical devices to check for a potential compromise proves to be challenging due to the set-up and full-disk encryption that Ivanti utilises. Hence, we dug a little deeper to extract the private key and figure out the encryption algorithm to decrypt the data for further investigation. Something that has been done for recent versions of Ivanti appliances, where the disks were encrypted with LUKS⁴. However, we encountered a few cases with Linux Loaders (LILO) that were not using LUKS encryption. In this blog, we explain the process to obtain the cryptographic key and decrypt the disks so that forensics can be properly conducted. With this blog we aim to aid incident responders in investigating potential compromises.

Description of the vulnerabilities

Ivanti's Connect Secure VPN, previously known as Pulse Secure VPN, is an SSL VPN solution for remote users to connect to a network from anywhere⁵. Its setup usually consists of a server in the network, and VPN client software installed on systems that connect to the server.

The vulnerability tracked as CVE-2023-46805 is an authentication bypass vulnerability in the web component of Ivanti Connect Secure (9.x, 22.x) and Ivanti Policy Secure, which allows a remote attacker to access restricted resources by bypassing control checks.

The other vulnerability, tracked as CVE-2024-21887, is a command injection vulnerability in web components of Ivanti Connect Secure (9.x, 22.x) and Ivanti Policy Secure. It allows an authenticated administrator to send specially crafted requests and execute arbitrary commands on the appliance. What makes this vulnerability particularly concerning is its potential for exploitation over the internet, posing a direct threat to the security posture of the affected systems.

The following versions of the software are vulnerable:

Ivanti Connect Secure (9.x, 22.x)
Ivanti Policy Secure (9.x, 22.x)

Exploitation in the wild

CVE-2023-46805 & CVE-2024-21887 have been widely reported in the media as being utilised by nation-state-linked APT groups to compromise Ivanti appliances⁶. Volexity has identified a webshell called GIFTEDVISITOR on over 2100 compromised Ivanti devices. The threat actor UTA0178 has been exfiltrating data associated with accounts, session data, and more from Ivanti Connect Secure VPN appliances.

Victims are globally distributed and vary greatly in size, from small businesses to some of the largest organisations in the world, including multiple Fortune 500 companies across multiple industry verticals.

Dissecting Ivanti’s full-disk encryption

The LILO based Ivanti systems use a modified version of loop-AES compiled into the kernel. A function is added to the kernel which wraps the root device specified on the commandline with a crypto loopback with hardcoded key. This function is called loop_setup_root and is called from prepare_namespace, just before mounting the root filesystem.

The normal ioctl handler also sets this same hardcoded key when invoked from user space.

The hardcoded key varies per kernel build. The systems we examined had a current and a rollback install. Each had different kernel versions and different keys. There also is a factory-reset kernel, which we have not examined yet.

Each kernel also has its own partition with just a kernel, a compiled LILO config and/or LILO config and version texts. These partitions are the only ones that are not encrypted.

Dumping

You can either dump a live system or use qemu to run an imaged disk.

For qemu use qemu-system-x86_64 -serial stdio -device ich9-ahci,id=ahci -drive file=ivanti-backup.img,id=disk,if=none -device ide-hd,drive=disk,bus=ahci.0 and add network config as appropriate.
For a live system hook up network and the serial console. On the system we examined there were two serial ports present, a rollover RJ45 connector and a DB9 (hidden behind a bezel). The serial console presented itself on the RJ45 port.

To dump a live or emulated system, conduct the following steps:

At the LILO prompt: current init=//bin/sh debug verbose=5
System now boots with a shell as init.
/sbin/insmod /$kernel-version/modules/igb.ko substitute appropriate kernel version and NIC driver
/sbin/ifconfig eth1 192.168.0.2 netmask 255.255.255.0 substitute appropriate NIC and IP range. For our system, eth1 corresponded to int on the label.
Start a netcat listener on 192.168.0.1 and pipe the output to a file nc -l 1337 > loop6.img
On the Ivanti do mount -t proc none /proc to mount the /proc filesystem.
grep loop /proc/partitions Should show a single loop device, was loop6 on our system.
dd if=/dev/loop6 bs=1M > /dev/tcp/192.168.0.1/1337
Wait for it to complete.

To dump other partitions, you can enable them using losetup /dev/loopXX /dev/sdaXX. As not all partitions are decryptable with all kernels you cannot decrypt all in a single run. When running this losetup without specifying a backing device the specified device will be encrypted (as mentioned by the helptext).

In our case, sda6, sda7 and sda12 belonged to current and sda8, sda9 and sda13 belonged to rollback.

Decrypting

The regular loop-AES software supports multiple modes of encryption⁷. For AES, they all boil down to AES in CBC with different ways of generating a per sector IV.

It also allows for the use of multiple keys, each generated from the same key material. The key to use is chosen by taking the modulus of the sector by the number of keys.

Ivanti chose to use the single key mode. Hardcoding the 16-byte key into the kernel. Ivanti has taken the simplest IV generation mode, the per sector IV. This is simply the sector number encoded as a little endian 8-byte integer with 8 trailing zeroes.

The CBC mode, however, was modified. The ciphertext is XOR-ed with a decrypted version of the per sector IV before AES decryption resulting in a xor-encrypt-xor mode.

In python code:

def xor(s1, s2): return bytes([c1 ^ c2 for c1, c2 in zip(s1, s2)]) def ivanti_cbc(key, sector, data): cipher = AES.new(key=key, mode=AES.MODE_ECB) def x(iv): pre_iv = cipher.decrypt(iv) for i in range(0, len(data), 16): ciphertext = xor(data[i:i+16], pre_iv) plaintext = cipher.decrypt(ciphertext) yield xor(plaintext, iv) iv = ciphertext iv = sector.to_bytes(length=16, byteorder='little') return b''.join(x(iv))

Conclusion

With this write-up we hope to have demonstrated how to decrypt older LILO-based Ivanti appliances to help incident responders in investigating potential compromises. For the kernel versions that we encountered, we have provided their respective cryptographic keys. For other versions, you should be able to extract the keys using a similar method.

We have published a tool on Github to extract the cryptographic key from a kernel and to decrypt the encrypted partition data.

Northwave wants to highlight two key points regarding mitigation:

The threat actor attempted ways to bypass the built-in Integrity Checker Tool. Therefore, it is important for organisations to also run the external Integrity Checker Tool⁸.
To prevent reinfection, it is important to take in to account the order of mitigation steps: first restore configuration backups and then apply the mitigation⁹ shared by Ivanti. The configuration backups will overwrite the applied mitigation.

Northwave expects that threat actors known for ransomware and data exfiltration activity will also make use of the vulnerable Ivanti systems. For that reason, it is paramount to patch once Avanti makes a patch available.

To investigate potential compromises, related indicators can be found on GitHub¹⁰.

Authors

Michael Schrijver, Rang Salih & Patrick van Looy.

Kernel	Key
2.6.32-0008-g2f6ef39-dirty	D4 A5 7D 2A 87 D3 D2 CE A2 79 0E E3 CF 4E 7D B3
2.6.32-0032-g2005e8d-dirty	29 09 40 A1 74 F1 8E 2B E9 80 F9 FA 51 72 A7 4B

Cyber Incident?

Call 00800 1744 0000

Investigating a possible Ivanti compromise

Description of the vulnerabilities

Exploitation in the wild

Dissecting Ivanti’s full-disk encryption

Dumping

Decrypting

Conclusion

Authors

We are here for you

Business

Bytes

Behaviour