Advania UK logo  Advania UK logo compact

A practical guide to bypassing userland API Hooking

Advania - Guide to bypassing API hooking
Posted On
Written by
Duration of read
8  min
Share Article
Subscribe via email
During a recent red-team engagement, we encountered hosts protected by different types of Endpoint Detection and Response solution (EDR). Most of these EDRs utilised a technique known as API hooking, modifying function definitions found in Windows Dynamic Link Libraries (DLLs).

After the engagement, we decided to look at one of these well-known EDRs – which we prefer not to name –and implemented some of the common techniques one can use to evade detection in these types of environments.

In this blog, we’ll demonstrate how we were able to use multiple different techniques to bypass API hooking.

Shellcode Encoder/Encryption

The Cobalt Strike shellcode was encoded using a simple XOR encoding with a static key. This was sufficient to bypass the EDR static analysis.

Even though we did not look at other encoding methods, we also recommend Shikata-Ga-Nai polymorphic XOR encoder implementation. For .NET, AES encryption also can be a good candidate as its implementation is easier to be written using .NET language instead of native C language.

Detecting API Unhooking

API hooking is a technique used by many EDR or antivirus vendors to monitor the process or code execution in real-time for malicious behaviours. To detect if the EDR implements API hooking, we can simply look at the first few instructions of the function calls that potentially could be hooked by the EDR. These hooked function calls normally consist of those function calls that are used by process injections such as NtOpenProcess, NtCreateThread or NtCreateUserProcess.

If the following process injection below is used, we can see API is successfully hooked. The EDR’s DLLs will be loaded and the execution is blocked.

Figure 1: Process injection using NtOpenProcess

Figure 2: EDR’s DLLs loaded for inspection

The following is an example of the original instructions on the NtOpenProcess function call within ntdll.dll when the execution is not hooked by the EDR.

Figure 3: Function call (NtOpenProcess)

The following is an example of the NtOpenProcess function call within ntdll.dll that’s hooked by the EDR using jmp instruction on the same address to change execution flow to point to EDR’s code to detect suspicious behaviours when the EDR is enabled.

Figure 4: Hooked function call (NtOpenProcess)

To identify all the potential hooked function calls, we used a tool called Telemetry Sourcerer. This tool would compare the first few bytes of the instructions of the function calls during execution with the clean version of the same functions calls within the system DLLs.

Figure 5 – Telemetry Sourcerer

Technique 1 – Avoiding the Hooked APIs

Once we had figured out which function calls can be intercepted by the EDR, we could use code execution or process injection techniques to avoid these function calls. The following is an example of a shellcode execution technique via the CreateThreadpoolWait API, which would not be hooked by the EDR:

Figure 6: Shellcode execution using CreateThreadpoolWait

Figure 7: CreateThreadpoolWait API not hooked by EDR

Technique 2: Patching the hooked function call

Another technique is to restore the original instructions of the function calls by patching the few bytes of the instructions that had been overwritten by the EDR.

As we can see in Figure 8, the first 16 bytes of the memory address for NtOpenProcess function call (7FFD749CD220) were overwritten by the EDR to redirect the function’s execution flow to the EDR’s code. If we can patch these memory addresses with the same instructions as the original instructions shown in Figure 9, we can prevent the EDR from redirecting the execution flow to the EDR.

Figure 8: NtOpenProcess function call hooked by EDR

Figure 9: NtOpenProcess function call not hooked by EDR

The following is the section of the code where the instructions within NtOpenProcess function call are patched so it could be restored to its original instructions.

Figure 10: NtOpenProcess function call restored

Technique 3: Full DLL Unhooking

As the previous technique may require you to patch more than one function call, as multiple function calls could be hooked at the same time, we can simply use the full DLL unhooking technique as explained by the @spotheplanet on his iredteam blog.  This technique essentially relies on replacing the code of ntdll that resides in memory (and had been tampered with by the EDR) with the version stored on disk, which contains the original instructions.

In the following example, the OpenProcess is used to get a handle of the remote process. OpenProcess itself calls NtOpenProcess which is hooked by the EDR.

 Figure 11: NtOpenProcess function call hooked by EDR

The EDR could be bypassed after the full API unhooking code is added into the process injection code below:

Figure 12: Full DLL unhooking

Technique 4: Cobalt Strike reflective DLL injection

This technique was discovered by Stefan Fewer and could be used to load the library from memory into a host process. The ReflectiveLoader will process the newly loaded copy of its image’s import table, loading any additional libraries and resolving their respective imported function addresses. The advantage of this technique is the library itself is not registered on the host system and could potentially be used to bypass memory scanning and API hooking.  

Let’s modify the Reflective DLL injection source code and add our process injection within the DllMain. In this case, OpenProcess is used as we’re aware that this function call should be normally hooked by most of the EDRs if executed as a standalone executable.

Figure 13 Reflective DLL injection

The injector itself (inject.c) also is found to utilise OpenProcess function call which should be inspected by the EDR.

Figure 14 Reflective DLL injection loader

By observing the execution using API monitor, we see the OpenProcess itself calls NtOpenProcess and this function call was successfully inspected by the EDR’s DLLs.

Figure 15: EDR’s DLLs loaded for inspection

Reflective DLL injection is heavily used by Cobalt Strike for its post-exploitation. Let’s use the Cobalt Strike dllinject module with a default Malleable profile to perform Reflective DLL injection into a process using the same DLL file we generated previously.

As the DLL is not hosted on the victim machine and would be injected through the C2 communication channel into the memory, we are no longer having to worry about the DLL being detected through static or sandbox analysis.

Figure 16: Cobalt Strike Reflective DLL injection

Our injected DLL now had been successfully injected into the memory. There was no OpSec consideration here as the permissions for the (read, write, and execute), as depicted in the capture below.

Figure 17 – Injected DLL in Memory

The EDR’s DLLs were found to be successfully loaded and our DLL entry-point function was successfully called. We believe the injector did a good job of not exposing dangerous calls that could potentially be detected by the EDR.

Figure 18: Loaded EDR’s DLLs

It was identified that sometimes not all processes are protected by the EDR. The Reflective DLL injection module on Cobalt Strike allows us to choose any process we can inject. Even though it was not necessary, it would still be safer to just inject our DLL or shellcode into the process that is not protected by the EDR.

The following is an example of one of the running processes within Windows 10 which was not protected by the EDR.

Figure 19: Loaded EDR’s DLLs  

By observing the injected process using API Monitor, it was confirmed that no inspection was conducted by the EDR’s DLLs on these processes.

Figure 20: EDR’s DLLs not loaded for inspection

Technique 5: D/Invoke

Dynamic invocation (D/Invoke) is a technique to dynamically invoke unmanaged code without using P/Invoke to evade API hooking. P/Invoke is a technology to invoke managed code from unmanaged code and is commonly used by malware developers to execute shellcode using .NET language.

D/Invoke was originally used as part of the SharpSploit project by loading the DLL at runtime and the function is called using a pointer to its location in the memory to avoid suspicious P/Invokes that might be picked up by EDR.

DInvoke was designed by @TheRealWover. It supports the manual mapping of PE modules on the disk. This may allow us to load the fresh version of DLL and execute the payload to bypass the API hooking.

We looked at the sample of the D/Invoke code that utilises this technique and found an EDR evasion tool call Inceptor designed by @KlezVirus. One of its templates was found to use the DLL mapping technique and should fit our purpose.

However, the main issue we found is the DLL (DInvoke.dll) generated by the Dinvoke project or Nuget package is always has been signatured by many EDR vendors.

In the following template used from the Inceptor tool, the relevant fresh version of ntdll.dll is copied and loaded into the memory manually.

Figure 21: D/Invoke manual mapping

For the process injection, the NtOpenProcess, NtAllocateVirtualMemory, NtWriteVirtualMemory and NtProtectVirtualMemory and NtCreateThreadEx are dynamically invoked to execute our shellcode.

Figure 22: Process injection using delegates

As expected, the generated executable was detected by the EDR when compiled and our process injection was successfully blocked. We clearly could see the generated executable and its DInvoke.dll dependencies that were merged via ILMerge was considered malicious when scanned.

Figure 23: DInvoke.dll

Fortunately, Inception comes with an obfuscation feature and we could simply use ConfuserEx to obfuscate our binary. This was sufficient enough to bypass the EDR.

Figure 24: Dinvoke.dll After obfuscation

Now, let’s monitor this execution using API Monitor. The EDR’s DLLs were successfully loaded but the NTOpenProcess function call was not hooked.

Figure 25: NTOpenProcess function call was not hooked by EDR

Technique 6: Execute-Assembly

While looking at the .NET tradecraft against our EDR, we found that it was possible to load .NET assemblies in the memory using Cobalt Strike with execute-assembly without being blocked.

Figure 26: Fork & run with Execute-Assembly

As Execute-Assembly would spawn a new child process (rundll32.exe is used by default Malleable profile) using fork & run and this process was protected by EDR’s DLLs. However, our .NET assemblies were successfully loaded without being detected or blocked by the EDR.

Figure 27: Loaded EDR’s DLLs

The loaded .NET assemblies also clearly showed that our sacrificial process successfully loaded the Seatbelt offensive tool. No OpSec such as patching the ETW was required to avoid detection.

Figure 28: Loaded .NET assemblies

Even though it was not required, it was possible to block the EDR’s DLLs from being imported into the sacrificial process by using the Cobalt Strike BlockDLLs module. Cobalt Strike BlockDLLs module is used to protect the spawned process from loading non-Microsoft signed DLL; this should fit the criteria for the EDR as its DLLs are not signed by Microsoft.

Figure 29: Cobalt Strike blockdlls module

Figure 30: EDR’s DLLs not loaded for inspection

Sign up to receive insights from our experts

Get the latest news and developments from Advania delivered to your inbox

Other blog articles that might interest you

Driven by client success

We’re proud to work with the some of the most ambitious and innovative organisations.


Sign up to receive insights from our experts

Get the latest news and developments from Advania delivered to your inbox.