RCE Endeavors 😅

This post will cover the topic of hooking DirectX in a running application. This post will cover DirectX9 specifically, but the general technique applies to any version. A previous and similar post covered virtual table hooking for DirectX10 and DirectX11 (with minor adjustments). Unlike the previous post, this one aims to establish a technique to hook running DirectX applications. This means that it can be installed at any time, unlike the previous technique, which required starting a process in a suspended state and then hooking to get the device pointer.

Motivations

The motivations are similar to the previous post. By hooking the DirectX device, we can inspect or change the properties of rendered scenes (i.e. depth testing, object colors), overlay text or images, better display visual information, or do anything else with the scene. However, to achieve anything beyond the basics, it also takes a lot of effort in reverse engineering the actual application; simply having access to the rendered scene won’t get you too far.

Typically when hooking DirectX, there are several popular options:

• Hook IDirect3D9::CreateDevice and store the IDirect3DDevice9 pointer that is initialized when the function returns successfully. This needs to be done when the process is started in a suspended state, otherwise the device will have already been initialized.
• Perform a byte pattern scan in memory for the signature of IDirect3DDevice9::EndScene, or any other DirectX function.
• Create a dummy IDirect3DDevice9 instance, read its virtual table, find the address of EndScene, and hook at the target site.
• Look for the CD3DBase::EndScene symbol in d3d9.dll and get its address.

Each one has its drawbacks, but my personal preference is the last option. It’s the one that offers the greatest reliability for the least amount of overhead code. The code for it is pretty straightforward, with the help of the Windows debugging APIs:

const DWORD_PTR GetAddressFromSymbols()
{
    BOOL success = SymInitialize(GetCurrentProcess(), nullptr, true);
    if (!success)
    {
        fprintf(stderr, "Could not load symbols for process.\n");
        return 0;
    }
 
    SYMBOL_INFO symInfo = { 0 };
    symInfo.SizeOfStruct = sizeof(SYMBOL_INFO);
 
    success = SymFromName(GetCurrentProcess(), "d3d9!CD3DBase::EndScene", &symInfo);
    if (!success)
    {
        fprintf(stderr, "Could not get symbol address.\n");
        return 0;
    }
 
    return (DWORD_PTR)symInfo.Address;
}

Once the address is retrieved, it’s simply a matter of installing the hook and writing code in the new hook function. The Hekate engine was used for hook installation/removal, making the code simple:

const bool Hook(const DWORD_PTR address, const DWORD_PTR hookAddress)
{
    pHook = std::unique_ptr<Hekate::Hook::InlineHook>(new Hekate::Hook::InlineHook(address, hookAddress));
 
    if (!pHook->Install())
    {
        fprintf(stderr, "Could not hook address 0x%X -> 0x%X\n", address, hookAddress);
    }
 
    return pHook->IsHooked();
}

The EndScene function was chosen specifically due to how DirectX9 applications are developed. For those unfamiliar with DirectX, the flow of rendering a scene generally goes as follows: BeginScene -> Draw the scene -> EndScene -> Present. Other DirectX9 hook implementations hook Present instead of EndScene, it becomes a matter of preference unless the target application does something special. In the example application, some text is overlaid on top of the scene:

HRESULT WINAPI EndSceneHook(void *pDevicePtr)
{
    using pFncOriginalEndScene = HRESULT (WINAPI *)(void *pDevicePtr);
    pFncOriginalEndScene EndSceneTrampoline =
        (pFncOriginalEndScene)pHook->TrampolineAddress();
 
    IDirect3DDevice9 *pDevice = (IDirect3DDevice9 *)pDevicePtr;
    ID3DXFont *pFont = nullptr;
 
    HRESULT result = D3DXCreateFont(pDevice, 30, 0, FW_NORMAL, 1, false,
        DEFAULT_CHARSET, OUT_DEFAULT_PRECIS, ANTIALIASED_QUALITY,
        DEFAULT_PITCH | FF_DONTCARE, L"Consolas", &pFont);
    if (FAILED(result))
    {
        fprintf(stderr, "Could not create font. Error = 0x%X\n", result);
    }
    else
    {
        RECT rect = { 0 };
        (void)SetRect(&rect, 0, 0, 300, 100);
        int height = pFont->DrawText(nullptr, L"Hello, World!", -1, &rect,
            DT_LEFT | DT_NOCLIP, -1);
        if (height == 0)
        {
            fprintf(stderr, "Could not draw text.\n");
        }
        (void)pFont->Release();
    }
 
    return EndSceneTrampoline(pDevicePtr);
}

Building as a DLL and injecting into the running application should show the text overlay (below):

Hekate supports clean unhooking, so unloading the DLL should remove the text and let the application continue undisturbed.

Code

The Visual Studio 2015 project for this example can be found here. The source code is viewable on Github here. The Hekate static library dependency is included in a separate download here and goes into the DirectXHook/lib folder. Capstone Engine is used as a runtime dependency, so capstone_x86.dll/capstone_x64.dll in DirectXHook/thirdparty/capstone/lib should be put in the same directory that the target application is running from.

Thanks for reading and follow on Twitter for more updates