使用PolyHook x64Detour无法Hook Direct3DCreate9，pDevice未赋值

Let's break down why your pDevice variable is never getting populated and fix the issues step by step:

Key Issues in Your Code #

1. Missing Error Checking for Hook Setup #

You're assuming the hook installs successfully, but you never verify the return values of SetupHook() or Hook(). If either call fails (e.g., invalid function address, memory protection conflicts), your hook won't activate—and hkDirect3DCreate9 will never run to assign pDevice.

2. Hook Timing Might Be Too Late #

If you inject the DLL into a process that already called Direct3DCreate9 before your hook was installed, your hooked function will never trigger. Most games/apps initialize Direct3D early in their lifecycle, so you need to inject before this initialization happens (e.g., via process creation injection with CREATE_SUSPENDED, or using a loader that injects at startup).

3. Compiler Optimization & Thread Safety #

Your global pDevice variable isn't marked as volatile, which means the compiler might optimize away your loop checking !pDevice—it could assume the value never changes since no modifications happen inside the loop's scope.

4. Unnecessary/Cluttered Code #

You have an unused pointer otDirect3DCreate9 declared, which doesn't cause the issue but adds clutter. Also, while your function pointer type matches __stdcall correctly for Direct3DCreate9, it's easy to overlook calling convention mismatches that break hook functionality.

Fixed Code Implementation #

Here's the revised code with fixes for all the above issues:

#include "PolyHook.hpp"
#define CATCH_CONFIG_MAIN
#include "CatchUnitTest.h"
#include <d3d9.h>
#include <iostream>
#include <cstdio>

// Correct function pointer type for Direct3DCreate9
typedef IDirect3D9*(__stdcall* tDirect3DCreate9)(UINT SDKVersion);
tDirect3DCreate9 oDirect3DCreate9 = nullptr;

// Mark as volatile to prevent compiler optimization across threads
volatile LPDIRECT3D9 pDevice = nullptr;

LPDIRECT3D9 __stdcall hkDirect3DCreate9(UINT SDKVersion) {
    // Call original function and capture the device instance
    pDevice = oDirect3DCreate9(SDKVersion);
    std::cout << "Hook triggered! pDevice address: " << pDevice << std::endl;
    return pDevice;
}

DWORD WINAPI inject() {
    AllocConsole();
    freopen_s((FILE**)stdout, "CONOUT$", "w", stdout);
    std::cout << "Starting injection setup..." << std::endl;

    HMODULE d3dModule = GetModuleHandleA("d3d9.dll");
    if (!d3dModule) {
        std::cout << "Failed to get d3d9.dll handle!" << std::endl;
        return 1;
    }

    FARPROC originalFunc = GetProcAddress(d3dModule, "Direct3DCreate9");
    if (!originalFunc) {
        std::cout << "Failed to locate Direct3DCreate9!" << std::endl;
        return 1;
    }

    PLH::Detour* d3dHook = new PLH::Detour();
    // Validate hook setup
    if (!d3dHook->SetupHook((BYTE*)originalFunc, (BYTE*)&hkDirect3DCreate9)) {
        std::cout << "SetupHook failed! Error: " << d3dHook->GetLastError() << std::endl;
        delete d3dHook;
        return 1;
    }

    // Validate hook activation
    if (!d3dHook->Hook()) {
        std::cout << "Hook activation failed! Error: " << d3dHook->GetLastError() << std::endl;
        delete d3dHook;
        return 1;
    }

    // Retrieve original function pointer
    oDirect3DCreate9 = d3dHook->GetOriginal<tDirect3DCreate9>();
    if (!oDirect3DCreate9) {
        std::cout << "Failed to get original function pointer!" << std::endl;
        delete d3dHook;
        return 1;
    }

    std::cout << "Hook installed successfully. Waiting for pDevice..." << std::endl;
    while (!pDevice) {
        // Add sleep to reduce CPU usage
        Sleep(100);
        std::cout << "Waiting for pDevice initialization..." << std::endl;
    }

    std::cout << "Success! pDevice: " << pDevice << std::endl;

    // Keep thread alive (clean up properly in production code)
    while (true) {
        Sleep(1000);
    }

    // Cleanup (unreachable here, but good practice for production)
    d3dHook->UnHook();
    delete d3dHook;
    return 0;
}

BOOL APIENTRY DllMain(HMODULE hModule, DWORD Reason, LPVOID lpReserved) {
    switch (Reason) {
        case DLL_PROCESS_ATTACH:
            DisableThreadLibraryCalls(hModule);
            CreateThread(nullptr, 0, inject, nullptr, 0, nullptr);
            break;
        case DLL_PROCESS_DETACH:
            // Optional: Unhook here if needed to clean up
            break;
    }
    return TRUE;
}

Additional Recommendations #

• Inject Early: Use a process injection method that injects your DLL when the target process starts (e.g., CreateProcess with CREATE_SUSPENDED, inject, then resume the process). This ensures your hook is active before Direct3DCreate9 is called.
• Match Project Settings: Ensure your project uses the same architecture (x86/x64) and runtime library (e.g., Multi-threaded DLL) as PolyHook to avoid compatibility issues.
• Thread Safety: If you plan to use pDevice across multiple threads, add a mutex or atomic operations to prevent race conditions.

内容的提问来源于stack exchange，提问作者Plixxer