Detour Hook拦截修改C++控制台cout输出失败，求解决方案

问题背景 #

需求为通过Detour Hook拦截并修改C++控制台程序的std::cout输出，已编写以下代码：

测试程序（Print.cpp） #

#include <iostream>
#include <Windows.h>
int main()
{
    while (true)
    {
        std::cout << "Hello World!\n";
        Sleep(500);
    }
}

Hook DLL代码（Hook.cpp） #

#include "pch.h"
#include <detours.h>
#include <iostream>
#pragma comment(lib,"detours.lib")

BOOL(WINAPI* OriginalWriteFile)(
    HANDLE       hFile,
    LPCVOID      lpBuffer,
    DWORD        nNumberOfBytesToWrite,
    LPDWORD      lpNumberOfBytesWritten,
    LPOVERLAPPED lpOverlapped
    ) = WriteFile;
BOOL WINAPI HookedWriteFile(
    HANDLE       hFile,
    LPCVOID      lpBuffer,
    DWORD        nNumberOfBytesToWrite,
    LPDWORD      lpNumberOfBytesWritten,
    LPOVERLAPPED lpOverlapped
) {
    HANDLE stdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
    std::cout << "HOOK" << std::endl;
    if (hFile == stdOutput) {
        const char* prefix = "[Hooked]: ";
        std::string newMessage = prefix + std::string((const char*)lpBuffer, nNumberOfBytesToWrite);
        return OriginalWriteFile(hFile, newMessage.c_str(), newMessage.size(), lpNumberOfBytesWritten, lpOverlapped);
    }

    return OriginalWriteFile(hFile, lpBuffer, nNumberOfBytesToWrite, lpNumberOfBytesWritten, lpOverlapped);
}
size_t(__cdecl* OriginalFWrite)(
    const void* buffer,
    size_t size,
    size_t count,
    FILE* stream
    ) = fwrite;

size_t __cdecl HookedFWrite(
    const void* buffer,
    size_t size,
    size_t count,
    FILE* stream
) {
    const char* prefix = "[Hooked]: ";
    std::string newMessage = prefix + std::string((const char*)buffer, size * count);
    return OriginalFWrite(newMessage.c_str(), size, count, stream);
}
std::ostream& (__cdecl* OriginalOstreamOperator)(std::ostream&, const char*) = nullptr;

std::ostream& __cdecl HookedOstreamOperator(std::ostream& os, const char* c)
{
    OutputDebugString(L"asdasdasd");
    const char* prefix = "[Hooked]: ";
    OriginalOstreamOperator(os, prefix); // Adding prefix
    return OriginalOstreamOperator(os, c); // Original call
}


BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
        std::cout << OriginalWriteFile << std::endl;
        OutputDebugString(L"HOOK START");
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        std::cout << OriginalFWrite << std::endl;
        DetourAttach(&(PVOID&)OriginalWriteFile, HookedWriteFile);
        DetourAttach(&(PVOID&)OriginalFWrite, HookedFWrite);
        DetourAttach(&(PVOID&)OriginalOstreamOperator, HookedOstreamOperator);
        DetourTransactionCommit();

        break;
    case DLL_THREAD_ATTACH:

    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourDetach(&(PVOID&)OriginalWriteFile, HookedWriteFile);
        DetourDetach(&(PVOID&)OriginalFWrite, HookedFWrite);
        DetourDetach(&(PVOID&)OriginalOstreamOperator, HookedOstreamOperator);
        DetourTransactionCommit();
        break;
    }
    return TRUE;
}

问题现象 #

尝试Hook WriteFile、fwrite、WriteConsoleA、WriteConsoleW均无效，控制台仍输出原始的Hello World!。通过x32dbg调试确认WriteFile和fwrite确实被调用，但自定义的Hook函数从未触发。程序运行输出如下：

759F3C50
776D6E80
Hello World!
Hello World!
Hello World!
Hello World!

解决方案 #

1. 修复ostream操作符的Hook地址问题 #

代码中OriginalOstreamOperator初始化为nullptr，直接调用DetourAttach会失败，因为未正确获取目标函数地址。需通过模板实例化获取std::operator<<的正确地址，同时匹配编译器的调用约定（MSVC下为__thiscall）：

// 替换原OriginalOstreamOperator定义
std::ostream& (__thiscall* OriginalOstreamOperator)(std::ostream*, const char*) = 
    (std::ostream& (__thiscall*)(std::ostream*, const char*))&std::operator<< <char, std::char_traits<char>>;

// 对应的Hook函数修改为__thiscall调用约定
std::ostream& __thiscall HookedOstreamOperator(std::ostream* os, const char* c)
{
    OutputDebugString(L"Ostream operator hooked!");
    const char* prefix = "[Hooked]: ";
    OriginalOstreamOperator(os, prefix);
    return OriginalOstreamOperator(os, c);
}

2. 切换到正确的底层Hook点 #

std::cout在Windows CRT中最终会调用_write函数，而非直接调用WriteFile，Hook该函数更可靠：

// 定义原始函数指针
int(__cdecl* OriginalWrite)(int, const char*, int) = _write;

// Hook函数实现
int __cdecl HookedWrite(int fd, const char* buf, int count) {
    if (fd == 1) { // 1对应标准输出的文件描述符
        const char* prefix = "[Hooked]: ";
        std::string newMsg = prefix + std::string(buf, count);
        return OriginalWrite(fd, newMsg.c_str(), newMsg.size());
    }
    return OriginalWrite(fd, buf, count);
}

在DllMain的DLL_PROCESS_ATTACH分支中添加DetourAttach(&(PVOID&)OriginalWrite, HookedWrite);。

3. 确保32/64位编译匹配 #

测试程序与Hook DLL必须编译为相同位数（均为32位或64位），否则Detour无法正确修改函数地址。调试时需使用对应位数的调试器（x32dbg对应32位程序，x64dbg对应64位程序）。

4. 调整注入时机 #

若测试程序在DLL注入前已初始化std::cout或调用CRT相关函数，会导致Hook失效。建议使用Detour提供的DetourCreateProcessWithDll函数，在进程启动时注入DLL：

// 单独的注入器程序代码
#include <detours.h>
#pragma comment(lib,"detours.lib")

int main() {
    STARTUPINFO si = { sizeof(si) };
    PROCESS_INFORMATION pi;
    LPCSTR dllPath = "Hook.dll"; // 替换为你的DLL路径
    DetourCreateProcessWithDllA(NULL, "Print.exe", NULL, NULL, FALSE, CREATE_DEFAULT_ERROR_MODE, NULL, NULL, &si, &pi, NULL, dllPath, NULL);
    WaitForSingleObject(pi.hProcess, INFINITE);
    CloseHandle(pi.hProcess);
    CloseHandle(pi.hThread);
    return 0;
}

5. 避免Hook函数递归 #

当前HookedWriteFile中调用std::cout会再次触发WriteFile，导致无限递归。需替换为非控制台输出方式，比如OutputDebugString：

// 替换HookedWriteFile中的std::cout输出
OutputDebugStringA("HOOKED WriteFile called!\n");

内容的提问来源于stack exchange，提问作者Relaxing