如何通过除零异常静默检测C++程序中的调试器？

Let's break down why your original code behaves unexpectedly and fix it for silent, reliable detection:

Why Your Original Code Fails #

The div eax instruction (with eax=0) triggers a division-by-zero exception (INT 0). Here's what goes wrong in both scenarios:

• No debugger attached: Your UnhandledExceptionFilter catches the exception, adjusts the instruction pointer to skip the faulty div instruction, and the program continues normally.
• Debugger attached: Windows prioritizes sending exceptions to the debugger first. Your process-level handlers (__try/__except or SetUnhandledExceptionFilter) won't run unless the debugger explicitly passes the exception back to the process. Most debuggers pause execution at the exception, causing your program to freeze or crash if the exception isn't handled by the debugger.

Solution 1: Improved Division-by-Zero Detection #

We can adjust the logic to detect whether the debugger intercepted the exception without crashing or freezing. The key is checking if our exception handler was invoked—if not, a debugger is present.

Here's the fixed code:

#include <windows.h>
#include <iostream>
using namespace std;

bool g_debuggerDetected = false;

LONG WINAPI ExceptionFilter(PEXCEPTION_POINTERS pExcepPointers) {
    // Skip the 2-byte `div eax` instruction
    pExcepPointers->ContextRecord->Eip += 2;
    return EXCEPTION_CONTINUE_EXECUTION;
}

int main() {
    SetUnhandledExceptionFilter(ExceptionFilter);

    bool exceptionWasHandled = false;

    __try {
        __asm {
            xor eax, eax
            div eax
            // We only reach this line if the exception was handled (no debugger)
            mov exceptionWasHandled, 1
        }
    }
    __except (EXCEPTION_EXECUTE_HANDLER) {
        // This block runs only if no debugger caught the exception first
        exceptionWasHandled = true;
    }

    g_debuggerDetected = !exceptionWasHandled;

    if (g_debuggerDetected) {
        cout << "DEBUGGER DETECTED (silent)" << endl;
    } else {
        cout << "NO DEBUGGER FOUND" << endl;
    }

    return 0;
}

How This Works: #

• Without a debugger: The exception is caught by our filter, the faulty instruction is skipped, and exceptionWasHandled is set to true.
• With a debugger: The debugger intercepts the exception before our handlers run. The code after div eax never executes, and the __except block doesn't trigger—leaving exceptionWasHandled as false, confirming a debugger is active.

Solution 2: Using INT 2D (More Reliable Silent Detection) #

The INT 2D instruction is a specialized debugger breakpoint that behaves differently based on whether a debugger is attached:

• No debugger attached: INT 2D triggers an invalid opcode exception (INT 0C), which we can catch.
• Debugger attached: The debugger treats INT 2D as a breakpoint and stops execution unless configured to ignore it—we detect this by checking if our exception handler runs.

Here's the code for this method:

#include <windows.h>
#include <iostream>
using namespace std;

bool g_debuggerDetected = false;

LONG WINAPI ExceptionFilter(PEXCEPTION_POINTERS pExcepPointers) {
    if (pExcepPointers->ExceptionRecord->ExceptionCode == EXCEPTION_ILLEGAL_INSTRUCTION) {
        // Skip the 1-byte `INT 2D` instruction
        pExcepPointers->ContextRecord->Eip += 1;
        return EXCEPTION_CONTINUE_EXECUTION;
    }
    return EXCEPTION_CONTINUE_SEARCH;
}

int main() {
    SetUnhandledExceptionFilter(ExceptionFilter);

    bool exceptionWasHandled = false;

    __try {
        __asm {
            int 0x2D
            mov exceptionWasHandled, 1
        }
    }
    __except (EXCEPTION_EXECUTE_HANDLER) {
        exceptionWasHandled = true;
    }

    g_debuggerDetected = !exceptionWasHandled;

    if (g_debuggerDetected) {
        cout << "DEBUGGER DETECTED (INT 2D method)" << endl;
    } else {
        cout << "NO DEBUGGER FOUND" << endl;
    }

    return 0;
}

Why This Is Better: #

• INT 2D is less likely to be caught by generic debugger exception handlers compared to division-by-zero, making detection more reliable.
• It's a single-byte instruction, so adjusting the instruction pointer is simpler.

Key Notes #

• Both methods run silently—no crashes or freezes, even with a debugger attached.
• Advanced debuggers can bypass these checks (e.g., by modifying exception handling behavior), but they work effectively against standard tools like Visual Studio Debugger or x64dbg.

内容的提问来源于stack exchange，提问作者Segy