如何解决PInvokeStackImbalance错误？鼠标钩子DLL调用问题排查

问题描述 #

在DLL中编写MouseHook类，通过PInvoke调用user32.dll的钩子相关函数；在另一个项目引用该DLL并实例化MouseHook后，触发PInvokeStackImbalance错误，提示托管PInvoke签名与非托管目标签名不匹配。

错误信息：

Managed Debugging Assistant 'PInvokeStackImbalance' : 'A call to PInvoke function 'BarcodeTestApp!BarcodeTestApp.Form1+MouseHook::CallNextHookEx' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.'

错误原因 #

• 参数传递类型不匹配：CallNextHookEx的第四个参数lParam在非托管API中是指向MSLLHOOKSTRUCT的指针（LPARAM类型），但原代码中声明为按值传递的结构体ByVal lParam As MSLLHOOKSTRUCT，导致栈布局与非托管函数预期不符，直接引发栈不平衡。
• 32/64位兼容性缺失：原代码使用Integer表示句柄、指针类型（如hmod、hHook），在64位系统下Integer仅为32位，无法容纳64位指针，会造成数据截断，进一步加剧参数传递错误。
• 钩子委托签名不符：原钩子委托的返回值和参数类型未与非托管钩子过程完全匹配，比如返回值用Integer而非对应LRESULT的IntPtr，在64位环境下会导致类型长度不匹配。

修复方案 #

1. 修正PInvoke函数签名 #

使用DllImport替代旧的Declare语法，显式指定调用约定，并将指针/句柄类型替换为IntPtr以适配32/64位系统；修正CallNextHookEx的参数类型为指针类型。

2. 调整钩子委托与回调逻辑 #

修改委托的返回值和参数类型，匹配非托管钩子过程；在回调函数中通过Marshal.PtrToStructure将指针封送为结构体，正确读取钩子数据。

3. 更新调用代码 #

将模块句柄的类型改为IntPtr，避免不必要的类型转换错误。

修改后的完整代码 #

MouseHook类代码

Imports System.Runtime.InteropServices

Public Class MouseHook
    <DllImport("user32.dll", CallingConvention:=CallingConvention.StdCall, CharSet:=CharSet.Auto)>
    Private Shared Function SetWindowsHookEx(ByVal idHook As Integer, ByVal lpfn As MouseProcDelegate, ByVal hmod As IntPtr, ByVal dwThreadId As Integer) As IntPtr
    End Function

    <DllImport("user32.dll", CallingConvention:=CallingConvention.StdCall)>
    Private Shared Function CallNextHookEx(ByVal hHook As IntPtr, ByVal nCode As Integer, ByVal wParam As IntPtr, ByVal lParam As IntPtr) As IntPtr
    End Function

    <DllImport("user32.dll", CallingConvention:=CallingConvention.StdCall)>
    Private Shared Function UnhookWindowsHookEx(ByVal hHook As IntPtr) As Boolean
    End Function

    Private Delegate Function MouseProcDelegate(ByVal nCode As Integer, ByVal wParam As IntPtr, ByVal lParam As IntPtr) As IntPtr

    <StructLayout(LayoutKind.Sequential)>
    Private Structure MSLLHOOKSTRUCT
        Public pt As Point
        Public mouseData As Integer
        Public flags As Integer
        Public time As Integer
        Public dwExtraInfo As IntPtr
    End Structure

    Public Enum Wheel_Direction
        WheelUp
        WheelDown
    End Enum

    Private Const HC_ACTION As Integer = 0
    Private Const WH_MOUSE_LL As Integer = 14
    Private Const WM_MOUSEMOVE As Integer = &H200
    Private Const WM_LBUTTONDOWN As Integer = &H201
    Private Const WM_LBUTTONUP As Integer = &H202
    Private Const WM_LBUTTONDBLCLK As Integer = &H203
    Private Const WM_RBUTTONDOWN As Integer = &H204
    Private Const WM_RBUTTONUP As Integer = &H205
    Private Const WM_RBUTTONDBLCLK As Integer = &H206
    Private Const WM_MBUTTONDOWN As Integer = &H207
    Private Const WM_MBUTTONUP As Integer = &H208
    Private Const WM_MBUTTONDBLCLK As Integer = &H209
    Private Const WM_MOUSEWHEEL As Integer = &H20A

    Private MouseHook As IntPtr
    Private MouseHookDelegate As MouseProcDelegate

    Public Event Mouse_Move(ByVal ptLocat As Point)
    Public Event Mouse_Left_Down(ByVal ptLocat As Point)
    Public Event Mouse_Left_Up(ByVal ptLocat As Point)
    Public Event Mouse_Left_DoubleClick(ByVal ptLocat As Point)
    Public Event Mouse_Right_Down(ByVal ptLocat As Point)
    Public Event Mouse_Right_Up(ByVal ptLocat As Point)
    Public Event Mouse_Right_DoubleClick(ByVal ptLocat As Point)
    Public Event Mouse_Middle_Down(ByVal ptLocat As Point)
    Public Event Mouse_Middle_Up(ByVal ptLocat As Point)
    Public Event Mouse_Middle_DoubleClick(ByVal ptLocat As Point)
    Public Event Mouse_Wheel(ByVal ptLocat As Point, ByVal Direction As Wheel_Direction)

    Public Sub New(windowHandle As IntPtr)
        MouseHookDelegate = New MouseProcDelegate(AddressOf MouseProc)
        MouseHook = SetWindowsHookEx(WH_MOUSE_LL, MouseHookDelegate, windowHandle, 0)
    End Sub

    Private Function MouseProc(ByVal nCode As Integer, ByVal wParam As IntPtr, ByVal lParam As IntPtr) As IntPtr
        If nCode = HC_ACTION Then
            Dim hookStruct As MSLLHOOKSTRUCT = Marshal.PtrToStructure(Of MSLLHOOKSTRUCT)(lParam)
            Select Case wParam.ToInt32()
                Case WM_MOUSEMOVE
                    RaiseEvent Mouse_Move(hookStruct.pt)
                Case WM_LBUTTONDOWN
                    RaiseEvent Mouse_Left_Down(hookStruct.pt)
                Case WM_LBUTTONUP
                    RaiseEvent Mouse_Left_Up(hookStruct.pt)
                Case WM_LBUTTONDBLCLK
                    RaiseEvent Mouse_Left_DoubleClick(hookStruct.pt)
                Case WM_RBUTTONDOWN
                    RaiseEvent Mouse_Right_Down(hookStruct.pt)
                Case WM_RBUTTONUP
                    RaiseEvent Mouse_Right_Up(hookStruct.pt)
                Case WM_RBUTTONDBLCLK
                    RaiseEvent Mouse_Right_DoubleClick(hookStruct.pt)
                Case WM_MBUTTONDOWN
                    RaiseEvent Mouse_Middle_Down(hookStruct.pt)
                Case WM_MBUTTONUP
                    RaiseEvent Mouse_Middle_Up(hookStruct.pt)
                Case WM_MBUTTONDBLCLK
                    RaiseEvent Mouse_Middle_DoubleClick(hookStruct.pt)
                Case WM_MOUSEWHEEL
                    Dim wDirection As Wheel_Direction
                    If (hookStruct.mouseData >> 16) < 0 Then
                        wDirection = Wheel_Direction.WheelDown
                    Else
                        wDirection = Wheel_Direction.WheelUp
                    End If
                    RaiseEvent Mouse_Wheel(hookStruct.pt, wDirection)
            End Select
        End If
        Return CallNextHookEx(MouseHook, nCode, wParam, lParam)
    End Function

    Protected Overrides Sub Finalize()
        If MouseHook <> IntPtr.Zero Then
            UnhookWindowsHookEx(MouseHook)
        End If
        MyBase.Finalize()
    End Sub
End Class

调用代码

Public Class Form1
    Private WithEvents mHook As MouseHook
    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        Dim moduleHandle As IntPtr = Marshal.GetHINSTANCE(System.Reflection.Assembly.GetExecutingAssembly.GetModules()(0))
        mHook = New MouseHook(moduleHandle)
        Timer1.Start()
    End Sub
End Class

内容的提问来源于stack exchange，提问作者Mesos