如何解决WinHttpReq导入Unicode数据时FoxPro字符乱码问题

问题背景 #

用WinHttpRequest从Nordigen导入含Windows-1257编码字符的银行对账单时，FoxPro会自动将带重音的字符替换为问号。添加COMPROP(WinHttpReq,"UTF8",1)后问题有所缓解，但WinHttpReq.ResponseText里的重音字符变成了4字节乱码（前3字节固定，第4字节随字符不同变化）。目前只能手动替换少量字符，需要通用的转换方案。

当前使用的请求代码 #

WinHttpReq = CREATEOBJECT("WinHttp.WinHttpRequest.5.1")
WinHttpReq.Open("GET", "https://ob.nordigen.com/api/v2/accounts/11111-22222-333333-4444-/transactions/", .f.)
WinHttpReq.setrequestheader('accept', 'application/json')
WinHttpReq.setrequestheader('Authorization', "Bearer afde554a5sd45a45as4")
WinHttpReq.Send()
* 移除该行后，带重音的字符会显示为单个问号：
COMPROP(WinHttpReq ,"UTF8",1)

仅支持少量字符的手动替换函数 #

（注：原代码中strt应为FoxPro字符串替换函数strtran，已修正）

Function NordigetConvert(cStr)
cstr = strtran( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x9c) , 'Ü' ) 
cstr = strtran( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0xBC) , 'ü' ) 
cstr = strtran( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x84) , 'Ä' ) 
cstr = strtran( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0xB6) , 'ö' ) 
return cStr

尝试过但无效的方法 #

1. 嵌套使用STRCONV转换，预期返回Ü但得到两个问号：

STRCONV(strconv(CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x9c) ,10) ,11, 1257, 1)

2. 调用WideCharToMultiByte的代码，结果同样返回两个问号：

SET ECHO OFF
SET TALK OFF
CLEAR

#DEFINE CF_UNICODETEXT  13
DO decl
 
LOCAL hData, lcUnicode

MESSAGEBOX( uconv (CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x9c), 1257)  )

* end of main
 
FUNCTION uconv (lcSrc, lnCodePage)
    LOCAL lcDst, lnUsedDefault, lnResult
    lcDst = Repli(Chr(0), Len(lcSrc) * 2)
    lnUsedDefault = 0
 
    lnResult = WideCharToMultiByte (lnCodePage, 0,;
        lcSrc, Len(lcSrc),;
        @lcDst, Len(lcDst), "?", 0)
 
    IF lnResult = 0
    *   87 - ERROR_INVALID_PARAMETER
    *  122 - ERROR_INSUFFICIENT_BUFFER
    * 1004 - ERROR_INVALID_FLAGS
        ? "Error code:", GetLastError()
        lcDst = ""
    ELSE
        lcDst = SUBSTR(lcDst, 1, lnResult)
    ENDIF
RETURN SUBSTR(lcDst, 1, AT(Chr(0),lcDst)-1)
 
FUNCTION  memwchar2str (lnMemBlock)
* copies Unicode characters (two-byte) from a memory address to a VFP string
RETURN  mem2str(lnMemBlock, Chr(0)+Chr(0))
 
FUNCTION  mem2str(lnMemBlock, end_sequence)
#DEFINE BUFFER_SIZE   16
#DEFINE EMPTY_BUFFER  Repli(Chr(0), BUFFER_SIZE)
 
    DECLARE RtlMoveMemory IN kernel32 As Heap2Str;
        STRING @, INTEGER, INTEGER
 
    LOCAL lnPtr, lcResult, lcBuffer, lnPos
    lnPtr = lnMemBlock
    lcResult = ""
 
    DO WHILE .T.
        lcBuffer = EMPTY_BUFFER
        = Heap2Str (@lcBuffer, lnPtr, BUFFER_SIZE)
        lnPos = AT(end_sequence, lcBuffer)
        IF lnPos > 0
            lcResult = lcResult + SUBSTR(lcBuffer, 1, lnPos-1)
            RETURN  lcResult
        ELSE
            lcResult = lcResult + lcBuffer
            lnPtr = lnPtr + BUFFER_SIZE
        ENDIF
    ENDDO
 
PROCEDURE decl
    DECLARE INTEGER GetLastError IN kernel32
    DECLARE INTEGER OpenClipboard IN user32 INTEGER hwnd
    DECLARE INTEGER CloseClipboard IN user32
    DECLARE INTEGER GetClipboardData IN user32 INTEGER uFormat
    DECLARE INTEGER IsClipboardFormatAvailable IN user32 INTEGER wFormat
 
    DECLARE INTEGER WideCharToMultiByte IN kernel32;
        INTEGER CodePage, INTEGER dwFlags, STRING lpWideCharStr,;
        INTEGER cchWideChar, STRING @lpMultiByteStr, INTEGER cbMultiByte,;
        STRING lpDefaultChar, INTEGER lpUsedDefaultChar

解决方案 #

出现4字节乱码的核心原因是双重UTF-8编码：Nordigen返回的是Windows-1257编码内容，开启COMPROP(WinHttpReq,"UTF8",1)后，WinHttpRequest错误地将Windows-1257字节当作UTF-8解码成Unicode，相当于把原始字符先转成UTF-8再被二次解码，最终导致每个字符变成4字节乱码。

方案一：直接获取原始字节流转换（推荐） #

不要开启UTF8属性，直接读取原始二进制字节流，再用STRCONV转换为Windows-1257编码的字符串：

WinHttpReq = CREATEOBJECT("WinHttp.WinHttpRequest.5.1")
WinHttpReq.Open("GET", "https://ob.nordigen.com/api/v2/accounts/11111-22222-333333-4444-/transactions/", .f.)
WinHttpReq.setrequestheader('accept', 'application/json')
WinHttpReq.setrequestheader('Authorization', "Bearer afde554a5sd45a45as4")
WinHttpReq.Send()

* 获取原始二进制响应，直接转换为Windows-1257编码字符串
lcRawBytes = WinHttpReq.ResponseBody
lcCorrectStr = STRCONV(lcRawBytes, 11, 1257)

方案二：保留UTF8属性时的修正转换 #

如果因其他需求必须开启UTF8属性，可以先将乱码字符串还原为原始UTF-8字节，再重新解码为Windows-1257字符串：

WinHttpReq = CREATEOBJECT("WinHttp.WinHttpRequest.5.1")
WinHttpReq.Open("GET", "https://ob.nordigen.com/api/v2/accounts/11111-22222-333333-4444-/transactions/", .f.)
WinHttpReq.setrequestheader('accept', 'application/json')
WinHttpReq.setrequestheader('Authorization', "Bearer afde554a5sd45a45as4")
WinHttpReq.Send()
COMPROP(WinHttpReq ,"UTF8",1)

* 从乱码字符串还原原始UTF-8字节，再解码为Windows-1257
lcMangledStr = WinHttpReq.ResponseText
lcUtf8Bytes = STRCONV(lcMangledStr, 10)
lcCorrectStr = STRCONV(lcUtf8Bytes, 11, 1257)

原理说明 #

• ResponseBody返回的是未经过任何编码转换的原始二进制数据，直接用STRCONV(...,11,1257)可以准确将Windows-1257字节转换为FoxPro可识别的字符串。
• 开启UTF8属性时，ResponseText会错误地将Windows-1257字节当作UTF-8解码，导致字符被扩展为4字节乱码。此时通过STRCONV(...,10)将乱码字符串转回UTF-8字节，再按Windows-1257解码，即可还原正确字符。

内容的提问来源于stack exchange，提问作者Andrus