使用ObjectMapper.readTree()处理API数据时字符编码异常求助

我在开发游戏王（Yu-Gi-Oh!）游戏时，从https://db.ygoprodeck.com/api/v7/cardinfo.php获取卡牌数据并写入SQL Server数据库。遇到的问题是：用BufferedReader读取API数据流并打印时，特殊字符（比如●，对应Unicode \u25cf）显示正常，但通过ObjectMapper.readTree()将数据转换为JsonNode后，该特殊字符被替换为问号?。

问题代码 #

ApiEngine.java

public class ApiEngine {
    /**
     * This function will collect all cards data from 'db.ygoprodeck.com/api/v7/cardinfo.php?'
     *@return the result of API research that is a JSonNode type, null otherwise
     * 
     */
    public static JsonNode getCardData()
    {
        try 
        {
             URL url = new URL("https://db.ygoprodeck.com/api/v7/cardinfo.php?name=astral kuriboh");

             HttpURLConnection conn = (HttpURLConnection) url.openConnection();
             
             conn.setRequestMethod("GET");
             
             conn.connect();

             //Check if connect is made
             int responseCode = conn.getResponseCode();

             
             if (responseCode != 200) 
             {
                 throw new RuntimeException("HttpResponseCode: " + responseCode);
             } 
             else 
             {
                BufferedReader br = new BufferedReader(new InputStreamReader( url.openStream(), StandardCharsets.UTF_8));    
           

                /*String strCurrentLine;
                
                while ((strCurrentLine = br.readLine()) != null) {
                        System.out.println(strCurrentLine);
                    }*/
                    
       
                ObjectMapper mapper = new ObjectMapper();
                
                JsonNode cardData = mapper.readTree(br);
                
                            
                return cardData;
             }
         } 
        catch (Exception e) 
        {
             e.printStackTrace();
             
             return null;
        }
    }
}

ExporterCardDataEngine.java

public class ExporterCardDataEngine {
    /**
     * This function will export API data to database
     * @throws SQLException 
     */
    public static void exportApiData() throws SQLException
    {
        JsonNode cards = ApiEngine.getCardData().get("data");
        
        Connection connexion = DatabaseEngine.connect();
        
        int size = cards.size();
        
        for (int i = 0; i < size; i++)
        {
            System.out.println((i + 1) +"/" + size + cards.get(i).get("desc"));
            
            DatabaseEngine.insertCard(connexion, cards.get(i));
        }
    }
    
    public static void main(String args[])
    {
        try {
            ExporterCardDataEngine.exportApiData();
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }
}

问题原因 #

虽然创建InputStreamReader时指定了StandardCharsets.UTF_8，但通过BufferedReader中转给ObjectMapper时，可能存在流读取状态干扰，或Jackson处理Reader时的隐性编码转换导致字符丢失。更可靠的方式是让Jackson直接处理原始输入流，避免中间Reader的潜在问题。

修复方案 #

方案1：直接让ObjectMapper读取URL（推荐）

Jackson的ObjectMapper支持直接读取URL，会自动处理HTTP响应的编码和流，代码更简洁可靠：

public static JsonNode getCardData() {
    try {
        URL url = new URL("https://db.ygoprodeck.com/api/v7/cardinfo.php?name=astral kuriboh");
        ObjectMapper mapper = new ObjectMapper();
        // 直接读取URL，Jackson自动处理编码
        return mapper.readTree(url);
    } catch (Exception e) {
        e.printStackTrace();
        return null;
    }
}

方案2：显式用UTF-8编码的InputStreamReader读取流

如果需要保留HttpURLConnection的自定义配置（比如请求头），则直接将带UTF-8编码的InputStreamReader传给ObjectMapper，跳过BufferedReader中转：

public static JsonNode getCardData() {
    try {
        URL url = new URL("https://db.ygoprodeck.com/api/v7/cardinfo.php?name=astral kuriboh");
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        conn.setRequestMethod("GET");
        conn.connect();

        int responseCode = conn.getResponseCode();
        if (responseCode != 200) {
            throw new RuntimeException("HttpResponseCode: " + responseCode);
        } else {
            ObjectMapper mapper = new ObjectMapper();
            // 直接用UTF-8编码的InputStreamReader读取，跳过BufferedReader
            try (InputStreamReader isr = new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8)) {
                return mapper.readTree(isr);
            }
        }
    } catch (Exception e) {
        e.printStackTrace();
        return null;
    }
}

额外注意事项 #

• 确保SQL Server数据库的表字段使用nvarchar类型（支持Unicode字符），或设置编码为UTF-8，避免写入数据库时再次丢失特殊字符。
• 如果API返回压缩数据（比如gzip），可添加请求头并使用GZIPInputStream处理：

conn.setRequestProperty("Accept-Encoding", "gzip");
// 读取流时添加GZIP包装
try (InputStream inputStream = new GZIPInputStream(conn.getInputStream());
     InputStreamReader isr = new InputStreamReader(inputStream, StandardCharsets.UTF_8)) {
    return mapper.readTree(isr);
}

内容的提问来源于stack exchange，提问作者Lyna DJELMOUDI