Python 2.7升级至3.10后json.dump执行失败的原因排查及解决方法

Hey there, let's work through this issue you're facing after upgrading to Python 3.10. The error message gives us a clear clue, so let's break down what's happening and how to fix it.

Root Cause #

The core problem here is Python 3's strict separation between str (Unicode text) and bytes (binary data)—a distinction Python 2 didn't enforce. When you upgraded and used 2to3 to convert your scripts, some values (likely in field_value_dictionary, or even existing fields in ogdata) ended up as bytes objects instead of str. The Python 3 json module can't serialize bytes by default, which is why json.dump() fails.

Troubleshooting Steps to Pinpoint the Issue #

Before jumping to fixes, let's confirm exactly where the bytes values are hiding:

• Inspect field_value_dictionary directly: Add this code right before assigning it to ogdata['results']['fields'] to check each value's type:

  for key, value in field_value_dictionary.items():
      print(f"Key: {key}, Value Type: {type(value)}")
  

• Recursively scan the entire ogdata object: Nested structures can hide bytes values, so use a helper function to dig through everything:

  def find_bytes(obj, path=""):
      if isinstance(obj, bytes):
          print(f"Found bytes at {path}: {obj}")
      elif isinstance(obj, dict):
          for k, v in obj.items():
              find_bytes(v, f"{path}.{k}")
      elif isinstance(obj, list):
          for i, item in enumerate(obj):
              find_bytes(item, f"{path}[{i}]")
  
  find_bytes(ogdata)
  

• Double-check file loading: While you confirmed the file content is correct, quickly verify that json.load() isn't returning unexpected bytes (though this is rare—Python 3's json.load() should return str for text values by default).

Fixes to Resolve the Serialization Error #

Once you've identified the bytes values, here are two reliable ways to fix the issue:

1. Recursively Convert Bytes to Strings #

Write a helper function to traverse your data structure and convert all bytes objects to str using the correct encoding (most likely utf-8, adjust if your data uses another encoding like gbk):

def convert_bytes_to_str(obj):
    if isinstance(obj, bytes):
        return obj.decode('utf-8')  # Match your data's actual encoding
    elif isinstance(obj, dict):
        return {k: convert_bytes_to_str(v) for k, v in obj.items()}
    elif isinstance(obj, list):
        return [convert_bytes_to_str(item) for item in obj]
    else:
        return obj  # Leave other data types unchanged

# Apply the conversion before dumping to JSON
ogdata = convert_bytes_to_str(ogdata)
with open(param_response_result_file, 'w+') as outfile:
    json.dump(ogdata, outfile)
    outfile.write("\n")

2. Use a Custom JSON Encoder #

If you prefer not to modify the original data structure, create a custom encoder that handles bytes during serialization:

import json

class BytesHandlingEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, bytes):
            return obj.decode('utf-8')  # Use the correct encoding for your data
        # Let the base encoder handle all other data types
        return super().default(obj)

# Pass the custom encoder to json.dump()
with open(param_response_result_file, 'w+') as outfile:
    json.dump(ogdata, outfile, cls=BytesHandlingEncoder)
    outfile.write("\n")

Bonus: Fix the Source of Bytes Values #

If you can track down where field_value_dictionary is generating bytes (e.g., from file reads in binary mode, subprocess outputs, or leftover Python 2-style code), fix that at the source. For example, if reading from a file, use 'r' mode instead of 'rb', or decode bytes immediately when you receive them.

内容的提问来源于stack exchange，提问作者J. Carrillo