关于以二进制处理S3上Parquet文件实现数值匿名化的技术疑问

Great question—this gets into some of the nitty-gritty of how Parquet stores data, so let's unpack it clearly.

1. Can you scan Parquet files as binary to locate a target value, and does compression block this? #

Short answer: No, you can't reliably locate the target numeric value via binary scanning, and compression makes this even more impossible. Here's why:

• Parquet is a columnar, binary-encoded format—not plaintext. Numeric values are stored using compact binary encodings (like variable-length integers for ints, IEEE 754 for floats) instead of their ASCII string representations. So the "1234" you see in Athena isn't stored as the bytes 0x31 0x32 0x33 0x34; it's packed into a 4-byte integer or similar.
• Parquet often uses encoding optimizations like RLE (Run-Length Encoding) or dictionary encoding to reduce size. This means identical values are stored once and referenced, so your target value might not even appear in the raw bytes multiple times.
• Compression (Snappy, Gzip, etc.) scrambles the raw byte structure of the file. Compressed data is a single contiguous block (or split into blocks) where the original byte patterns are completely unrecognizable—you can't search for a specific value without decompressing first.

2. If you could locate the value, could you modify one digit in binary without breaking the Parquet file? #

Even if you somehow managed to find the exact byte pattern of your target value (which we've established is unlikely), modifying it directly would almost certainly corrupt the file. Here's the reasoning:

• Parquet files have strict metadata structures, including column statistics (min/max values for each block), checksum values, and schema definitions. Changing a single value would make the metadata inconsistent with the actual data—readers like Athena would either fail to parse the file or return incorrect results.
• Numeric encodings are not "human-readable" at the byte level. For example, changing one byte in a 4-byte integer could turn 1234 into a completely unrelated number (like 1234 + 256 if you modify the second byte) or even an invalid value that breaks the encoding format.
• Many Parquet readers rely on the integrity of the file structure; even a tiny byte modification can throw off the offset pointers that tell the reader where data blocks start and end, leading to parsing errors.

The Right Way to Anonymize Parquet Data on S3 #

Instead of messing with binary files, use tools designed to work with Parquet:

• Use Athena CTAS statements: Write a query that selects all columns, modifies the target numeric field (e.g., CASE WHEN id = 'target-id' THEN my_value + 1 ELSE my_value END to adjust a digit), and creates a new table with the anonymized data stored in S3.
• Use Spark or Pandas: Load the Parquet files from S3, modify the target record's field programmatically, then write the updated data back to S3 as a new Parquet file. This ensures the file structure and metadata remain valid.

内容的提问来源于stack exchange，提问作者Nir