Hadoop输入分片大小是否一致？实际不一致的原因咨询

Great question—this is a super common point of confusion when first working with Hadoop's input processing! Let’s break down why you might see uneven Input Split sizes, even though the Hadoop Definitive Guide mentions consistent split sizes by default.

First, let’s clarify the key misunderstanding: the "consistent split size" from the guide applies to single large files with default configurations. In real-world scenarios, several factors lead to variable split sizes, and none of them are random—here’s what’s happening:

• Multiple input files (especially small ones): Hadoop doesn’t split individual small files into smaller splits, even if they’re way smaller than your HDFS block size. If your input directory has files of varying sizes (e.g., 20MB, 150MB, 50MB), each small file will become its own split, and larger files will be split into chunks matching (or close to) the block size. This immediately creates splits of different sizes.
• Non-default InputFormats: If you’re using a custom InputFormat (or even built-in ones like SequenceFileInputFormat for structured data, or formats for columnar storage), the split logic is tailored to the data type. For example, some formats split based on record boundaries or data blocks instead of strict HDFS block sizes, leading to uneven splits.
• Compressed files: Not all compression formats are splittable. For example, gzip files can’t be split—so an entire 300MB gzip file becomes one single split, even if your block size is 128MB. Splittable formats like bzip2 or LZO (with indexes) will split along block sizes, but non-splittable ones force full-file splits.
• Record boundary enforcement: Even with the default TextInputFormat, Hadoop ensures splits don’t cut mid-way through a line of text. This means the actual split size might be slightly larger or smaller than the block size to avoid breaking records, leading to minor variations.

To directly answer your question: the system does NOT generate splits randomly. Every split’s size is determined by a strict calculation in the InputFormat’s getSplits() method—you’re just seeing the result of real-world input conditions overriding the "ideal" consistent split scenario from the guide.

If you want to get more consistent split sizes, try these tweaks:

• Use CombineTextInputFormat to merge small files into larger splits, configuring mapreduce.input.fileinputformat.split.maxsize and mapreduce.input.fileinputformat.split.minsize to set your desired split range.
• Consolidate small input files into larger ones before processing (e.g., using HDFS concat or a pre-processing job).
• Choose splittable compression formats if you’re working with compressed data.

内容的提问来源于stack exchange，提问作者Daniel Lee