Mask RCNN架构疑问：掩码预测是否仅使用FPN最后一层特征图？

Great question—this is a common point of confusion from the paper's simplified diagrams, so let's break down how FPN integrates with mask prediction in Mask R-CNN clearly:

First, a quick recap of FPN's role: FPN generates a pyramid of feature maps (typically labeled P2 to P6, where P2 is the highest-resolution layer, downsampled 2x from the input, and P6 is the lowest-resolution, downsampled 64x).

Key Detail: RoI-to-FPN Layer Matching #

Mask prediction does NOT only use the final FPN layer. Instead, each Region of Interest (RoI) is assigned to the specific FPN feature layer that best matches its size:

• Small RoIs (e.g., tiny objects) are mapped to higher-resolution layers like P2, which preserve fine-grained spatial details critical for accurate small masks.
• Larger RoIs are mapped to lower-resolution layers like P5, which capture broader contextual features that work better for bigger objects.

Once an RoI is matched to the right FPN layer, ROIAlign is used to extract a fixed-size feature map from that layer. This feature map is then fed into the mask prediction branch (a small stack of convolutional layers) to generate the final object mask.

Why the Paper Diagram Might Mislead #

The paper's diagram simplifies this multi-scale matching process for readability, so it doesn't explicitly show each RoI being routed to different FPN layers. But the original Mask R-CNN paper explicitly states:

Each RoI is assigned to the level of the feature pyramid that best matches its scale, and the RoI feature is extracted from that level via ROIAlign.

This design is how Mask R-CNN fully leverages FPN's strengths—ensuring every object, no matter its size, uses the most appropriate feature scale for mask prediction, which directly boosts overall mask accuracy compared to relying on a single feature layer.

内容的提问来源于stack exchange，提问作者Shiro