Hey there, this is such a relatable pain point—anyone who’s dabbled in person re-ID knows how quickly accuracy tanks when people change clothes. Let’s break down your questions one by one, then get into actionable steps and tooling recommendations.

First, your core technical questions #

1. Are most image-based re-ID models inherently clothing-dependent? #

Short answer: Yes, for the most part. The majority of off-the-shelf re-ID models (like those trained on Market1501 or DukeMTMC) are optimized to learn the most discriminative features available in their training data—and clothing is way easier to pick up than subtle body shape or proportions. These models don’t inherently "care" about biological traits; they just learn whatever lets them split classes fastest. So if your training data doesn’t force them to ignore clothing, they’ll lean hard on it.

2. Is clothing-invariant recognition possible with just static RGB (no gait)? #

Absolutely—though it’s not a "perfect" solution, it’s definitely practical for many real-world use cases (like office building access, retail monitoring, etc.). The key is to train models to focus on intrinsic biological traits instead of clothing:

• Body morphology: Shoulder width, torso-to-leg ratio, hip shape, overall frame size
• Consistent skin tone patterns (even on arms/neck if face is covered)
• Head shape, hairstyle (if visible), or permanent markers like scars/tattoos
• Static posture tendencies (e.g., someone who always stands with one shoulder slightly higher)

You won’t get 100% accuracy in every edge case (e.g., full face cover + baggy clothes that hide all body shape), but you can get to a point where it’s reliable for controlled or semi-controlled environments.

3. Is multi-modal fusion (face + body) the right direction? And are static RGB fundamentally limited? #

Multi-modal fusion is exactly the right call—this is where you’ll get the biggest gains. If the face is visible, face embeddings (from models like ArcFace) are already clothing-agnostic and super discriminative. For back/partial views, you lean on body shape and morphological features.

That said, static RGB does have hard limits. For example:

• If someone’s entire body is covered (burka, heavy winter coat) and face is hidden, there’s almost no intrinsic data to work with.
• Extreme pose changes can warp perceived body shape enough to throw off matching.

But these are edge cases—for most scenarios where you have a clear view of at least a majority of the body, static RGB can get you very far.

Tooling Recommendations #

• TorchReID: My go-to for re-ID work. It has pre-trained models, easy-to-use training pipelines, and supports custom losses/backbones. You can fine-tune models on clothing-agnostic datasets or add adversarial training to strip out clothing features.
• MediaPipe: Perfect for extracting human keypoints (17+ points for body, 468 for face) to compute morphological features (e.g., shoulder width ratio) that are 100% clothing-independent.
• Detectron2: Great for instance segmentation—you can crop out the exact human region from background, then run feature extraction only on that area to avoid noise.
• FaceNet-PyTorch: If you’re incorporating face embeddings, this library has pre-trained ArcFace/CosFace models that are trivial to use for extracting clothing-agnostic face features.
• OpenCV: For all the foundational preprocessing—histogram equalization to handle lighting, pose alignment using keypoints, and basic body detection.

Architectural & Training Insights #

Here’s a step-by-step approach to build your system:

1. Preprocess to isolate relevant features #

• First, use instance segmentation (Detectron2’s Mask R-CNN) to crop the human from the background—this eliminates distractions like cars, walls, etc.
• Use MediaPipe to extract body keypoints, then normalize the pose (warp the human to a standard standing posture). This ensures that body shape features are consistent across different poses.

2. Train models to ignore clothing #

• Adversarial Training: Add a secondary "clothing classifier" head to your re-ID model. Train the main model to generate embeddings that the classifier can’t use to guess clothing type—this forces the model to discard clothing features and focus on intrinsic traits.
• Contrastive Loss with Hard Positive Pairs: Instead of just using random same-person pairs, explicitly use same-person, different-clothing images as positive pairs in triplet loss or SimCLR-style training. This teaches the model that "same person = similar embedding, regardless of clothes."
• Multi-Feature Fusion: Combine three types of embeddings:
  • Face embedding (ArcFace, if visible)
  • Body shape embedding (from a model trained on normalized poses)
  • Geometric features (keypoint ratios like shoulder-width-to-height, hip-width-to-shoulder-width)
    Use a weighted fusion layer to give higher priority to face embeddings when available, and shape/geometric features when face is hidden.

3. Dataset tips #

• Use clothing-agnostic re-ID datasets to fine-tune pre-trained models: PRCC (Person Re-identification in Changing Clothes) or VC-Clothes are great options—they have thousands of images of people in different outfits.
• If you’re collecting your own data, make sure to capture each person in at least 3-5 different outfits, across different poses/lighting conditions.

Realistic Expectations #

Don’t aim for "perfect" clothing invariance—no static RGB system will handle every possible edge case. But you can absolutely build a system that works reliably for most practical scenarios (e.g., matching a person entering a building in a jacket to their gallery image in a t-shirt). Start small: build a baseline with TorchReID on PRCC, then add pose normalization and adversarial training to boost accuracy.

If you hit specific roadblocks (e.g., struggling to implement adversarial loss in TorchReID), feel free to follow up with code snippets or error details—happy to help troubleshoot!