Azure Custom Vision下标注文本与汽车部件关联方案咨询

Hey there! Let’s work through how to link those annotation texts (like "9. BXCU12") to their corresponding car parts (like Bonnet) since you’ve already got the part detection sorted with Azure Custom Vision. Here are some practical, beginner-friendly approaches:

1. 关联标注法：把文本和部件绑定在标注阶段 #

Azure Custom Vision excels at object detection, but we need to add a layer of association during the labeling process:

• When you label a part (e.g., Bonnet) with a bounding box, also draw a box around its corresponding annotation text (e.g., "9. BXCU12"). Use a shared identifier in their labels—like tagging the part as Bonnet_9 and the text as Text_9 (the number 9 acts as the linking marker).
• After training, when you run inference, you can pair detections by matching the shared identifier in their labels.

Step-by-Step Example: #

1. Labeling Phase: On your blueprint, draw a box around the Bonnet and assign the label Component_Bonnet_9. Then draw a box around "9. BXCU12" and assign Label_9.
2. Inference Phase: After calling the Custom Vision API, loop through all detected objects. Match any Component_* label with the Label_* that shares the same suffix (e.g., Component_Bonnet_9 pairs with Label_9). Your output would then be: "9. BXCU12指向Bonnet".

2. OCR + Position Matching：用文本提取加位置关联 #

If you don’t want to rework your existing labels, combine Custom Vision with OCR (Optical Character Recognition) to connect text and parts:

1. First, use Custom Vision to detect all car parts and record their bounding box coordinates (x, y, width, height).
2. Use an OCR tool (like Azure Computer Vision’s OCR feature) to extract all text from the blueprint, along with each text block’s bounding box.
3. Match via spatial logic: Blueprints usually have arrows pointing from text to parts, so the text block is often near the arrow’s start, and the part is near the end. Calculate the distance between text blocks and part boxes, or use directional cues (e.g., text is left/above the part) to pair them.

Quick Pseudo-Code Snippet: #

# Assume we already have results from Custom Vision and OCR
cv_detections = get_custom_vision_results(image_file)
ocr_text_blocks = get_ocr_results(image_file)

# Pair each component with its closest relevant text
for component in cv_detections:
    comp_center = (component.bbox.x + component.bbox.width/2, component.bbox.y + component.bbox.height/2)
    closest_text = None
    min_distance = float('inf')
    
    for text in ocr_text_blocks:
        text_center = (text.bbox.x + text.bbox.width/2, text.bbox.y + text.bbox.height/2)
        # Calculate Euclidean distance between centers
        distance = ((comp_center[0] - text_center[0])**2 + (comp_center[1] - text_center[1])**2)**0.5
        # Add directional check (e.g., text is to the left of the component, matching arrow direction)
        if distance < min_distance and text.bbox.x < component.bbox.x:
            min_distance = distance
            closest_text = text.content
    
    # Output the matched pair
    print(f"{closest_text}指向{component.tag_name}")

3. Advanced: Train a Joint Matching Model (Optional) #

If you’re ready to dive a bit deeper, you could build a simple classification model on Azure ML that takes the position features of detected parts and text blocks, then outputs whether they’re a matched pair. But for a beginner, the first two methods are way more straightforward and get the job done.

Pro Tips: #

• Keep your annotation text format consistent (e.g., always "Number. Code")—this makes it easier to extract the shared identifier after OCR.
• If arrows are prominent in your blueprints, train a tiny Custom Vision model just to detect arrows. You can then use the arrow’s start and end points to supercharge your pairing accuracy.

内容的提问来源于stack exchange，提问作者Kiran