Hey there, I’ve absolutely encountered this exact problem with Core ML models before—you’re definitely not the only one scratching their head over this!

From what I’ve gathered through testing and digging into how Core ML interacts with the Apple Neural Engine (ANE), the core issue boils down to optimization limitations with variable input shapes. Here’s why your detection model is hitting that 60ms mark vs. 20ms for the fixed shape:

• ANE relies on fixed-shape pre-compilation: The ANE is built to optimize heavily for static input dimensions. When your model has a fixed input shape (3×416×416), Core ML can pre-compile the entire computation pipeline specifically for that size—locking in memory layouts, operation sequences, and hardware-specific optimizations that let the ANE run at full speed.
• Variable shapes force runtime flexibility (and overhead): When you modify the model to use variable input dimensions with coremltools, Core ML can’t pre-optimize those fixed pipelines anymore. Even if you feed in the exact same 3×416×416 input, the runtime still has to handle the possibility of different sizes, which means it can’t leverage the ANE’s full fixed-shape optimizations. In some cases, parts of the computation might even fall back to the CPU/GPU instead of running entirely on the ANE, adding extra latency.
• Target detection models are particularly sensitive: Detection models often have layers that depend heavily on spatial dimensions (like anchor boxes or feature pyramid networks). Variable shapes disrupt the pre-computed optimizations for these layers more than they would for simpler models like image classifiers.

If you need to support variable input sizes but want to mitigate the slowdown, a few workarounds I’ve tried include:

• Restricting variable dimension ranges: Instead of making the input fully arbitrary (e.g., 3×…×…), define a limited range of supported sizes (like 3×[416, 640]×[416, 640]) using coremltools. This lets Core ML pre-optimize for those specific ranges, reducing runtime overhead compared to fully open variable shapes.
• Resizing inputs to a fixed shape at runtime: If your use case allows, resize all incoming images to the model’s original fixed size (416×416) instead of using variable shapes. This keeps the ANE optimizations intact while still handling different input sizes via pre-processing.

Hope this helps clarify what’s going on—this is a super common pain point in Core ML development!

内容的提问来源于stack exchange，提问作者YiZhaoYanBo