TensorFlow tf.profile计算FLOPs与论文值不符，寻求技术帮助

Hey there, let's dig into why your TF Profiler FLOPs results don't match the paper's numbers—this is a common gotcha with different FLOPs counting definitions and tooling behavior!

Key Reasons for the Mismatch #

• Definition Differences: Papers typically count only forward-pass core operations (convolution, fully connected layer multiply-adds) and often treat one "multiply-add" (w*x + b) as 1 FLOP. TensorFlow's Profiler, by default, counts all floating-point operations—including Batch Norm calculations, ReLU operations, and separates multiplies and adds (so one multiply-add counts as 2 FLOPs).
• Model Structure Gaps: The tf.slim resnet_v1_50 defaults to num_classes=None, which returns the final residual block output (no global average pooling or 1000-class fully connected layer). The original ResNet-50 paper includes this FC layer, and skipping it adds to the discrepancy.
• Profiler Scope: The default cmd='op' counts every operation in the graph (like placeholder setup, arg_scope overhead) that papers never include.

Fixed Code to Align with Paper Results #

Here's adjusted code to filter non-core operations and match paper-style counting:

For ResNet-v1-50

run_meta = tf.RunMetadata()
im = tf.placeholder(tf.float32, [1, 224, 224, 3])

# Enable the 1000-class fully connected layer to match the paper
with arg_scope(resnet_v1.resnet_arg_scope(use_batch_norm=True)):
    ims, endpoints = resnet_v1.resnet_v1_50(im, num_classes=1000, is_training=False)

print(get_num_of_params(tf.get_default_graph()))

# Customize profiler to only count Conv2D and MatMul (FC layer) operations
opts = tf.profiler.ProfileOptionBuilder.float_operation()
opts['select'] = ['float_ops']
opts['show_name_regexes'] = ['.*Conv2D.*', '.*MatMul.*']  # Filter non-core ops

flops = tf.profiler.profile(tf.get_default_graph(), run_meta=run_meta, cmd='op', options=opts)
total_flops = flops.total_float_ops

# Convert to paper-style FLOPs (treat multiply-add as 1 FLOP)
paper_aligned_flops = total_flops / 2
print(f"ResNet-50 Paper-Aligned FLOPs: {paper_aligned_flops / 1e9:.2f} GFLOPs")

For VGG-19

from tensorflow.contrib.slim.nets import vgg

run_meta = tf.RunMetadata()
im = tf.placeholder(tf.float32, [1, 224, 224, 3])

with arg_scope(vgg.vgg_arg_scope()):
    ims, endpoints = vgg.vgg_19(im, num_classes=1000, is_training=False)

opts = tf.profiler.ProfileOptionBuilder.float_operation()
opts['select'] = ['float_ops']
opts['show_name_regexes'] = ['.*Conv2D.*', '.*MatMul.*']

flops = tf.profiler.profile(tf.get_default_graph(), run_meta=run_meta, cmd='op', options=opts)
paper_aligned_flops = flops.total_float_ops / 2
print(f"VGG-19 Paper-Aligned FLOPs: {paper_aligned_flops / 1e9:.2f} GFLOPs")

Extra Notes #

• Setting is_training=False switches Batch Norm to inference mode, removing training-only ops that add unnecessary FLOPs to the count.
• Some papers ignore bias addition entirely, but dividing by 2 will get you extremely close to the published numbers for most standard models.

内容的提问来源于stack exchange，提问作者Z Zheng