如何在TensorFlow中复制logits层及其所有依赖张量与操作？

Absolutely, this is totally doable! Let’s break down how to pull this off for the two most common frameworks—TensorFlow and PyTorch—since the approach varies a bit depending on which you’re using.

TensorFlow Implementation #

TensorFlow’s graph structure (whether static in v1 or eager with tracked operations in v2) lets you traverse backwards from the logits tensor to collect all dependencies, then rebuild them in a new scope to avoid name clashes.

Step 1: Collect All Dependent Operations & Tensors #

First, map out every tensor and operation that feeds into your logits. We’ll traverse recursively from the logits tensor back to the input placeholders/variables:

def collect_dependencies(tensor):
    visited_ops = set()
    visited_tensors = set()
    
    def traverse(t):
        if t in visited_tensors:
            return
        visited_tensors.add(t)
        if hasattr(t, 'op'):
            op = t.op
            if op not in visited_ops:
                visited_ops.add(op)
                for input_tensor in op.inputs:
                    traverse(input_tensor)
    
    traverse(tensor)
    return visited_ops, visited_tensors

Step 2: Recreate the Subgraph in a New Scope #

Next, rebuild the collected subgraph in a dedicated scope. You can choose to either reuse the original weights (for shared parameters) or create new ones (for independent copies). Here’s how to make an independent copy:

def copy_subgraph(source_tensor, target_scope="logits_copy"):
    visited_ops, _ = collect_dependencies(source_tensor)
    tensor_map = {}  # Maps original tensors to their copied counterparts
    
    with tf.name_scope(target_scope):
        # First, duplicate all variables from the original subgraph
        for op in visited_ops:
            if op.type == 'VariableV2':
                var = op.outputs[0]
                copied_var = tf.Variable(var.numpy(), name=f"{var.name.split(':')[0]}_copy")
                tensor_map[var] = copied_var
        
        # Recreate all non-variable operations in topological order
        for op in tf.compat.v1.get_default_graph().get_operations():
            if op in visited_ops and op.type != 'VariableV2':
                # Get copied versions of the operation's inputs
                copied_inputs = [tensor_map.get(t, t) for t in op.inputs]
                # Recreate the operation using its original attributes
                copied_op = tf.raw_ops.__dict__[op.type](*copied_inputs, **op.get_attrs())
                # Map original outputs to the new copied outputs
                for orig_out, copied_out in zip(op.outputs, copied_op.outputs):
                    tensor_map[orig_out] = copied_out
    
    # Return the copied logits tensor
    return tensor_map[source_tensor]

PyTorch Implementation #

PyTorch uses dynamic graphs, so we’ll leverage torch.fx—PyTorch’s built-in tool for tracing and manipulating computation graphs—to extract and copy the logits subgraph.

Step 1: Trace the Logits Subgraph #

First, trace the part of the graph that leads to your logits tensor. We’ll use torch.fx to capture all operations and dependencies:

import torch
import torch.fx as fx

def trace_logits_subgraph(model, sample_input, logits_tensor):
    # Trace the entire model first, then extract the subgraph leading to logits
    tracer = fx.Tracer()
    traced_full_graph = tracer.trace(model, (sample_input,))
    
    # Find the node in the traced graph that produces the logits
    logits_node = None
    for node in traced_full_graph.nodes:
        if node.output == logits_tensor:
            logits_node = node
            break
    
    # Collect all nodes that are dependencies of the logits node
    dependent_nodes = set()
    def traverse_dependencies(node):
        if node in dependent_nodes:
            return
        dependent_nodes.add(node)
        # Check all arguments for nested nodes
        for arg in node.args:
            if isinstance(arg, fx.Node):
                traverse_dependencies(arg)
        for kwarg_val in node.kwargs.values():
            if isinstance(kwarg_val, fx.Node):
                traverse_dependencies(kwarg_val)
    
    traverse_dependencies(logits_node)
    
    # Build a new graph containing only the dependent nodes
    new_graph = fx.Graph()
    node_map = {}
    for node in dependent_nodes:
        new_node = new_graph.node_copy(node, lambda n: node_map[n])
        node_map[node] = new_node
    
    # Return a GraphModule that represents the copied subgraph
    return fx.GraphModule(model, new_graph)

Step 2: Create an Independent Copy #

Once you have the traced subgraph, you can deep-copy it to get independent weights (or reuse it directly for shared weights):

import copy

# Assume `model` is your original network, `sample_input` is a valid input tensor,
# and `original_logits` is the logits tensor from model(sample_input)
traced_subgraph = trace_logits_subgraph(model, sample_input, original_logits)
copied_subgraph = copy.deepcopy(traced_subgraph)

# Now you can use the copied subgraph to generate copied logits
copied_logits = copied_subgraph(sample_input)

General Tips #

• Shared vs. Independent Weights: If you want the copied subgraph to share weights with the original, skip the deep copy (PyTorch) or variable duplication (TensorFlow) steps.
• Topological Order: Always process operations in the order they’re executed (inputs first, then dependent ops) to avoid errors when rebuilding the graph.
• Framework Versions: For TensorFlow v2, you might need to adjust the graph traversal to work with eager execution (using tf.GradientTape to track operations if needed).

内容的提问来源于stack exchange，提问作者Vlad