Let's break down the issues you're facing and fix them step by step:

Core Problems in Your Current Setup #

1. Limited Class Loader Scope: Your agent only targets classes from the Application ClassLoader because you weren't configuring ByteBuddy to handle Bootstrap/Extension class loaders correctly (the old enableBootstrapInjection() method was replaced, not removed entirely).
2. Uncontrolled Instrumentation: Ignoring too few classes leads to infinite recursion (e.g., instrumenting PrintStream.println() which your advice uses) and JVM crashes from instrumenting sensitive core JDK classes.

Fixed Agent Implementation #

Here's a revised version of your agent that addresses both issues:

AgentBuilder mybuilder = new AgentBuilder.Default()
    // Critical: Exclude classes that cause recursion or JVM instability
    .ignore(
        nameStartsWith("net.bytebuddy.")
        .or(nameStartsWith("src.Agent")) // Avoid instrumenting your own agent
        .or(nameStartsWith("java.lang.System"))
        .or(nameStartsWith("java.io.PrintStream"))
        .or(nameStartsWith("java.io.PrintWriter"))
        .or(nameMatches("java.lang.reflect.*")) // Skip reflection classes to avoid issues
    )
    .disableClassFormatChanges()
    .with(RedefinitionStrategy.RETRANSFORMATION)
    .with(InitializationStrategy.NoOp.INSTANCE)
    .with(TypeStrategy.Default.REDEFINE)
    // Enable Bootstrap ClassLoader injection (replaces old enableBootstrapInjection())
    .bootstrapInjection();

mybuilder.type(nameMatches(".*")) // Target ALL classes (filtered by ignore rules above)
    .transform((builder, type, classLoader, module) -> {
        try {
            return builder
                .visit(Advice.to(TraceAdvice.class).on(isMethod()));
        } catch (SecurityException e) {
            e.printStackTrace();
            return builder; // Never return null! Return original builder to avoid class loading failures
        }
    })
    // Add a listener to debug instrumentation errors (optional but helpful)
    .with(new AgentBuilder.Listener.Simple() {
        @Override
        public void onError(String typeName, ClassLoader classLoader, JavaModule module, boolean loaded, Throwable throwable) {
            System.err.println("Failed to instrument: " + typeName);
            throwable.printStackTrace();
        }
    })
    .installOn(inst);
System.out.println("Agent installed successfully");

Key Improvements Explained #

1. Bootstrap ClassLoader Support:
   The .bootstrapInjection() method replaces the deprecated enableBootstrapInjection(). It injects your TraceAdvice class into the Bootstrap ClassLoader, allowing classes loaded by Bootstrap/Extension class loaders to access the advice code (which was impossible before, as those class loaders can't see classes from the Application ClassLoader).

2. Precise Ignore Rules:
   We've expanded the ignore list to exclude:

   • ByteBuddy's own classes (to avoid conflicts)
   • Your agent class (prevents self-instrumentation)
   • Core JDK classes that trigger recursion (PrintStream, System)
   • Reflection classes (reduces risk of JVM crashes from instrumenting sensitive APIs)

3. Safe Error Handling:
   Returning null from the transform method causes class loading failures. Instead, return the original builder to let the class load normally even if instrumentation fails.

4. Debug Listener:
   The added listener prints detailed errors when instrumentation fails, which helps you identify which classes are causing issues (instead of guessing from a JVM crash).

Advice Class Best Practices #

To avoid future recursion issues, modify your TraceAdvice to use a non-instrumented logging mechanism if possible. For example, if you need logging, use a file-based logger instead of System.out:

public class TraceAdvice {
    private static final FileWriter writer;

    static {
        try {
            writer = new FileWriter("instrumentation.log", true);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    @Advice.OnMethodEnter
    static void onEnter(@Origin Method method) {
        try {
            writer.write("[+] Entered: " + method.getDeclaringClass().getName() + "." + method.getName() + "\n");
            writer.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Advice.OnMethodExit
    static void onExit(@Origin Method method) {
        try {
            writer.write("[-] Exited: " + method.getDeclaringClass().getName() + "." + method.getName() + "\n");
            writer.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

This way, you avoid triggering your own instrumentation logic while logging.

内容的提问来源于stack exchange，提问作者AntoineG