关于perf/strace/ltrace的性能影响、生产环境安全性及ltrace致进程崩溃问题的咨询

Hey there, let's break down your questions one by one based on practical experience with system tracing tools:

1. Do perf/strace/ltrace have performance impacts on the tracee process? #

Absolutely, but the impact varies a lot between tools:

• strace: It intercepts every system call the process makes. For I/O-heavy processes (like those doing lots of file reads/writes or network calls), this can slow things down significantly—sometimes even making the process run orders of magnitude slower. For CPU-bound processes with few system calls, the impact is much smaller.
• ltrace: It tracks user-space library function calls, which are usually far more frequent than system calls. So its overhead is often higher than strace, especially for processes that rely heavily on standard library functions (like printf, malloc, etc.).
• perf: This is the lightest of the three by default. It uses the kernel's performance event subsystem with sampling (not full interception). Unless you crank up the sampling rate to an extreme or enable heavy features like full call stack tracing, perf has almost no noticeable impact on the target process.

2. Is it safe to use perf/strace/ltrace in a production environment? #

Safety depends entirely on the tool and your use case:

• perf: Generally safe for production. Its default sampling mode is non-intrusive, and it's widely used to diagnose performance issues in live production systems. Just avoid overly aggressive settings (like 1us sampling intervals) that could add unnecessary overhead.
• strace: Proceed with caution. Short-term tracing of low-load, non-critical processes might be okay, but using it on high-traffic or latency-sensitive production services can cause severe delays, timeouts, or even service degradation. Also, strace generates massive log files—make sure you have enough disk space to avoid filling up partitions.
• ltrace: Strongly not recommended for production. It works by inserting breakpoints into the target process's memory, which is inherently risky. As you saw in your dev environment, it can easily crash processes, especially in complex scenarios like multi-threaded apps or those handling signals. There's too much risk of disrupting critical services.

3. Why did ltrace crash the tracee process? How to interpret the dmesg message? What is trap int3? #

Let's start with decoding that dmesg line:

[Fri Aug 2 11:02:43 2024] traps: writer1[4137194] trap int3 ip:4092c1 sp:7f1eda53db58 error:0 in service_prog[400000+1ea000]

• writer1[4137194]: The name and PID of the crashed process.
• trap int3: The process triggered an int3 trap. On x86 systems, int3 is a special debug breakpoint instruction that forces the CPU to throw a debug exception.
• ip:4092c1: The instruction pointer (the memory address where the crash happened).
• sp:7f1eda53db58: The stack pointer address at the time of the crash.
• in service_prog[400000+1ea000]: The crash address falls within the service_prog executable—its base memory address is 400000, and it's 1ea000 bytes in size.

Now, why did ltrace cause this?
Ltrace uses the ptrace system call to insert int3 breakpoints at the entry points of library functions in the target process. When the process hits a breakpoint, ltrace logs the function call, then replaces the int3 instruction with the original code so the process can continue.

This process can fail in several ways:

• Multi-thread race conditions: If the target process is multi-threaded, one thread might execute the int3 breakpoint while ltrace is still in the middle of setting it up or restoring the original instruction.
• Signal handling conflicts: If the process is handling a signal when ltrace inserts a breakpoint, it can disrupt the signal handling flow and lead to an unhandled exception.
• Self-modifying code: If the process uses self-modifying code (like JIT compilers or packed binaries), ltrace's breakpoints can get overwritten or interfere with the process's normal execution.
• Recovery errors: Sometimes ltrace fails to properly restore the original instruction after a breakpoint, leaving the int3 in place. When the process hits it again without ltrace handling it, the kernel treats it as an unhandled debug trap and crashes the process.

In short, ltrace's breakpoint-based approach is fragile in complex process environments, which is why it's prone to causing crashes.

备注：内容来源于stack exchange，提问作者Xiaoyong Guo