如何用GCC将ML64生成的.obj文件编译为Linux下的.so文件？

问题描述 #

我有一个由ml64生成的.obj文件，原本通过VS的.vcxproj编译为Windows DLL，其中C代码仅作为MASM代码中某一函数的透传层。想知道是否可以通过GCC在Linux环境下将其编译为.so文件？

补充信息1：OBJ文件格式与转换错误 #

执行objdump -f test.obj的返回结果：

test.obj :
file format pe-x86-64
architecture: i386:x86-64, flags 0x0000003d :
HAS RELOC, HAS LINENO, HAS DEBUG, HAS SYMS, HAS LOCALS
start address 0X0000000000000000

尝试用objcopy转换格式时，执行objcopy -I pe-x86-64 -O elf64-x86-64 test.obj test.o得到错误：

objcopy: test.o: (null)unsupported
objcopy: test.o: sorry, cannot handle this file

补充信息2：GCC编译输出 #

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04) 
COLLECT_GCC_OPTIONS='-v' '-shared' '-o' 'EmulatorCore.so' '-fPIC' '-Wall' '-g' '-m64' '-mtune=generic' '-march=x86-64' '-dumpdir' 'EmulatorCore.so-'
 /usr/lib/gcc/x86_64-linux-gnu/11/cc1 -quiet -v -imultiarch x86_64-linux-gnu core.c -quiet -dumpdir EmulatorCore.so- -dumpbase core.c -dumpbase-ext .c -m64 -mtune=generic -march=x86-64 -g -Wall -version -fPIC -fasynchronous-unwind-tables -fstack-protector-strong -Wformat-security -fstack-clash-protection -fcf-protection -o /tmp/ccNiOQu1.s
GNU C17 (Ubuntu 11.3.0-1ubuntu1~22.04) version 11.3.0 (x86_64-linux-gnu)
    compiled by GNU C version 11.3.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.24-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/11/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/11/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
GNU C17 (Ubuntu 11.3.0-1ubuntu1~22.04) version 11.3.0 (x86_64-linux-gnu)
    compiled by GNU C version 11.3.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.24-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 3f6cb05d963ad324b8f9442822c95179
COLLECT_GCC_OPTIONS='-v' '-shared' '-o' 'EmulatorCore.so' '-fPIC' '-Wall' '-g' '-m64' '-mtune=generic' '-march=x86-64' '-dumpdir' 'EmulatorCore.so-'
 as -v --gdwarf-5 --64 -o /tmp/ccs6mQj9.o /tmp/ccNiOQu1.s
GNU assembler version 2.38 (x86_64-linux-gnu) using BFD version (GNU Binutils for Ubuntu) 2.38
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/11/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-shared' '-o' 'EmulatorCore.so' '-fPIC' '-Wall' '-g' '-m64' '-mtune=generic' '-march=x86-64' '-dumpdir' 'EmulatorCore.so.'
 /usr/lib/gcc/x86_64-linux-gnu/11/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/11/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper -plugin-opt=-fresolution=/tmp/cc2PLxAY.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -shared -z relro -o EmulatorCore.so /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. /tmp/ccs6mQj9.o core.obj -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/11/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crtn.o
COLLECT_GCC_OPTIONS='-v' '-shared' '-o' 'EmulatorCore.so' '-fPIC' '-Wall' '-g' '-m64' '-mtune=generic' '-march=x86-64' '-dumpdir' 'EmulatorCore.so.'

调用约定包装器代码 #

为适配调用约定，我写了一个简单的C包装器：

#include <stdint.h>

extern int32_t asm_func(void* state);

int wrapper(void *state)
{
  int32_t toReturn = 0;

  __asm__ volatile(
     "mov %rdi, %rsi 	
"
     "call asm_func  	
");

  return toReturn;
}

int32_t fnEmulatorCode(void* state) {
  return wrapper(state);
}

核心问题 #

你遇到的主要障碍是：ml64生成的是PE格式的x86_64目标文件，而Linux使用ELF格式，GNU binutils的objcopy目前无法完整转换这类PE OBJ到ELF格式，尤其是包含MASM特定汇编逻辑的文件。

可行方案 #

1. 重新汇编MASM代码为ELF格式（最可靠）
   直接拿到原始的MASM汇编代码（.asm文件），用Linux下的汇编器重新编译：

   • 用NASM：nasm -f elf64 your_code.asm -o your_code.o，注意调整MASM语法到NASM兼容格式（比如替换invoke、修正伪指令差异）
   • 用GAS：需将MASM语法转换为AT&T语法，门槛较高，优先选NASM
   • 编译完成后，用GCC链接生成SO：gcc -shared -fPIC core.c your_code.o -o EmulatorCore.so

2. 尝试第三方转换工具
   用objconv替代objcopy，它对PE到ELF的转换支持更好：

   • 安装：sudo apt install objconv
   • 转换命令：objconv -felf64 test.obj test.o
   • 若转换成功，再尝试GCC链接

3. 修复包装器的调用约定与返回值问题
   你的包装器存在两个关键问题：

   • 未捕获返回值：asm_func的返回值存在rax寄存器，需将其赋值给toReturn
   • 调用约定适配错误：Windows x64用fastcall（第一个参数在rcx），Linux x64用System V ABI（第一个参数在rdi），需正确转换寄存器
     修正后的包装器代码：

   #include <stdint.h>
   
   extern int32_t asm_func(void* state);
   
   int32_t wrapper(void *state)
   {
     int32_t toReturn = 0;
   
     __asm__ volatile(
        "mov %%rdi, %%rcx\n"  // Linux的rdi参数转Windows约定的rcx
        "call asm_func\n"
        "mov %%eax, %0"       // 将rax的返回值存入toReturn
        : "=r"(toReturn)
        : "r"(state)
        : "rcx", "rax", "memory"  // 告知编译器这些寄存器会被修改
     );
   
     return toReturn;
   }
   
   int32_t fnEmulatorCode(void* state) {
     return wrapper(state);
   }
   

总结 #

优先选择重新汇编原始代码的方案，这是最稳定且无潜在兼容问题的方式。若无法获取原始汇编，再尝试objconv转换，同时修正包装器的逻辑。

内容的提问来源于Stack Exchange，提问作者BJury