Cygwin环境下TCL建立20个TCP连接时接收数据异常问题咨询

问题背景 #

开发多系统互联项目，使用TCL搭建模拟测试环境，以客户端身份连接各TCP服务器，测试在Linux和Windows（通过Cygwin）环境运行。当系统数量增加后，Windows环境出现异常：某服务器持续触发可读事件，但读取无数据，且socket未关闭；Linux下无此问题。

环境版本 #

• TCL 8.6.12
• Cygwin 3.4.9-1.x86_64
• Windows 10 Enterprise 22H2

复现测试脚本 #

创建N个TCL线程各对应一个TCP服务器，主线程创建N个客户端连接并发送4字节数据，变量nbSockets对应N值：

set nbSockets 19
set portNbStart 20000

set ::auto_path [linsert $::auto_path end "Packages/Thread/Windows"]
package require Thread

# Variables used to wait for the receiver thread to be started 
tsv::set VARIABLES COND [thread::cond create]
tsv::set VARIABLES MUTEX [thread::mutex create]

# Create all receiver threads and their TCP socket server
for {set i 0} {$i < $nbSockets} {incr i} {

    # Lock mutex that will be used to check the readiness of the receiver thread
    thread::mutex lock [tsv::get VARIABLES MUTEX]
    
    set portNb [expr $portNbStart + $i]
    tsv::set VARIABLES PORT_NB $portNb
    tsv::set VARIABLES THREAD_READY 0

    set threadIds($i) [thread::create {
    
        # Initialize variables
        set ::CLIENT_SOCKET ""
        set ::SERVER_SOCKET ""
        set ::PORT_NB [tsv::get VARIABLES PORT_NB]
        
        # Actions to perform when a message is received from the client
        proc handleMessage {chan} {
            set data [chan read $chan 4]
            
            if {[eof $chan]} {
                puts "On port Nb $::PORT_NB, socket is closed"
            } else {
                puts "On port Nb $::PORT_NB, data size = [string length $data]"
            }
        }
        
        # Configuration of the client socket when a new connection is received
        proc accept {chan addr port} {
            set ::CLIENT_SOCKET $chan
            chan configure $chan -translation binary -buffering none -blocking 0
            chan event $chan readable [list handleMessage $chan]
        }
        
        # Actions to perform on disconnection
        proc Disconnect {} {
            # Close server socket
            CloseServerSocket

            # Close client socket
            CloseClientSocket
        }
        
        # Only close server socket
        proc CloseServerSocket {} {
            close $::SERVER_SOCKET
            set ::SERVER_SOCKET ""
        }
        
        # Only close client socket
        proc CloseClientSocket {} {
            fileevent $::CLIENT_SOCKET readable {}
            close $::CLIENT_SOCKET
            set ::CLIENT_SOCKET ""
        }

        # Accept incoming connections
        set ::SERVER_SOCKET [socket -server accept -myaddr 127.0.0.1 $::PORT_NB]
        
        # Inform the main thread that this one is ready to listen
        tsv::set VARIABLES THREAD_READY 1
        thread::mutex lock [tsv::get VARIABLES MUTEX]
        thread::cond notify [tsv::get VARIABLES COND]
        thread::mutex unlock [tsv::get VARIABLES MUTEX]
        
        # Enter the event loop
        thread::wait
    }]
    
    # Wait for the thread to be ready
    thread::cond wait [tsv::get VARIABLES COND] [tsv::get VARIABLES MUTEX] 10000; # Wait at most 10 seconds
    thread::mutex unlock [tsv::get VARIABLES MUTEX]
    
    if {[tsv::get VARIABLES THREAD_READY] != 1} {
        error "Could not launch TCP server thread on port number $portNb"
    }
}

# Connect to the receiver threads and send 4 bytes
for {set i 0} {$i < $nbSockets} {incr i} {
    set portNb [expr $portNbStart + $i]

    puts "Sending data at port number $portNb"

    # Connect to the server
    set Sockets($i) [socket -myaddr 127.0.0.1 127.0.0.1 $portNb]
    
    # Wait a little
    after 200
    
    # Send a simple message consisting of a 4 bytes integer representing the port number
    puts -nonewline $Sockets($i) [binary format I $portNb]
    flush $Sockets($i)
    
    # Wait a little again
    after 200
}

after 2000

# Cleaning...
puts "Closing..."
for {set i 0} {$i < $nbSockets} {incr i} {
    thread::send $threadIds($i) {
        Disconnect
    }
    close $Sockets($i)
    
    thread::release $threadIds($i)
}

异常现象 #

• 当nbSockets=19时，所有连接正常接收4字节数据
• 当nbSockets=20时，端口20019的服务器持续输出On port Nb 20019, data size = 0，重复超9000次

解答 #

1. 测试设计的潜在问题 #

• 非阻塞读逻辑缺陷：handleMessage中指定读取4字节，但非阻塞模式下，若当前无数据且socket未触发EOF，chan read $chan 4会返回空字符串，同时文件事件会被持续触发，导致无限循环输出空数据日志。
• 线程同步竞态风险：主线程用全局VARIABLES表共享PORT_NB和THREAD_READY，多个线程创建时可能出现变量覆盖——前一个线程未读取PORT_NB，主线程就已修改该变量供下一个线程使用，导致端口分配错误。
• 清理顺序不合理：主线程先发送Disconnect命令给子线程，再关闭客户端socket；但子线程的Disconnect先关闭服务器socket，再关闭客户端socket，可能出现客户端socket已被主线程关闭，子线程重复关闭的异常。

2. Cygwin下TCL的已知限制 #

• 文件句柄数量阈值：Cygwin默认文件描述符上限较低，当创建20个服务器socket+20个客户端socket，加上系统本身的句柄，可能接近阈值，导致部分socket的事件处理异常。
• 多线程事件循环兼容性：Cygwin的POSIX线程模拟与Windows原生线程存在差异，TCL的thread包在Cygwin环境下，多线程socket事件处理在线程数量较多时，可能出现异常触发的边缘情况。

3. 可调整的TCL参数与修复方案 #

• 修正非阻塞读逻辑：在handleMessage中处理空数据场景，避免无限触发事件，示例代码：

proc handleMessage {chan} {
    if {[eof $chan]} {
        puts "On port Nb $::PORT_NB, socket is closed"
        fileevent $chan readable {}
        return
    }
    set data [chan read $chan 4]
    set len [string length $data]
    if {$len > 0} {
        puts "On port Nb $::PORT_NB, data size = $len"
        # 未读满4字节时继续监听
        if {$len < 4} {
            chan event $chan readable [list handleMessage $chan]
        } else {
            chan event $chan readable {}
        }
    } else {
        # 无数据时暂时取消事件，避免频繁触发
        chan event $chan readable {}
        after 100 [list chan event $chan readable [list handleMessage $chan]]
    }
}

• 优化线程同步机制：为每个线程创建独立的共享变量空间，避免全局变量覆盖，示例：

# 创建线程时
set threadId [thread::create {
    set ::THREAD_ID [thread::id]
    set ::PORT_NB [tsv::get THREAD_$::THREAD_ID PORT_NB]
    # ... 其他逻辑
    tsv::set THREAD_$::THREAD_ID THREAD_READY 1
    thread::mutex lock [tsv::get THREAD_$::THREAD_ID MUTEX]
    thread::cond notify [tsv::get THREAD_$::THREAD_ID COND]
    thread::mutex unlock [tsv::get THREAD_$::THREAD_ID MUTEX]
    thread::wait
}]
# 为当前线程初始化独立变量
tsv::set THREAD_$threadId PORT_NB $portNb
tsv::set THREAD_$threadId COND [thread::cond create]
tsv::set THREAD_$threadId MUTEX [thread::mutex create]
tsv::set THREAD_$threadId THREAD_READY 0

• 调整Cygwin文件描述符限制：启动Cygwin终端时执行ulimit -n 1024，提高文件描述符上限。
• 优化socket缓冲设置：显式设置socket缓冲大小，比如chan configure $chan -buffersize 4096，改善数据读取稳定性。

内容的提问来源于stack exchange，提问作者Mathieu