压测遇dial tcp: socket: too many open files错误，求ulimit外解决方案

Great question! The dial tcp: socket: too many open files error under load typically happens because your app is exhausting the OS's file descriptor limit—each open TCP socket counts as a file descriptor. Instead of tweaking ulimit, here are practical, code-focused fixes tailored to your Go implementation:

1. Optimize HTTP Client Connection Reuse (Most Critical) #

Your current code explicitly disables connection reuse with req.Close=true and Connection: close, which forces a new TCP socket for every request. This is a fast track to hitting file descriptor limits. Instead, configure the http.Client's Transport to use a connection pool, so sockets are reused across requests:

Update your HttpClient initialization:

import (
    "net"
    "net/http"
    "time"
)

var HttpClient = &http.Client{
    Transport: &http.Transport{
        // Global max idle connections across all hosts
        MaxIdleConns: 100,
        // Max idle connections per target host (prevents overwhelming one server)
        MaxIdleConnsPerHost: 20,
        // Timeout for idle connections to free up resources
        IdleConnTimeout: 90 * time.Second,
        // Max concurrent connections per host (caps new socket creation)
        MaxConnsPerHost: 20,
        // Disable HTTP/2 if you don't need it (reduces connection overhead)
        ForceAttemptHTTP2: false,
        // Tuned dialer for faster connection setup and keepalives
        DialContext: (&net.Dialer{
            Timeout:   30 * time.Second,
            KeepAlive: 30 * time.Second,
        }).DialContext,
    },
}

Then remove these lines from your request setup—let the connection pool handle reuse automatically:

req.Header.Set("Connection", "close")
req.Close=true

2. Strictly Close Response Bodies (Prevent Connection Leaks) #

Your code only closes the response body if the request succeeds with a 200 status. If the request fails (e.g., non-200 status, network error), the body might stay open, leaking connections and file descriptors. Use defer to guarantee cleanup, even if errors occur:

Revise your request handling code:

resp, err := HttpClient.Do(req)
if err != nil {
    fmt.Println(err)
    return // Handle error appropriately
}
// Always clean up the response body, no matter what
defer func() {
    // Read the entire body before closing to avoid blocking connection reuse
    io.Copy(ioutil.Discard, resp.Body)
    resp.Body.Close()
}()

if resp.StatusCode == 200 {
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return
    }
    var Response Struct_Response
    err = json.Unmarshal(body, &Response)
    if err != nil {
        fmt.Println(err)
    }
}

Note: You need to read the full response body (even if you don't care about it) before closing—otherwise, the HTTP client won't return the connection to the pool.

3. Limit Concurrent Requests (Client-Side Throttling) #

Even with a connection pool, a sudden flood of concurrent requests can create more sockets than the pool can handle. Use a semaphore to cap the number of simultaneous requests:

For batch request scenarios, add this to your main function:

import (
    "sync"
    "bytes"
)

func main() {
    maxConcurrentRequests := 50 // Adjust based on your server's capacity
    semaphore := make(chan struct{}, maxConcurrentRequests)
    var wg sync.WaitGroup

    totalRequests := 1000 // Example: number of requests to send
    wg.Add(totalRequests)

    for i := 0; i < totalRequests; i++ {
        semaphore <- struct{}{} // Acquire a slot
        go func() {
            defer func() {
                <-semaphore // Release the slot
                wg.Done()
            }()

            // Your request logic here (fixed the bytes.NewBuffer bug too!)
            apiUrl := "http://example.com"
            jsonStr := "teststr"
            connToken := "1233333333333"
            
            req, err := http.NewRequest("POST", apiUrl, bytes.NewBufferString(jsonStr))
            if err != nil {
                fmt.Println(err)
                return
            }
            req.Header.Set("Content-Type", "application/json")
            req.Header.Set("requestid", "1234")
            req.Header.Set("Authorization", "Bearer "+connToken)

            resp, err := HttpClient.Do(req)
            if err != nil {
                fmt.Println(err)
                return
            }
            defer func() {
                io.Copy(ioutil.Discard, resp.Body)
                resp.Body.Close()
            }()

            if resp.StatusCode == 200 {
                body, err := ioutil.ReadAll(resp.Body)
                if err != nil {
                    fmt.Println(err)
                    return
                }
                var Response Struct_Response
                err = json.Unmarshal(body, &Response)
                if err != nil {
                    fmt.Println(err)
                }
            }
        }()
    }
    wg.Wait()
}

Side note: Fixed bytes.NewBuffer(jsonStr) to bytes.NewBufferString(jsonStr)—the original would throw a type error since jsonStr is a string, not a byte slice.

4. Audit for Other File Descriptor Leaks #

If you still hit the limit after the above fixes, check if your code opens other files (e.g., logs, local files) without closing them. Every open file consumes a descriptor, so ensure all os.File handles are closed with defer.

These changes should drastically reduce the number of open sockets your app uses, eliminating the need to adjust ulimit for most load scenarios.

内容的提问来源于stack exchange，提问作者Vani Polnedi