Golang微服务Apache AB压测超时问题排查求助

Alright, let's dig into why your Golang microservice is timing out under Apache AB load testing when your Java version works fine. I've seen similar issues with Golang HTTP services and RabbitMQ integrations, so here are the most likely culprits and how to fix them:

1. Missing HTTP Server Timeouts in Golang #

Unlike Java's Tomcat (which has sensible default timeout settings), the default Golang http.Server has no built-in read/write timeouts. This means that if a request gets stuck waiting on RabbitMQ or hangs for any reason, the connection will stay open indefinitely—eating up resources and exhausting AB's keep-alive connections (from the -k flag), which leads to timeouts.

Fix: Add explicit timeouts to your HTTP server #

import (
    "net/http"
    "time"
    "log"
)

func main() {
    router := http.HandlerFunc(yourRequestHandler)
    
    srv := &http.Server{
        Addr:         ":443", // Your TLS-enabled port
        Handler:      router,
        ReadTimeout:  5 * time.Second,  // Max time to read request headers/body
        WriteTimeout: 10 * time.Second, // Max time to send full response
        IdleTimeout:  120 * time.Second, // Keep-alive connection idle timeout
        TLSConfig:    yourTLSConfig,     // Your existing TLS setup
    }
    
    log.Fatal(srv.ListenAndServeTLS("cert.pem", "key.pem"))
}

2. Poor RabbitMQ Client Handling #

Java's Spring AMQP handles RabbitMQ connections/channels with robust pooling and automatic reconnection out of the box. Golang's popular streadway/amqp library, by contrast, is low-level and requires manual management. Common mistakes here include:

• Creating a new RabbitMQ connection/channel per request (this overwhelms RabbitMQ with connections, leading to backpressure)
• No error handling for broken channels/connections (if RabbitMQ drops a channel, your Golang code may block indefinitely waiting to publish)
• No publish timeouts (if RabbitMQ is slow, your request will hang until AB's -s timeout triggers)

Fixes for RabbitMQ Integration #

• Reuse a single RabbitMQ connection (connections are heavy; channels are lightweight, so reuse channels or use a channel pool)
• Add publish timeouts using context to prevent hanging requests:

  import (
      "context"
      "time"
      "github.com/streadway/amqp"
  )
  
  func publishMessage(ch *amqp.Channel, exchange, routingKey, msg string) error {
      // Set a 2-second timeout for publishing to RabbitMQ
      ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
      defer cancel()
  
      return ch.PublishWithContext(ctx,
          exchange,
          routingKey,
          false, // mandatory
          false, // immediate
          amqp.Publishing{
              ContentType: "application/json",
              Body:        []byte(msg),
          })
  }
  

• Implement reconnection logic for RabbitMQ connections/channels (listen for ch.NotifyClose() events and recreate connections when they drop)

3. TLS Compatibility Issues #

Your test uses TLS, and Golang's default TLS settings can differ from Java's, leading to slow handshakes or compatibility issues with AB:

• Golang enables HTTP/2 by default, but AB is primarily an HTTP/1.1 tool—HTTP/2 misconfiguration can cause connection hangs
• Mismatched cipher suites or missing certificate validation (though this would likely cause hard failures, not timeouts)

Fix: Force HTTP/1.1 in your TLS config #

import "crypto/tls"

func createTLSConfig() *tls.Config {
    cfg := &tls.Config{
        // Your existing TLS settings (certificates, root CAs, etc.)
        NextProtos: []string{"http/1.1"}, // Disable HTTP/2 to match AB's behavior
    }
    return cfg
}

4. Concurrency & Goroutine Leaks #

Golang's HTTP server spawns a goroutine per request, but if your request handler leaks goroutines (e.g., not cleaning up goroutines waiting on RabbitMQ), you'll quickly exhaust memory and CPU, leading to slow response times.

Debugging Step: Use pprof to check for leaks #

Run your Golang service with pprof enabled, then during the AB test, inspect goroutine counts:

// Add this import to your main() to enable pprof
import _ "net/http/pprof"

// During load test, run this command to check stuck goroutines
go tool pprof http://your-service:6060/debug/pprof/goroutine?debug=2

Look for goroutines stuck in amqp.Publish or waiting on unbuffered channels—these are likely culprits.

5. Validate AB Test Parameters #

You mentioned using -s 2 -s 60—this is a typo (the last -s overrides the first). Make sure your AB timeout (-s 60) is longer than your Golang server's write timeout, otherwise AB will mark requests as timed out before your service can respond.

Debugging Checklist to Narrow It Down #

1. Isolate the problem: Run AB against a Golang endpoint that doesn't touch RabbitMQ—if it works, the issue is definitely in your RabbitMQ integration
2. Check RabbitMQ metrics: Monitor connection/channel counts, message rates, and queue depth in the RabbitMQ management UI—if connections are spiking or messages are piling up, that's a red flag
3. Log request timings: Add logs to your Golang handler to track how long each request takes to process, especially the time spent publishing to RabbitMQ
4. Inspect connection states: Use ss -tulpn | grep your-service-port to see if there are hundreds of stuck ESTABLISHED or TIME_WAIT connections

内容的提问来源于stack exchange，提问作者RedCollarPanda