Docker环境下Nginx作为Gmail SMTP代理的配置方法及502 5.5.1错误排查与疑问

Hey, I get it—setting up Nginx as a proxy for Gmail's SMTP server and hitting that 502 5.5.1 Unrecognized command error after AUTH LOGIN is super frustrating. Let's break down what's going wrong and how to fix it, plus answer your question about Auth-Server values.

Why This Error Happens #

The core issue here is that Nginx isn't properly passing your authentication request to Gmail's SMTP server, or some part of your configuration doesn't align with Gmail's strict requirements. Let's go through the most likely culprits:

1. You're Using an IP Instead of FQDN for Auth-Server #

Gmail's SMTP server checks that the server you're connecting to matches the expected domain. When you hardcode the IP (108.177.119.108) in Auth-Server, Nginx sends requests that don't pass Gmail's domain validation, which can cause it to reject the AUTH command as unrecognized.

Fix:
Swap the IP for Gmail's official SMTP FQDN in your smtpauthssl location:

add_header Auth-Server "smtp.gmail.com";

2. Your SMTP Auth Mechanisms Include Unsupported Methods #

Your Nginx mail server block has smtp_auth login plain cram-md5, but Gmail doesn't support CRAM-MD5 for SMTP authentication over STARTTLS. Having this extra mechanism can confuse the handshake between Nginx and Gmail, leading to the 502 error.

Fix:
Trim the auth mechanisms to only what Gmail supports:

smtp_auth login plain;

3. Minor Tweaks to Your Auth HTTP Response #

Your regex for extracting the domain from $http_auth_user works, but can be simplified to avoid unnecessary capture groups (this is a small tweak, but keeps things clean):

if ($http_auth_user ~* "@(\S+)$" ) {
    set $domain $1;
}

Also, double-check that your auth_http endpoint returns only the required headers (Auth-Status, Auth-Server, Auth-Port) without any extra content—Nginx expects a clean response here.

4. Gmail's Security Settings #

Even with "Less secure app access" enabled, Google now prefers App Passwords if you have 2FA turned on for your account. If you haven't already, try generating an app-specific password and using that instead of your regular account password during authentication—it might resolve hidden security blocks.

Your Auth-Server Question: FQDN vs IP #

Always use a FQDN (fully qualified domain name) for Auth-Server, not an IP address. Here's why:

• Most SMTP servers (including Gmail) validate that the HELO/EHLO domain matches the actual server domain. Using an IP will fail this check.
• FQDNs let Nginx automatically resolve to the latest available IPs for the server, which is more reliable than hardcoding a single IP that might change.

Testing the Fix #

After updating your config, reload Nginx to apply changes:

nginx -s reload

Then run your openssl test again:

openssl s_client -connect mail.localhost.com:12465 -starttls smtp

Once you send the EHLO command, try AUTH LOGIN with your base64-encoded email and app password (or regular password if 2FA is off)—you should be able to authenticate successfully this time.

内容的提问来源于stack exchange，提问作者EnterSB