如何为Phoenix应用配置Prerender服务

Let's fix that Google crawling issue by setting up Nginx to route crawler requests through Prerender, while keeping your existing proxy to the Phoenix app. Here's exactly how to configure it:

Step 1: Update Your Nginx Server Block #

Modify your existing Nginx server config (the one that proxies to localhost:39499) to add logic for detecting crawler user agents and routing them to Prerender.

server {
    listen 80;
    server_name your-domain.com; # Replace with your actual domain

    # Define crawler user agents that should trigger Prerender
    set $prerender 0;
    if ($http_user_agent ~* "googlebot|bingbot|slurp|duckduckbot|baiduspider|yandexbot|sogoubot|exabot|facebot|ia_archiver") {
        set $prerender 1;
    }
    # Also handle requests with _escaped_fragment_ (for legacy AJAX crawling scheme)
    if ($query_string ~ "_escaped_fragment_") {
        set $prerender 1;
    }

    # Route crawler requests to Prerender
    if ($prerender = 1) {
        # Pass the full URL of your Phoenix app to Prerender
        set $prerender_url http://localhost:39499$request_uri;
        proxy_pass http://localhost:3000;
        proxy_set_header X-Prerender-Url $prerender_url;
        # Set necessary proxy headers for Prerender and Phoenix
        proxy_set_header Host $host;
        proxy_set_header User-Agent $http_user_agent;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        break;
    }

    # Regular user requests go to your Phoenix app as before
    location / {
        proxy_pass http://localhost:39499;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Step 2: Key Explanations #

• Crawler Detection: The first set of if blocks identify crawler user agents and the _escaped_fragment_ query parameter (used for older AJAX crawling standards) to trigger Prerender.
• Prerender Routing: When a crawler is detected, we set the X-Prerender-Url header to point directly to your Phoenix app's local port, then proxy the request to Prerender at localhost:3000. Prerender will fetch the fully rendered AngularJS content from your Phoenix app and return it to the crawler.
• Header Preservation: We pass through critical headers like Host, User-Agent, and IP-related headers to ensure both Prerender and your Phoenix app receive accurate request context.

Step 3: Test the Configuration #

1. Reload Nginx to apply the changes:

   sudo nginx -s reload
   

2. Verify crawler routing with curl:

   curl -A "googlebot" http://your-domain.com/your-angular-page
   

   You should see the fully rendered HTML (not just the AngularJS placeholder) in the response.

Optional: Add Caching #

To reduce load on Prerender and your Phoenix app, add Nginx caching for Prerendered responses. Insert this at the top of your server block:

proxy_cache_path /var/cache/nginx/prerender levels=1:2 keys_zone=prerender_cache:10m max_size=10g inactive=7d use_temp_path=off;

Then update the Prerender proxy block to use the cache:

if ($prerender = 1) {
    set $prerender_url http://localhost:39499$request_uri;
    proxy_pass http://localhost:3000;
    proxy_set_header X-Prerender-Url $prerender_url;
    # ... existing headers ...
    proxy_cache prerender_cache;
    proxy_cache_valid 200 7d;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    break;
}

That's it! This setup lets regular users hit your Phoenix app directly, while crawlers get the Prerendered version of your AngularJS content—all through a single Nginx server block with nested proxy logic.

内容的提问来源于stack exchange，提问作者user1694767