为Drupal配置Varnish查询参数白名单，大幅提升性能

去年年底，来自中国和新加坡的爬虫激增导致服务器瘫痪，排查后发现大量请求存在冗余、错位、重复或损坏的查询参数，具体表现如下：

• 新闻页出现无关参数：/news?page=1&site=domain.org&type=page
• 重复参数请求：/list?page=1&page=15&page=120
• 损坏格式参数：/news?;page=1

这类问题直接导致Varnish无法正常缓存，缓存池中充斥大量重复内容，进而引发服务器负载异常。

解决方案：Varnish查询参数白名单配置 #

通过在Varnish的vcl_recv子程序中配置查询参数白名单，使用自定义临时头X-Query存储允许保留的参数，针对JS资源、图片样式及特定页面的查询参数做过滤处理，具体代码如下：

sub vcl_recv {
    ...
    # Remove unnecessary query strings to improve caching.
    # Check if URL has query string
    if (req.url ~ "\?") {
        # JS
        if (req.url ~ "\.js\?v=") {
            set req.http.X-Query = "";
            if (req.url ~ "\?v=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+\?(v=[^&]+).*?$", "&\1");
            }
            set req.url = regsub(req.url, "\?.+", regsub(req.http.X-Query, "^&", "?"));
            unset req.http.X-Query;
        }
        # Image styles
        elseif (req.url ~ "(\?|&)itok=" && req.url ~ "\.(webp|jpg|png)") {
            set req.http.X-Query = "";
            if (req.url ~ "(\?|&)itok=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+?(\?|&)(itok=[^&]+).*?$", "&\2");
            }
            set req.url = regsub(req.url, "\?.+", regsub(req.http.X-Query, "^&", "?"));
            unset req.http.X-Query;
        }
        # Some page with multiple query params
        elseif (req.url ~ "/some-page" && req.url ~ "(\?|&)(group|t1|t2|t3|t4n|a)") {
            set req.http.X-Query = "";
            if (req.url ~ "(\?|&)group=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+?(\?|&)(group=[^&]+).*?$", "&\2");
            }
            if (req.url ~ "(\?|&)t1=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+?(\?|&)(t1=[^&]+).*?$", "&\2");
            }
            if (req.url ~ "(\?|&)t2=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+?(\?|&)(t2=[^&]+).*?$", "&\2");
            }
            if (req.url ~ "(\?|&)t3=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+?(\?|&)(t3=[^&]+).*?$", "&\2");
            }
            if (req.url ~ "(\?|&)t4n=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+?(\?|&)(t4n=[^&]+).*?$", "&\2");
            }
            if (req.url ~ "(\?|&)a=") {
                set req.http.X-Query = req.http.X-Query + regsub(req.url, "^.+?(\?|&)(a=[^&]+).*?$", "&\2");
            }
            set req.url = regsub(req.url, "\?.+", regsub(req.http.X-Query, "^&", "?"));
            unset req.http.X-Query;
        }
    ...
}

实施效果 #

方案部署后服务器负载迅速恢复正常，同时带来额外收益：

• Varnish内存占用显著降低
• 缓存命中率大幅提升
• URL格式更规范统一
• 减少恶意参数请求，增强站点安全性

优化建议征集 #

目前的实现中存在较多重复代码，诚邀各位提供优化思路，比如是否可以通过抽取子例程的方式减少代码冗余？

内容的提问来源于stack exchange，提问作者mpj