本地正常的Flask+Pytube程序部署Heroku时触发内部服务器错误

我开发了一个基于Flask和Pytube的程序，用于获取YouTube视频流信息，代码如下：

from pytube import YouTube
from flask import request
from flask import Flask
import jsonpickle

app = Flask(__name__)
@app.route('/login', methods=['GET', 'POST'])
def login():
    if request.method == 'POST':
           url = request.form["d_url"]
           yt_video = YouTube(url)
           videos = yt_video.streams
           res_list = list(videos)
           return jsonpickle.encode(res_list)
    else:
        return request.args.get('d_url')
if __name__ == "__main__":
    app.run(debug =True)

该程序在本地运行正常，但部署到Heroku后出现Internal Server Error，日志报错信息如下：

Traceback (most recent call last):
2022-08-07T07:48:45.427991+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app
2022-08-07T07:48:45.427991+00:00 app[web.1]:     response = self.full_dispatch_request()
2022-08-07T07:48:45.427992+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request
2022-08-07T07:48:45.427992+00:00 app[web.1]:     rv = self.handle_user_exception(e)
2022-08-07T07:48:45.427993+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
2022-08-07T07:48:45.427993+00:00 app[web.1]:     rv = self.dispatch_request()
2022-08-07T07:48:45.427993+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
2022-08-07T07:48:45.427994+00:00 app[web.1]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2022-08-07T07:48:45.427994+00:00 app[web.1]:   File "/app/app.py", line 11, in login
2022-08-07T07:48:45.427994+00:00 app[web.1]:     yt_video = YouTube(str(request.form["d_url"]))
2022-08-07T07:48:45.427995+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/pytube/__main__.py", line 91, in __init__
2022-08-07T07:48:45.427995+00:00 app[web.1]:     self.prefetch()
2022-08-07T07:48:45.427996+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/pytube/__main__.py", line 181, in prefetch
2022-08-07T07:48:45.427996+00:00 app[web.1]:     self.vid_info_raw = request.get(self.vid_info_url)
2022-08-07T07:48:45.427996+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/pytube/request.py", line 36, in get
2022-08-07T07:48:45.427997+00:00 app[web.1]:     return _execute_request(url).read().decode("utf-8")
2022-08-07T07:48:45.427997+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/pytube/request.py", line 24, in _execute_request
2022-08-07T07:48:45.427998+00:00 app[web.1]:     return urlopen(request)  # nosec
2022-08-07T07:48:45.427998+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 216, in urlopen
2022-08-07T07:48:45.427998+00:00 app[web.1]:     return opener.open(url, data, timeout)
2022-08-07T07:48:45.427998+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 525, in open
2022-08-07T07:48:45.427998+00:00 app[web.1]:     response = meth(req, response)
2022-08-07T07:48:45.427999+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 634, in http_response
2022-08-07T07:48:45.427999+00:00 app[web.1]:     response = self.parent.error(
2022-08-07T07:48:45.427999+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 557, in error
2022-08-07T07:48:45.428000+00:00 app[web.1]:     result = self._call_chain(*args)
2022-08-07T07:48:45.428000+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 496, in _call_chain
2022-08-07T07:48:45.428000+00:00 app[web.1]:     result = func(*args)
2022-08-07T07:48:45.428000+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 749, in http_error_302
2022-08-07T07:48:45.428001+00:00 app[web.1]:     return self.parent.open(new, timeout=req.timeout)
2022-08-07T07:48:45.428001+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 525, in open
2022-08-07T07:48:45.428001+00:00 app[web.1]:     response = meth(req, response)
2022-08-07T07:48:45.428002+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 634, in http_response
2022-08-07T07:48:45.428002+00:00 app[web.1]:     response = self.parent.error(
2022-08-07T07:48:45.428002+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 563, in error
2022-08-07T07:48:45.428002+00:00 app[web.1]:     return self._call_chain(*args)
2022-08-07T07:48:45.428003+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 496, in _call_chain
2022-08-07T07:48:45.428003+00:00 app[web.1]:     result = func(*args)
2022-08-07T07:48:45.428003+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.10/urllib/request.py", line 643, in http_error_default
2022-08-07T07:48:45.428003+00:00 app[web.1]:     raise HTTPError(req.full_url, code, msg, hdrs, fp)

原因分析 #

从报错栈可以定位到，问题出在Pytube请求YouTube视频信息时触发了HTTP错误，核心原因包括：

• Heroku共享IP被YouTube拦截：YouTube的反爬机制会限制频繁请求的IP，Heroku的共享IP池大概率已被标记，导致请求被拒绝。
• Pytube版本兼容性问题：旧版本Pytube可能无法适配YouTube的最新API逻辑，本地环境因缓存或网络差异未暴露问题，但在Heroku服务器上触发错误。
• 请求头未模拟真实浏览器：Pytube默认请求头缺少浏览器标识，被YouTube反爬机制识别为非人类请求。

解决方案 #

方案1：更换稳定版Pytube #

官方Pytube更新不及时时，可使用社区维护的稳定版本，修改requirements.txt：

Flask==2.2.2
jsonpickle==2.2.0
pytube==12.1.0

方案2：添加自定义请求头模拟浏览器 #

修改代码，为Pytube请求添加浏览器UA，绕过反爬检测：

from pytube import YouTube
from flask import request
from flask import Flask
import jsonpickle

app = Flask(__name__)

@app.route('/login', methods=['GET', 'POST'])
def login():
    if request.method == 'POST':
        url = request.form["d_url"]
        # 配置浏览器请求头
        yt_video = YouTube(
            url,
            use_oauth=False,
            allow_oauth_cache=False,
            headers={
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36'
            }
        )
        videos = yt_video.streams
        res_list = list(videos)
        return jsonpickle.encode(res_list)
    else:
        return request.args.get('d_url')

if __name__ == "__main__":
    app.run(debug=True)

方案3：禁用Pytube预取功能 #

预取机制会触发额外请求，可能导致被拦截，可尝试关闭：

yt_video = YouTube(url)
yt_video.prefetch = lambda: None  # 禁用预取逻辑
videos = yt_video.streams

方案4：配置代理请求 #

如果Heroku IP被彻底屏蔽，可添加代理服务，修改代码并在Heroku设置代理环境变量：

import os
from pytube import YouTube
from flask import request
from flask import Flask
import jsonpickle
import requests

# 替换Pytube默认请求方法为代理请求
proxy_url = os.getenv('PROXY_URL')
if proxy_url:
    def proxied_get(url):
        proxies = {'http': proxy_url, 'https': proxy_url}
        return requests.get(url, proxies=proxies).text
    from pytube import request
    request.get = proxied_get

app = Flask(__name__)

@app.route('/login', methods=['GET', 'POST'])
def login():
    if request.method == 'POST':
        url = request.form["d_url"]
        yt_video = YouTube(url)
        videos = yt_video.streams
        res_list = list(videos)
        return jsonpickle.encode(res_list)
    else:
        return request.args.get('d_url')

if __name__ == "__main__":
    app.run(debug=True)

在Heroku的环境变量中设置PROXY_URL为可用的代理地址即可。

内容的提问来源于stack exchange，提问作者Imran Iqbal