为何爬虫数组输出重复？多页面爬取返回重复第一页源码问题

你的问题出在循环请求URL的环节：

• 第一个for循环里，你写的是requests.get(URL_TEMPLATE[0])，这意味着不管循环到第几次，都只会请求数组里的第一个URL，所以两次结果都是第一页的源码。

只需要把URL_TEMPLATE[0]换成URL_TEMPLATE[i]，就能依次请求每个页面了，修正后的代码如下：

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

URL_TEMPLATE = [
    "https://colmagzhan.kz/content/nasledie-velikih-novye-vozmoghnosti-razvitiyasovremennogoobrazovaniya/lang-kz",
    "https://colmagzhan.kz/content/provedena-razyyasnitelynaya-beseda-so-studentami-1-kursov-po-profilakt/lang-kz"
]
r = [0]*len(URL_TEMPLATE)

for i in range(len(URL_TEMPLATE)):
    r[i] = requests.get(URL_TEMPLATE[i])  # 这里把0改成i

for i in range(len(r)):
    print(r[i].text)

另外，作为新手，也可以用更简洁的写法遍历URL列表，避免索引出错：

import requests

URL_TEMPLATE = [
    "https://colmagzhan.kz/content/nasledie-velikih-novye-vozmoghnosti-razvitiyasovremennogoobrazovaniya/lang-kz",
    "https://colmagzhan.kz/content/provedena-razyyasnitelynaya-beseda-so-studentami-1-kursov-po-profilakt/lang-kz"
]

responses = []
for url in URL_TEMPLATE:
    resp = requests.get(url)
    responses.append(resp)

for resp in responses:
    print(resp.text)

内容的提问来源于stack exchange，提问作者Alex