如何用Python将BeautifulSoup提取的链接保存到文本文件中

错误原因 #

• 你代码里的links变量已经是单条完整的URL字符串，额外嵌套for link in links:会把字符串拆成单个字符遍历，最终写入文件的是拆分后的单个字符，不是完整链接
• 频繁在循环内调用with open(...)执行追加写操作，会产生大量不必要的IO开销，运行效率低

修正代码 #

推荐先把所有爬取到的链接存入列表，全部爬取完成后一次性写入文件，效率更高：

import bs4 as bs4
from bs4 import BeautifulSoup
import requests

suffixeUrl = '_puis_nblignes_est_200.html'
all_links = []

for i in range(15):
    url = 'https://www.topachat.com/pages/produits_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_page_est_' + str(i) + suffixeUrl
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.text, 'html.parser')
    if response.ok:
        print('Page: ' + str(i))

    for data in soup.find_all('div', class_='price'):
        for a in data.find_all('a'):
            link = a.get('href')
            full_link = 'https://www.topachat.com/' + link
            print(full_link)
            all_links.append(full_link)

# 统一写入文件
with open("urls.txt", "w", encoding="utf-8") as f:
    f.write('\n'.join(all_links))

如果担心爬取中途程序崩溃丢失已爬取数据，可以采用边爬边写的方式，直接删除多余的for link in links循环即可：

import bs4 as bs4
from bs4 import BeautifulSoup
import requests

suffixeUrl = '_puis_nblignes_est_200.html'

for i in range(15):
    url = 'https://www.topachat.com/pages/produits_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_page_est_' + str(i) + suffixeUrl
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.text, 'html.parser')
    if response.ok:
        print('Page: ' + str(i))

    for data in soup.find_all('div', class_='price'):
        for a in data.find_all('a'):
            link = a.get('href')
            full_link = 'https://www.topachat.com/' + link
            print(full_link)
            # 直接追加写入，无需额外循环
            with open("urls.txt", "a", encoding="utf-8") as f:
                f.write(full_link + '\n')

内容的提问来源于stack exchange，提问作者Val-55