如何用Python存储谷歌搜索结果为JSON/字典，提取公司地址用于GIS

问题 #

我通过谷歌搜索查询公司信息，目标是获取公司地址或经纬度用于GIS绘图。为练习Python技能，不想使用在线付费服务。之前用GeoPy Nominatim处理数据集时出现报错：AttributeError: 'NoneType' object has no attribute 'latitude'。

目前已用以下代码获取到googlesearch.SearchResult类型的搜索结果：

import requests
import json
import pandas as pd
import time
from bs4 import BeautifulSoup
try:
    from googlesearch import search
except ImportError:
    print("No module named 'google' found")

# 公司名称
query = "ADG Creative address"

# 打印5条结果，避免地址不在第一条
for i in search(query, advanced=True, sleep_interval=5, num_results=5):
    print(i)

搜索结果示例：

SearchResult(url=https://www.adgcreative.net/, title=ADG Creative | Branding, Design, Content, Video, UX/UI, description=ADG is a creative studio specializing in branding, design, content, video, UX/UI, and complex communications to change behaviors and impact audiences.)
SearchResult(url=https://www.zoominfo.com/c/adg-creative-llc/461999, title=ADG Creative - Overview, News & Competitors, description=ADG Creative's headquarters are located at 7151 Columbia Gateway Dr Ste B, Columbia, Maryland, 21046, United States What is ADG Creative's phone number? ADG ...)
SearchResult(url=https://www.dnb.com/business-directory/company-profiles.adg_creative_llc.4f06c15e0e3edf7fa31353b2ba0736cd.html, title=Adg Creative, LLC Company Profile | Columbia, MD, description=Where is Adg Creative, LLC located? Adg Creative, LLC is located at 7151 Columbia Gateway Dr Ste B Columbia, MD, 21046-2108 United States · What is Adg Creative, ...)
SearchResult(url=https://pitchbook.com/profiles/company/128768-86, title=ADG Creative Company Profile: Acquisition & Investors, description=ADG Creative General Information · 7151 Columbia Gateway Drive · Suite B · Columbia, MD 21046 · United States.)
SearchResult(url=https://www.winmo.com/open/agency/md/columbia/adg-creative/59344, title=ADG Creative - Columbia MD | Agency Profile, Contacts, AOR ..., description=ADG Creative | Agency Profile, Contacts, AOR, Client Relationships. Service: full-service. Main Telephone: (443) 285-0008. Primary Address.)
SearchResult(url=https://www.linkedin.com/company/adg-creative, title=ADG Creative, description=We're a full-stack creative studio, uncomplicating brand and business stories for commercial and public-sector clients for nearly 30 years. Our clients have ...)

核心需求：

• 遍历公司列表，批量获取每家公司的地址
• 存储搜索结果并解析提取地址
• 将地址保存至DataFrame或本地文件
• 更高效的Python实现方案

解决方案 #

1. 存储SearchResult数据 #

先将搜索结果存入列表，方便后续统一处理：

# 初始化列表存储搜索结果
search_results = []
query = "ADG Creative address"

for result in search(query, advanced=True, sleep_interval=5, num_results=5):
    search_results.append(result)

每个SearchResult对象可直接通过.访问属性：result.url、result.title、result.description。

2. 解析提取地址 #

地址通常出现在description字段中，用正则表达式匹配结构化地址格式（以下针对美国地址，可根据需求调整）：

import re

def extract_address(description):
    # 匹配完整地址格式：街道+城市+州+邮编
    pattern = r'\d+ [\w\s]+(Dr|St|Ave|Blvd|Way) [\w\s]*,? [\w\s]+, [A-Z]{2} \d{5}(-\d{4})?'
    match = re.search(pattern, description)
    if match:
        return match.group().strip()
    # 匹配简化格式
    pattern_simple = r'\d+ [\w\s]+, [\w\s]+, [A-Z]{2} \d{5}'
    match_simple = re.search(pattern_simple, description)
    if match_simple:
        return match_simple.group().strip()
    return None

# 提取有效地址（找到第一个有效地址就停止，可根据需求改为收集所有）
valid_addresses = []
for result in search_results:
    addr = extract_address(result.description)
    if addr:
        valid_addresses.append(addr)
        break

3. 保存到DataFrame或文件 #

保存至DataFrame

import pandas as pd

# 构建DataFrame
df = pd.DataFrame({
    'company_name': ['ADG Creative'] * len(valid_addresses),
    'address': valid_addresses
})
# 去重
df = df.drop_duplicates(subset='address')
print(df)

保存至本地文件

# 保存为CSV
df.to_csv('company_addresses.csv', index=False)

# 保存为JSON
df.to_json('company_addresses.json', orient='records')

4. 高效实现建议 #

• 批量处理封装：将查询逻辑封装为函数，遍历公司列表批量处理：

  def get_company_address(company_name):
      query = f"{company_name} address"
      for result in search(query, advanced=True, sleep_interval=3, num_results=3):
          addr = extract_address(result.description)
          if addr:
              return addr
      return None
  
  # 示例公司列表
  companies = ["ADG Creative", "Example Corp"]
  address_list = []
  for comp in companies:
      address_list.append({
          'company': comp,
          'address': get_company_address(comp)
      })
  
  df = pd.DataFrame(address_list)
  

• 修复GeoPy报错：针对之前的NoneType报错，增加判断避免崩溃：

  from geopy.geocoders import Nominatim
  
  geolocator = Nominatim(user_agent="my_gis_app")
  location = geolocator.geocode(valid_addresses[0])
  if location:
      latitude = location.latitude
      longitude = location.longitude
  else:
      latitude = None
      longitude = None
  

• 多线程加速：公司数量较多时，用线程池并行处理（注意控制速率，避免触发谷歌反爬）：

  from concurrent.futures import ThreadPoolExecutor
  
  with ThreadPoolExecutor(max_workers=5) as executor:
      results = executor.map(get_company_address, companies)
  
  address_list = [{'company': comp, 'address': addr} for comp, addr in zip(companies, results)]
  

• 反爬优化：增加随机sleep时长、使用自定义User-Agent，降低被封禁风险。

内容的提问来源于stack exchange，提问作者codingcat