Instagram爬虫XPath定位失败求助：无法获取点赞、日期等数据

我正在编写Instagram爬虫，尝试用XPath提取点赞数、发布日期、描述等数据，但程序始终返回空值。试过修改XPath，比如最初用：

likes_xpath = '//*[@id="react-root"]/section/main/div/div[1]/article/div[3]/section[2]/div/div/button/span'

后来改成：

likes_xpath = '//div[@class="likes"]/section/div/div/span/a/span'

但事后发现写法有误，现在不清楚怎么写简洁可靠的XPath，也不确定数据提取方式是否正确，甚至怀疑周边代码有问题。以下是相关代码：

def __lcondition(self, link):
    return '.com/p/' in link.get_attribute('href')

def __get_user(self):
    user = self.__driver.find_element(By.XAPTH, '//*[@id="react-root"]\
      /section/main/div/div[1]/article/header/div[2]/div[1]/div[1]/span/a')
    user = user.get_attribute('href').split('/')[-2]
    return user


def filter_links(self, links):
    post_links = []
    for link in links:
        try:
            if '.com/p/' in link.get_attribute('href'):
                post_links.append(link)
        except:
            logger.warning("A https://www.instagram.com/p/ link was not found")
            continue
    return post_links


def __get_subtitles(self,):
    subtitles_xpath = '/html/body/div[1]/section/main/div/div[1]/article/div[3]/div[1]/ul/div/li/div/div/div[2]/span'
    subtitles = self.__driver.find_element(By.XPATH, subtitles_xpath)
    subtitles = subtitles.get_attribute('innerHTML')
    subtitles = BeautifulSoup(subtitles).get_text()
    return subtitles

def __get_image_description(self) -> str:
    images = self.__driver.find_elements(By.TAG_NAME, 'img')
    post_infos = images[1].get_attribute('alt')

    if len(post_infos) > 0:
        post_infos = post_infos.split("Image may contain: ")
        if len(post_infos) >=2:
            #image
            return post_infos[1]
        else:
            #video
            logger.warning("Description is not available")
            return ''
    else:
        return ''
    
def __get_likes(self):
    likes_xpath = '//div[@class="likes"]/section/div/div/span/a/span'

    likes = self.__driver.find_element(By.XPATH, likes_xpath)
    likes = likes.get_attribute('innerHTML')
    return likes

def __get_views(self):
    '''
    Getting views
    '''
    views_xpath = '/html/body/div[1]/section/main/div/div[1]/article/div[3]/section[2]/div/span/span'
    views = self.__driver.find_element(By.XPATH, views_xpath)
    views = views.get_attribute('innerHTML')
    return views

def __get_date(self):
    date_xpath = '//time[@class="_aaqe"]/div/a/span/time'
    date = self.__driver.find_element(By.XPATH, date_xpath)
    date = date.get_attribute('datetime')
    return date

def __get_image_data(self, link):
    infos = {}
    infos['date'] = self.__get_date()
    infos['type'] = 'image'
    if len(infos['date']) == 0:
        return None
    infos['user'] = self.__get_user()
    infos['subtitles'] = self.__get_subtitles()
    infos['image_description'] = self.__get_image_description()
    infos['likes'] = self.__get_likes()
    #infos['views'] = None
    infos['link'] = link

    return infos


def __get_video_data(self, link):
    infos = {}
    infos['date'] = self.__get_date()
    infos['type'] = 'video'
    if len(infos['date']) == 0:
        return None
    infos['user'] = self.__get_user()
    infos['subtitles'] = self.__get_subtitles()
    #infos['views'] = self.__get_views()
    infos['likes'] = self.__get_likes()
    infos['link'] = link

    return infos

核心错误点排查 #

1. XPath定位逻辑错误 #

• 绝对路径完全不可靠：你用的/html/body/div[1]/...这类绝对路径依赖页面固定层级，而Instagram是动态渲染的单页应用，页面结构随时可能变动，直接用绝对路径大概率找不到元素。
• 臆造class名称：//div[@class="likes"]里的likes不是Instagram真实的class名，Instagram的class都是动态生成的短字符串（比如_aamz），不能用这类自定义的class定位。
• 定位方向错误：应该基于元素的语义属性定位，比如点赞按钮的aria-label会包含"赞"或"likes"关键词，日期元素自带datetime属性，用这些特征写XPath才稳定。

2. 代码拼写低级错误 #

__get_user方法里的By.XAPTH是拼写错误，正确写法是By.XPATH，这个错误会直接导致元素查找失败，返回空值。

3. 缺少动态元素等待 #

Instagram页面加载是异步的，直接调用find_element会在元素未渲染完成时执行，自然返回空。必须添加显式等待，等待元素可见或可交互后再查找。

4. 数据提取逻辑不稳定 #

• __get_image_description依赖images[1]的索引定位图片，页面图片数量变化时会取错元素，应该直接定位帖子的主图元素。
• __get_subtitles用innerHTML转BeautifulSoup完全没必要，直接用元素的text属性即可获取内容。

5. 未区分帖子类型的交互逻辑 #

图片帖子直接显示点赞数，视频帖子默认显示播放量，需要点击"查看赞"按钮才会显示点赞数。你的代码没有区分这两种情况，直接提取点赞必然失败。

修正后的关键方法示例 #

修正后的点赞数提取（区分图片/视频） #

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def __get_likes(self):
    try:
        # 图片帖子直接找点赞数
        likes_elem = WebDriverWait(self.__driver, 10).until(
            EC.presence_of_element_located((By.XPATH, '//section[contains(@class, "_ae4h")]//span[contains(text(), "赞")]/preceding-sibling::span'))
        )
        return likes_elem.text.strip()
    except:
        # 视频帖子先点击查看赞按钮
        try:
            view_btn = WebDriverWait(self.__driver, 10).until(
                EC.element_to_be_clickable((By.XPATH, '//button[contains(text(), "查看赞")]'))
            )
            view_btn.click()
            likes_elem = WebDriverWait(self.__driver, 10).until(
                EC.presence_of_element_located((By.XPATH, '//span[contains(text(), "赞")]/preceding-sibling::span'))
            )
            return likes_elem.text.strip()
        except:
            return "0"

修正后的日期提取 #

def __get_date(self):
    date_elem = WebDriverWait(self.__driver, 10).until(
        EC.presence_of_element_located((By.XPATH, '//time[@datetime]'))
    )
    return date_elem.get_attribute('datetime')

内容的提问来源于stack exchange，提问作者ARJUN RAMASWAMY