VBA网页抓取：如何从HTML中提取总页数并检测Next分页按钮

针对你遇到的分页迭代卡住的问题，结合你提供的页面代码片段，我们可以从提取总页数和检测「Next」按钮存在性两个方向来解决，具体方案如下：

一、提取总页数 #

你的页面分页区域里已经明确标注了Total pages: 8这类文本，我们可以通过两种可靠方式提取这个数值：

方法1：直接解析分页区域的文本内容 #

分页区域在id="pagination"的div中，我们可以先获取这个元素的文本，再通过字符串处理提取总页数：

Dim paginationDiv As MSHTML.IHTMLElement
Set paginationDiv = page.getElementById("pagination")
If Not paginationDiv Is Nothing Then
    Dim pageText As String
    pageText = paginationDiv.innerText
    ' 定位"Total pages: "的起始位置，截取后面的数字
    Dim totalPagesStart As Integer
    totalPagesStart = InStr(pageText, "Total pages: ") + Len("Total pages: ")
    Dim totalPagesEnd As Integer
    totalPagesEnd = InStr(totalPagesStart, pageText, ".")
    Dim totalPages As Integer
    totalPages = CInt(Mid(pageText, totalPagesStart, totalPagesEnd - totalPagesStart))
    Debug.Print "总页数：" & totalPages
End If

方法2：通过分页链接统计（备选方案） #

如果文本提取出现异常，也可以通过统计分页链接来获取总页数——因为页面里最后一个数字链接就是总页数（倒数第二个链接，最后一个是「Next」）：

Dim pageLinks As MSHTML.IHTMLElementCollection
Set pageLinks = paginationDiv.getElementsByTagName("a")
If pageLinks.Count > 0 Then
    ' 取倒数第二个链接的文本作为总页数
    Dim lastPageNum As Integer
    lastPageNum = CInt(pageLinks(pageLinks.Count - 2).innerText)
    Debug.Print "总页数：" & lastPageNum
End If

二、检测页面中是否存在「Next」按钮 #

你之前用If page.body.innerHTML = "Next" Then的判断逻辑是错误的——innerHTML是整个页面的HTML内容，不可能等于单一的"Next"文本。正确的检测方式有两种：

方法1：直接查找「Next」链接元素 #

遍历分页区域内的所有<a>标签，匹配文本为「Next」的元素：

Dim nextLink As MSHTML.IHTMLElement
Set nextLink = Nothing
For Each linkEl In pageLinks
    If Trim(linkEl.innerText) = "Next" Then
        Set nextLink = linkEl
        Exit For
    End If
Next

If Not nextLink Is Nothing Then
    Debug.Print "找到Next按钮，链接地址：" & nextLink.href
    ' 这里可以直接用该URL请求下一页
    url = nextLink.href
    http.Open "GET", url, False
    http.send
    page.body.innerHTML = http.responseText
    ' 继续处理下一页数据...
End If

方法2：检查分页文本中是否包含「Next」 #

如果不需要获取Next的链接，只是判断是否存在，也可以直接检查分页区域的文本：

If InStr(paginationDiv.innerText, "Next") > 0 Then
    Debug.Print "存在Next按钮"
End If

三、修改后的完整VBA代码示例 #

结合上面的方法，调整你的代码实现完整的分页迭代：

Sub rawData(tool As String, row As Long)
 Dim http As New MSXML2.XMLHTTP60
 Dim page As New MSHTML.HTMLDocument
 Dim dataE As MSHTML.IHTMLElementCollection
 Dim dataEr As MSHTML.IHTMLElement
 Dim dataEc As MSHTML.IHTMLElement
 Dim dataO As MSHTML.IHTMLElementCollection
 Dim dataOr As MSHTML.IHTMLElement
 Dim dataOc As MSHTML.IHTMLElement
 Dim paginationDiv As MSHTML.IHTMLElement
 Dim pageLinks As MSHTML.IHTMLElementCollection
 Dim nextLink As MSHTML.IHTMLElement
 Dim totalPages As Integer
 Dim currentPage As Integer
 Dim url As String
 
 currentPage = 1
 url = "Actual_Link"
 
 Do
    ' 请求当前页
    http.Open "GET", url, False
    http.send
    page.body.innerHTML = http.responseText
    
    ' 处理当前页数据
    Set dataE = page.getElementsByClassName("even")
    Set dataO = page.getElementsByClassName("odd")
    Call dataEven(dataE, dataEr, dataEc, row)
    Call dataOdd(dataO, dataOr, dataOc, row)
    
    ' 获取分页区域，没有则结束循环
    Set paginationDiv = page.getElementById("pagination")
    If paginationDiv Is Nothing Then Exit Do
    
    ' 提取总页数
    Dim pageText As String
    pageText = paginationDiv.innerText
    Dim totalPagesStart As Integer
    totalPagesStart = InStr(pageText, "Total pages: ") + Len("Total pages: ")
    Dim totalPagesEnd As Integer
    totalPagesEnd = InStr(totalPagesStart, pageText, ".")
    totalPages = CInt(Mid(pageText, totalPagesStart, totalPagesEnd - totalPagesStart))
    
    ' 查找Next链接
    Set pageLinks = paginationDiv.getElementsByTagName("a")
    Set nextLink = Nothing
    For Each linkEl In pageLinks
        If Trim(linkEl.innerText) = "Next" Then
            Set nextLink = linkEl
            Exit For
        End If
    Next
    
    ' 判断是否还有下一页，没有则退出循环
    If nextLink Is Nothing Or currentPage >= totalPages Then
        Exit Do
    End If
    
    ' 更新下一页URL和当前页码
    url = nextLink.href
    currentPage = currentPage + 1
 Loop
 
End Sub

关键注意点： #

• 不要用page.getElementsByTagName("pagination")，pagination是元素ID，应该用getElementById("pagination")
• 文本匹配时记得用Trim()去除多余空格，避免页面里的 导致匹配失败
• 循环时同时判断Next按钮存在性和当前页是否小于总页数，避免无限循环

内容的提问来源于stack exchange，提问作者Killroy