使用Jsoup递归爬取HTML时如何避免重复与无效路径

常见问题场景还原 #

待爬取的书签HTML示例 #

<!DOCTYPE NETSCAPE-Bookmark-file-1>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
<TITLE>Bookmarks</TITLE>
<H1>Bookmarks</H1>
<DL><p>
    <DT><H3 ADD_DATE="1620000000" LAST_MODIFIED="1630000000">技术网站</H3>
    <DL><p>
        <DT><A HREF="https://www.example.com" ADD_DATE="1620000000" LAST_MODIFIED="1630000000">示例站点</A>
        <DT><H3 ADD_DATE="1620000000" LAST_MODIFIED="1630000000">前端资源</H3>
        <DL><p>
            <DT><A HREF="https://www.frontend.com" ADD_DATE="1620000000" LAST_MODIFIED="1630000000">前端教程</A>
        </DL><p>
    </DL><p>
    <DT><A HREF="https://www.google.com" ADD_DATE="1620000000" LAST_MODIFIED="1630000000">谷歌搜索</A>
</DL><p>

典型错误日志 #

重复条目：前端教程
不存在路径：技术网站/未知文件夹/前端教程

问题代码示例（有缺陷的递归实现） #

import org.jsoup.Jsoup
import org.jsoup.nodes.Element

data class Bookmark(val name: String, val url: String?, val path: String, val children: List<Bookmark> = emptyList())

fun parseBookmarks(html: String): List<Bookmark> {
    val doc = Jsoup.parse(html)
    val root = doc.selectFirst("dl")
    return parseFolder(root, "")
}

private fun parseFolder(folderElement: Element, currentPath: String): List<Bookmark> {
    val bookmarks = mutableListOf<Bookmark>()
    val children = folderElement.children()
    var currentFolderName = ""
    var currentSubPath = currentPath

    for (element in children) {
        when (element.tagName()) {
            "h3" -> {
                currentFolderName = element.text()
                currentSubPath = if (currentPath.isEmpty()) currentFolderName else "$currentPath/$currentFolderName"
            }
            "a" -> {
                val bookmark = Bookmark(
                    name = element.text(),
                    url = element.attr("href"),
                    path = currentSubPath
                )
                bookmarks.add(bookmark)
                // 错误：递归调用当前文件夹导致重复遍历
                bookmarks.addAll(parseFolder(folderElement, currentSubPath))
            }
            "dl" -> {
                bookmarks.addAll(parseFolder(element, currentSubPath))
            }
        }
    }
    return bookmarks
}

问题根源分析 #

• 条目重复：处理<a>标签时错误递归当前文件夹，触发重复遍历；同时递归路径传递未区分上下文，导致子内容被多次挂载。
• 路径错误：<h3>处理时直接覆盖路径变量，遍历回到父层级时未重置路径，导致子文件夹路径被错误继承到父级书签。

修正后的递归实现 #

import org.jsoup.Jsoup
import org.jsoup.nodes.Element

data class Bookmark(val name: String, val url: String?, val path: String, val children: List<Bookmark> = emptyList())

fun parseBookmarks(html: String): List<Bookmark> {
    val doc = Jsoup.parse(html)
    val rootDl = doc.selectFirst("dl") ?: return emptyList()
    return parseDlElement(rootDl, "")
}

/**
 * 递归处理DL容器元素，严格对应书签文件夹结构
 */
private fun parseDlElement(dlElement: Element, parentPath: String): List<Bookmark> {
    val result = mutableListOf<Bookmark>()
    var currentFolder: Pair<String, Element>? = null

    // 仅直接遍历DT子元素，避免跨层级干扰
    val dtElements = dlElement.select("> dt")
    for (dt in dtElements) {
        val childNode = dt.child(0)
        when (childNode.tagName()) {
            "h3" -> {
                // 绑定文件夹名称与后续的DL子容器
                val folderName = childNode.text()
                val nextDl = dt.nextElementSibling()
                if (nextDl?.tagName() == "dl") {
                    currentFolder = folderName to nextDl
                }
            }
            "a" -> {
                // 直接解析书签链接，生成正确路径
                val bookmarkName = childNode.text()
                val bookmarkUrl = childNode.attr("href")
                val fullPath = if (parentPath.isEmpty()) bookmarkName else "$parentPath/$bookmarkName"
                result.add(Bookmark(bookmarkName, bookmarkUrl, fullPath))
            }
        }

        // 处理当前文件夹的子内容，完成后立即重置上下文
        currentFolder?.let { (folderName, dl) ->
            val folderPath = if (parentPath.isEmpty()) folderName else "$parentPath/$folderName"
            result.add(Bookmark(folderName, null, folderPath, parseDlElement(dl, folderPath)))
            currentFolder = null
        }
    }
    return result
}

核心优化点 #

1. 路径隔离：进入子文件夹时基于父路径生成新路径，退出时自动回退，避免路径污染。
2. 避免重复：仅在<h3>对应后续<dl>时递归子文件夹，<a>标签直接添加，无多余递归。
3. 上下文控制：用currentFolder变量跟踪当前文件夹上下文，处理后立即重置，避免干扰后续元素解析。

内容的提问来源于stack exchange，提问作者nyefine