使用Scala Scraper爬取网站触发Incapsula拦截，求解决方法

1. 补全真实浏览器请求头 #

Incapsula会校验请求头的完整性，默认的JsoupBrowser请求头过于简单，容易被识别为爬虫。需要添加常见的浏览器请求头：

import net.ruippeixotog.scalascraper.browser.JsoupBrowser

val browser = JsoupBrowser()
val document = browser.get("https://esos.nv.gov/NotarySearchOnline/NotarySearchExternal",
  headers = Map(
    "User-Agent" -> "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
    "Accept" -> "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language" -> "en-US,en;q=0.5",
    "Referer" -> "https://esos.nv.gov/",
    "Upgrade-Insecure-Requests" -> "1"
  )
)
println(document)

2. 模拟浏览器执行JavaScript #

Incapsula依赖JavaScript生成验证令牌，Jsoup是无JS环境的爬虫工具，无法处理这类验证。可以使用支持JS渲染的工具，比如Playwright for Scala：

import com.microsoft.playwright._

object PlaywrightScraper {
  def main(args: Array[String]): Unit = {
    using(Playwright.create()) { playwright =>
      using(playwright.chromium.launch()) { browser =>
        val page = browser.newPage()
        page.navigate("https://esos.nv.gov/NotarySearchOnline/NotarySearchExternal")
        page.waitForLoadState(LoadState.NETWORK_IDLE)
        val content = page.content()
        println(content)
      }
    }
  }

  def using[A <: AutoCloseable, B](resource: A)(block: A => B): B = {
    try block(resource) finally resource.close()
  }
}

3. 维护会话Cookie #

Incapsula会在首次请求时设置会话相关的Cookie，后续请求需要携带这些Cookie才能通过验证：

import net.ruippeixotog.scalascraper.browser.JsoupBrowser
import org.jsoup.Connection.Response

val browser = JsoupBrowser()
// 先发起请求获取Cookie
val initialResponse: Response = browser.connection("https://esos.nv.gov/NotarySearchOnline/NotarySearchExternal")
  .headers(Map("User-Agent" -> "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"))
  .execute()

// 提取Cookie并复用
val cookies = initialResponse.cookies()
val document = browser.connection("https://esos.nv.gov/NotarySearchOnline/NotarySearchExternal")
  .headers(Map("User-Agent" -> "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"))
  .cookies(cookies)
  .get()

println(document)

4. 控制请求频率 #

频繁的请求会触发Incapsula的拦截机制，需要添加随机延迟模拟人类浏览行为：

import scala.util.Random

// 在两次请求之间添加1-3秒的随机延迟
Thread.sleep(Random.nextInt(2000) + 1000)

注意事项 #

• 确保你的爬取行为符合目标网站的robots.txt规则和服务条款，避免法律风险。
• Incapsula的反爬规则会动态更新，上述方案可能需要根据实际情况调整。

内容的提问来源于stack exchange，提问作者Zaryab Ali