如何在Flutter中用querySelector实现网页爬取并在Text Widget展示信息？

Hey there! Sorry to hear you've been stuck on this for two days—let's get this sorted out for you. First off, a quick heads-up: you can't directly use document.querySelectorAll() in Flutter Dart code because that's a browser-only API. Flutter runs outside the browser context, so we need to adjust our approach a bit.

Here's a step-by-step solution tailored to your goal:

1. 准备依赖包 #

First, you'll need two packages to make this work:

• http: To fetch the raw HTML content from the website
• html: To parse the HTML and use selector-based queries just like you would in the browser

Add these to your pubspec.yaml file:

dependencies:
  flutter:
    sdk: flutter
  http: ^1.1.0
  html: ^0.15.4

2. 静态网页内容提取（适用于非动态渲染的网站） #

If the website you're targeting serves static HTML (no JavaScript-based content loading), this method will work perfectly:

import 'package:flutter/material.dart';
import 'package:http/http.dart' as http;
import 'package:html/parser.dart' as parser;
import 'package:html/dom.dart' as dom;

class WebScrapeDemo extends StatefulWidget {
  const WebScrapeDemo({super.key});

  @override
  State<WebScrapeDemo> createState() => _WebScrapeDemoState();
}

class _WebScrapeDemoState extends State<WebScrapeDemo> {
  String displayText = 'Loading content...';

  @override
  void initState() {
    super.initState();
    // 页面初始化时就开始获取并解析内容
    _fetchAndParseWebContent();
  }

  Future<void> _fetchAndParseWebContent() async {
    try {
      // 1. 发送请求获取网页HTML
      final response = await http.get(Uri.parse('https://your-target-website.com'));
      
      // 检查请求是否成功
      if (response.statusCode != 200) {
        setState(() {
          displayText = 'Failed to load content. Status code: ${response.statusCode}';
        });
        return;
      }

      // 2. 解析HTML文档
      final dom.Document document = parser.parse(response.body);
      
      // 3. 用你指定的选择器提取元素
      final List<dom.Element> targetElements = document.querySelectorAll('td.title > a');
      
      // 4. 提取元素的文本内容并整理成字符串
      final List<String> extractedTexts = targetElements.map((element) => element.text.trim()).toList();
      
      // 更新UI，把内容展示在Text Widget里
      setState(() {
        displayText = extractedTexts.join('\n\n'); // 用换行分隔每个条目
      });
    } catch (error) {
      setState(() {
        displayText = 'Error occurred: $error';
      });
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Web Content Scraper')),
      body: Padding(
        padding: const EdgeInsets.all(16.0),
        child: Text(
          displayText,
          style: const TextStyle(fontSize: 16),
        ),
      ),
    );
  }
}

3. 动态网页内容提取（适用于JS渲染的网站） #

If the website uses JavaScript to load content dynamically (like React/Vue SPAs), the above method won't work because http.get() only fetches the initial static HTML framework. For these cases, use the webview_flutter package to load the page and run JavaScript to extract content:

First, add the package to pubspec.yaml:

dependencies:
  webview_flutter: ^4.4.0

Then use this code:

import 'package:flutter/material.dart';
import 'package:webview_flutter/webview_flutter.dart';

class DynamicWebScrapeDemo extends StatefulWidget {
  const DynamicWebScrapeDemo({super.key});

  @override
  State<DynamicWebScrapeDemo> createState() => _DynamicWebScrapeDemoState();
}

class _DynamicWebScrapeDemoState extends State<DynamicWebScrapeDemo> {
  String displayText = 'Loading content...';
  late WebViewController webViewController;

  @override
  void initState() {
    super.initState();
    webViewController = WebViewController()
      ..loadRequest(Uri.parse('https://your-dynamic-website.com'))
      ..setJavaScriptMode(JavaScriptMode.unrestricted)
      // 创建一个通道，让JS把提取的内容传回Dart
      ..addJavaScriptChannel(
        'ScrapeChannel',
        onMessageReceived: (JavaScriptMessage message) {
          setState(() {
            displayText = message.message;
          });
        },
      )
      ..setNavigationDelegate(
        NavigationDelegate(
          // 页面加载完成后执行JS代码提取内容
          onPageFinished: (String url) {
            webViewController.runJavaScript('''
              // 用你的选择器获取元素
              const elements = document.querySelectorAll('td.title > a');
              // 把元素文本转成字符串
              const textContent = Array.from(elements).map(el => el.textContent.trim()).join('\\n\\n');
              // 把内容传回Dart
              ScrapeChannel.postMessage(textContent);
            ''');
          },
        ),
      );
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Dynamic Web Scraper')),
      body: Column(
        children: [
          Expanded(child: WebViewWidget(controller: webViewController)),
          Padding(
            padding: const EdgeInsets.all(16.0),
            child: Text(
              displayText,
              style: const TextStyle(fontSize: 16),
            ),
          ),
        ],
      ),
    );
  }
}

Important Notes #

• Always check the target website's robots.txt and terms of service before scraping—don't violate their rules.
• Make sure your selector (td.title > a) is correct. You can test it in your browser's dev tools first to confirm it picks up the right elements.
• For mobile apps, you might need to add internet permissions (Android: add <uses-permission android:name="android.permission.INTERNET"/> to AndroidManifest.xml; iOS: no extra steps needed for modern Flutter versions).

内容的提问来源于stack exchange，提问作者William Davies