使用Webpack整合Google Map API及爬虫依赖时的模块错误求助

核心问题根源 #

Puppeteer、Selenium-webdriver是Node.js专属的后端爬虫工具，完全无法在浏览器环境运行。你试图用Webpack把这些后端模块打包到前端代码里，这是本质上的架构错误，也是所有报错的核心原因。

分步修复方案 #

1. 拆分前后端代码架构 #

• 前端代码（src/app.js）仅保留前端相关依赖，移除爬虫工具的导入：

  import { Loader } from '@googlemaps/js-api-loader';
  import * as THREE from 'three';
  import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader.js';
  import { Tween, Easing } from 'tween';
  // 移除 import puppeteer from 'puppeteer';
  

• 新建独立的Node.js脚本（比如crawler.js），专门处理爬虫逻辑，直接用node crawler.js运行，不需要Webpack打包：

  const puppeteer = require('puppeteer');
  
  (async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    // 你的爬虫逻辑
    await browser.close();
  })();
  

2. 修复Webpack配置与依赖 #

调整依赖包

• 卸载无用的后端专属依赖：

  npm uninstall puppeteer selenium-webdriver async_hooks child_process
  

• 安装浏览器兼容的url替代包：

  npm install url-browserify --save
  

修改Webpack.config.js

修正resolve.fallback和插件配置，移除浏览器不需要的Node端模块映射：

const path = require('path');
const webpack = require('webpack');

const CONFIG = {
  mode: 'development',
  entry: {
    app: './src/app.js',
  },
  devServer: {
    static: {
      directory: path.join(__dirname, 'src'),
    },
  },
  resolve: {
    fallback: {
      path: require.resolve('path-browserify'),
      fs: false,
      os: require.resolve('os-browserify/browser'),
      https: require.resolve('https-browserify'),
      stream: require.resolve('stream-browserify'),
      http: require.resolve('stream-http'),
      constants: require.resolve('constants-browserify'),
      zlib: require.resolve('browserify-zlib'),
      crypto: require.resolve('crypto-browserify'),
      tls: false, // 浏览器不需要tls模块
      net: false, // 浏览器不需要net模块
      async_hooks: false, // Node核心模块，浏览器无对应实现
      querystring: require.resolve('querystring-es3'),
      child_process: false,
      readline: false,
      url: require.resolve('url-browserify'), // 用浏览器兼容的url包
      dns: false,
      module: false,
    },
  },
  plugins: [
    // 移除fs的ProvidePlugin，浏览器环境不需要fs
  ],
};

module.exports = CONFIG;

3. 清理冗余依赖 #

调整后的package.json依赖示例（仅保留必要项）：

"dependencies": {
  "@googlemaps/js-api-loader": "^1.16.2",
  "browserify-zlib": "^0.2.0",
  "constants-browserify": "^1.0.0",
  "crypto-browserify": "^3.12.0",
  "graceful-fs": "^4.2.11",
  "https-browserify": "^1.0.0",
  "os-browserify": "^0.3.0",
  "path-browserify": "^1.0.1",
  "querystring-es3": "^0.2.1",
  "stream-browserify": "^3.0.0",
  "stream-http": "^3.2.0",
  "three": "^0.138.3",
  "tween": "^0.9.0",
  "url-browserify": "^1.1.0",
  "util": "^0.12.5"
},
"devDependencies": {
  "webpack": "^5.70.0",
  "webpack-cli": "^4.9.2",
  "webpack-dev-server": "^4.7.4"
}

验证修复 #

1. 运行前端：npx webpack serve，检查Google Maps和Three.js是否正常加载，无报错。
2. 运行爬虫：node crawler.js，验证爬虫逻辑正常执行。

内容的提问来源于stack exchange，提问作者Yan