如何在Webpack 5打包的JavaScript应用中读取Parquet格式数据？

针对你使用Webpack 5打包Parquet解码库失败的问题，以下是经过验证的可行方案：

方案1：优化Parquet-wasm的Webpack配置 #

Parquet-wasm是基于WebAssembly的轻量库，Webpack 5原生支持Wasm，但需正确配置：

核心Webpack配置修改 #

module.exports = {
  module: {
    rules: [
      {
        test: /\.wasm$/,
        type: 'webassembly/async',
      },
    ],
  },
  resolve: {
    fallback: {
      "fs": false,
      "path": require.resolve("path-browserify"),
      "crypto": require.resolve("crypto-browserify")
    }
  },
  experiments: {
    asyncWebAssembly: true,
  },
};

代码使用示例 #

采用动态导入避免打包时的预编译问题：

async function decodeParquetFromUrl(url) {
  const { ParquetReader } = await import('parquet-wasm');
  const response = await fetch(url);
  const arrayBuffer = await response.arrayBuffer();
  
  const reader = await ParquetReader.open(new Uint8Array(arrayBuffer));
  const cursor = reader.getCursor();
  
  let record;
  while ((record = await cursor.next())) {
    // 处理单条Parquet记录
    console.log(record);
  }
  
  reader.close();
}

方案2：正确配置@duckdb/duckdb-wasm #

DuckDB-wasm支持直接查询Parquet文件，需补充Webpack对Node.js核心模块的兼容配置：

核心Webpack配置修改 #

const webpack = require('webpack');

module.exports = {
  resolve: {
    fallback: {
      "stream": require.resolve("stream-browserify"),
      "buffer": require.resolve("buffer"),
      "util": require.resolve("util"),
      "crypto": require.resolve("crypto-browserify")
    }
  },
  plugins: [
    new webpack.ProvidePlugin({
      Buffer: ['buffer', 'Buffer'],
      process: 'process/browser',
    }),
  ],
  experiments: {
    asyncWebAssembly: true,
  },
};

代码使用示例 #

import * as duckdb from '@duckdb/duckdb-wasm';

async function queryParquetFile(url) {
  // 加载DuckDB Wasm资源
  const bundles = duckdb.getJsDelivrBundles();
  const bundle = await duckdb.selectBundle(bundles);
  const worker = new Worker(bundle.mainWorkerUrl);
  const db = new duckdb.AsyncDuckDB(worker, bundle.mainModuleUrl);
  await db.instantiate();

  // 下载并注册Parquet文件
  const response = await fetch(url);
  const buffer = await response.arrayBuffer();
  await db.registerFileBuffer('temp.parquet', new Uint8Array(buffer));

  // SQL查询获取数据
  const result = await db.query('SELECT * FROM temp.parquet LIMIT 20');
  console.log(result.toArray());

  await db.close();
}

方案3：Parquet-wasm + Apache Arrow 组合（高效数据处理） #

如果需要更高效的内存数据操作，可结合Apache Arrow库，将Parquet数据转为Arrow Table：

代码使用示例 #

import { Table } from 'apache-arrow';

async function loadParquetAsArrow(url) {
  const { ParquetReader } = await import('parquet-wasm');
  const response = await fetch(url);
  const arrayBuffer = await response.arrayBuffer();
  
  const reader = await ParquetReader.open(new Uint8Array(arrayBuffer));
  const arrowTable = await reader.readArrowTable();
  const table = Table.from(arrowTable);

  // 以Arrow Table格式处理数据，支持列操作、过滤等
  console.log(table.toArray());
  reader.close();
}

通用打包问题排查 #

1. Node.js核心模块缺失：Webpack 5不再自动 polyfill Node.js核心模块，需通过resolve.fallback补全对应依赖（如path-browserify、crypto-browserify）
2. Wasm加载失败：确保开启experiments.asyncWebAssembly，并使用异步导入方式加载Wasm相关库
3. 全局变量缺失：通过webpack.ProvidePlugin注入Buffer、process等全局变量

内容的提问来源于stack exchange，提问作者julien