基于vue-fuse的PDF搜索优化：缩减JSON数据量提升性能

Nice work getting vue-fuse up and running with PDF search—those large text payloads can be a real pain though! Let’s break down some performance-focused optimizations to shrink that JSON payload while keeping your search smooth:

1. 按需加载PDF文本（最直接的体积缩减） #

Instead of bundling the full PDF text with every article object, only fetch the text when you actually need it—either when a user initiates a search, or when they interact with a specific PDF. This cuts your initial JSON payload back to its original size immediately.

Frontend Adjustment Example: #

First, modify your article fetch API to exclude the text field by default (update your backend endpoint to support this, e.g., add a ?includeText=false parameter). Then, adjust your search logic to fetch text on-demand:

// Initialize your articles array without text fields
let articles = [];

// Fetch base article data first
async function loadArticles() {
  const response = await axios.get(API + '/articles?includeText=false');
  articles = response.data;
}

// Search function that fetches missing PDF text
async function performSearch(query) {
  // First, collect all PDF items that don't have text yet
  const pdfsToFetch = articles.filter(item => 
    item.originalName.includes('.pdf') && !item.text
  );

  // Fetch text for missing PDFs in parallel
  await Promise.all(pdfsToFetch.map(async item => {
    const textResponse = await axios.get(API + '/preprocess/pdf/' + item.id);
    item.text = textResponse.data;
  }));

  // Now run vue-fuse search on the full dataset
  const fuse = new Fuse(articles, {
    keys: ['fileName', 'originalName', 'text'],
    // Your existing vue-fuse options
  });

  return fuse.search(query);
}

2. 后端预生成搜索索引（彻底解决大文本传输） #

If you want to eliminate large text payloads entirely, move the search logic to the backend by generating a reverse index for your PDF content. Instead of sending full text to the frontend, the backend processes PDFs into a lightweight index (keywords mapped to article IDs), and you send only the search query to the backend—it returns matching article IDs, which you use to fetch the relevant article metadata.

Backend Indexing Example (Node.js/Express): #

// On PDF upload, generate and store an index entry
// Simplified example—add stemming/stopword removal for better results
async function processPDFText(articleId, text) {
  // Split text into meaningful keywords
  const keywords = text.toLowerCase().split(/\W+/).filter(word => word.length > 2);
  
  // Create a reverse index: keyword -> array of article IDs
  const indexEntry = keywords.reduce((acc, keyword) => {
    acc[keyword] = [...(acc[keyword] || []), articleId];
    return acc;
  }, {});

  // Save this index to your database (e.g., a separate "search_index" collection)
  await db.collection('search_index').insertOne(indexEntry);
}

// Search endpoint that uses the index
app.get('/search', async (req, res) => {
  const query = req.query.q.toLowerCase().split(/\W+/).filter(word => word.length > 2);
  
  // Find all article IDs matching any of the query keywords
  const matchingIds = new Set();
  for (const keyword of query) {
    const indexEntry = await db.collection('search_index').findOne({ [keyword]: { $exists: true } });
    if (indexEntry) indexEntry[keyword].forEach(id => matchingIds.add(id));
  }

  // Fetch full article metadata for matching IDs (no text needed!)
  const results = await db.collection('articles').find({ id: { $in: Array.from(matchingIds) } }).toArray();
  res.json(results);
});

Frontend Search Adjustment: #

async function performSearch(query) {
  const response = await axios.get(API + '/search?q=' + encodeURIComponent(query));
  return response.data; // Already filtered results—no need for vue-fuse on frontend!
}

3. Text Chunking + Lazy Loading（折中方案） #

For cases where you want to keep frontend search but need to reduce payload size, split large PDF texts into smaller chunks (e.g., by page, or every 1000 characters) and store them as separate entries in your database. Then, only load chunks as needed—either incrementally when the user scrolls, or dynamically during search.

Database Structure Adjustment: #

// Update your articles to have a "textChunks" array instead of a single "text" field
{
  fileName: String,
  id: String,
  originalName: String,
  url: String,
  textChunks: [{ id: String, content: String, pageNumber: Number }]
}

Frontend Lazy Search Example: #

async function searchChunks(query, articleId) {
  // Fetch only the chunks that might match
  const chunks = await axios.get(API + '/articles/' + articleId + '/text-chunks');
  const fuse = new Fuse(chunks.data, { keys: ['content'] });
  return fuse.search(query).length > 0; // Return if any chunk matches
}

async function performSearch(query) {
  // Check each article for matching chunks
  const matchingArticles = [];
  for (const article of articles) {
    if (!article.originalName.includes('.pdf')) {
      // Check non-PDF fields directly
      if (article.fileName.includes(query) || article.originalName.includes(query)) {
        matchingArticles.push(article);
      }
    } else {
      // Lazy load and check chunks
      const hasMatch = await searchChunks(query, article.id);
      if (hasMatch) matchingArticles.push(article);
    }
  }
  return matchingArticles;
}

4. Web Workers（UI体验优化） #

Even if you keep some text payloads, offload the vue-fuse search logic to a Web Worker so it doesn't block the main thread. This ensures your UI stays responsive even when processing large text datasets.

Web Worker File (search.worker.js): #

import Fuse from 'fuse.js'; // Make sure fuse.js is available in the worker

self.onmessage = async (e) => {
  const { articles, query, options } = e.data;
  const fuse = new Fuse(articles, options);
  const results = fuse.search(query);
  self.postMessage(results);
};

Frontend Usage: #

// Initialize the worker
const searchWorker = new Worker('./search.worker.js');

async function performSearch(query) {
  return new Promise((resolve) => {
    searchWorker.postMessage({
      articles: articles, // Pass your articles (with text if loaded)
      query: query,
      options: { keys: ['fileName', 'originalName', 'text'] }
    });

    searchWorker.onmessage = (e) => {
      resolve(e.data);
    };
  });
}

Pick the approach that fits your stack and needs best: 按需加载 is the quickest win, backend indexing is the most scalable long-term, and chunking works if you want to keep frontend search without full text payloads.

内容的提问来源于stack exchange，提问作者Tudor Sandu