基于Offset与Limit从多JSON文件中获取指定范围Item的算法实现方案咨询

Problem Statement #

I want to build a JSON object from multiple JSON files while controlling the offset and limit parameters. Each JSON file follows the naming format file-xxx.json where xxx is the file number; each file contains a maximum number of Items defined by a variable (e.g., const maxItemsPerJSONFile = 10). The core goal is to avoid fetching all JSON files and only retrieve the ones we need.

JSON file structure example:

[ 
  { "item": "value", "anotherItem": "value" },
  { "item": "value", "anotherItem": "value" },
  ...
]

Current dilemma: I'm struggling to design an algorithm that can fetch the specified range of Items (each top-level object in the JSON files) based on the offset and limit parameters.

Expected code framework:

function searchJSONFiles(offset, limit){
  return new Promise((resolve, reject) => {
    let finalResults = [];
    function main(){
      // Calculate "fileNumber"
      // ...
      $.ajax({
        url: `/directory/file-${fileNumber}.json`,
        type: 'GET',
        success: function(response){
          if(/* There are more files to be fetched, based on the limit */){
            // Push current results from "response" to "finalResults" while respecting the offset
            // ...
            // Recalculate offset and limit and execute the main function again
            // ...
            main();
          } else {
            // Push final results from "response" to "finalResults" while respecting the offset
            // ...
            // Resolve and return the results
            resolve(finalResults);
          }
        }
      });
    }
  });
}

How can I implement this requirement?

Let's break down the core logic first—we need to figure out which files to fetch, where to start in each file, and how much data to take from each. Here's a complete, functional implementation with detailed explanations:

// Define the maximum number of items per JSON file
const maxItemsPerJSONFile = 10;

function searchJSONFiles(offset, limit){
  return new Promise((resolve, reject) => {
    let finalResults = [];
    // Calculate the first file we need to fetch (assuming files start at 1: file-1.json)
    let currentFileNumber = Math.floor(offset / maxItemsPerJSONFile) + 1;
    // Calculate how many items to skip in the starting file
    let currentOffsetInFile = offset % maxItemsPerJSONFile;

    function main(){
      // Exit early if we've already collected all needed items
      if(limit <= 0){
        resolve(finalResults);
        return;
      }

      $.ajax({
        url: `/directory/file-${currentFileNumber}.json`,
        type: 'GET',
        dataType: 'json',
        success: function(response){
          const fileItemCount = response.length;
          // How many items we can take from this file: the smaller of remaining limit or available items in the file
          const takeCount = Math.min(limit, fileItemCount - currentOffsetInFile);

          if(takeCount > 0){
            // Slice the needed items from the current file and add to results
            const itemsToAdd = response.slice(currentOffsetInFile, currentOffsetInFile + takeCount);
            finalResults.push(...itemsToAdd);
            // Update remaining items we need to fetch
            limit -= takeCount;
          }

          // Check if we need to fetch the next file: still need items, and we've exhausted all available items in current file
          if(limit > 0 && (currentOffsetInFile + takeCount) >= fileItemCount){
            currentFileNumber++;
            // Reset offset for the next file (start from the first item)
            currentOffsetInFile = 0;
            main();
          } else {
            // We have all the items we need, or the current file has leftover items we don't need
            resolve(finalResults);
          }
        },
        error: function(xhr, status, error){
          // Handle request errors (e.g., file not found) to avoid hanging promises
          reject(new Error(`Failed to fetch file-${currentFileNumber}.json: ${error}`));
        }
      });
    }

    // Kick off the main fetch process
    main();
  });
}

Key Logic Breakdown #

1. Calculate Starting File & Offset:

   • currentFileNumber: Uses integer division to find which file contains the first item we need. For example, if offset is 15 and each file has 10 items, we start at file 2.
   • currentOffsetInFile: Uses modulus to find the index of the first item we need in the starting file. For offset 15, we start at index 5 of file 2.

2. Process Each File:

   • takeCount: Ensures we don't take more items than we need (remaining limit) or more than the file has available.
   • After slicing and adding items to results, we update the remaining limit. If we still need more items and have exhausted the current file, we move to the next file and reset the offset.

3. Termination Conditions:

   • Stop immediately if we've collected all limit items.
   • Stop when we can't get more items from the current file and don't need any more.

Additional Notes #

• If your files start at 0 (e.g., file-0.json), adjust currentFileNumber to Math.floor(offset / maxItemsPerJSONFile) (remove the +1).
• The dataType: 'json' ensures jQuery parses the response correctly as a JSON array.
• Error handling prevents the promise from staying in a pending state if a file can't be fetched.

内容的提问来源于stack exchange，提问作者Looky1173