Great question—handling filtered pagination efficiently in Solidity is no small feat, especially with a 100k-entity dataset. On-chain operations are gas-heavy, and we don’t have native query support, so your current setup’s pain points (rigid filters, split function calls) are totally understandable. Let’s break down the best approaches to fix this and build a cleaner, more flexible system.

First, Let’s Diagnose Your Current Implementation’s Issues #

Before diving into solutions, let’s call out why your current setup isn’t ideal:

• Split function calls: Requiring three separate calls to fetch a single entity’s full data adds unnecessary frontend complexity and gas overhead (each call has base gas costs).
• Rigid filtering: Using filterAsUint hardcodes filter logic—adding new conditions means modifying the contract, which is inflexible and costly if your contract is already deployed.
• Inefficient traversal: If your functions loop through the entire items array to find matching entities, this will get prohibitively expensive for large offsets or sparse filter results.

Best Solution: Pre-Indexed Pagination + Flexible Filtering #

The core idea here is to avoid full array traversals at all costs (they’re gas killers) and structure your contract to support both common and ad-hoc filters without constant redeployment.

1. Pre-Index Common Filter Criteria #

For frequently used filters (like your proposalStatus), maintain mapping-based indexes that link filter values to lists of entity IDs. This lets you jump directly to matching entities instead of scanning the entire dataset.

Example implementation:

// Assume ProposalStatus is defined as:
enum ProposalStatus { Pending, Approved, Rejected }

struct EntityA {
    string lessThen32ByteString1;
    string moreThen32ByteString1;
    string lessThen32ByteString2;
    string moreThen32ByteString3;
    bool flag;
    uint var1;
    uint var2;
    uint var3;
    uint var4;
    ProposalStatus proposalStatus;
}

EntityA[] public items;
// Index: Map each ProposalStatus to a list of entity IDs with that status
mapping(ProposalStatus => uint[]) public statusToEntityIds;

// Sync index when adding a new entity
function addEntity(EntityA calldata newEntity) external {
    uint entityId = items.length;
    items.push(newEntity);
    statusToEntityIds[newEntity.proposalStatus].push(entityId);
}

With this index, pagination becomes trivial and gas-efficient:

function getFilteredPageByStatus(ProposalStatus status, uint offset, uint limit) external view returns (uint[] memory pageEntityIds) {
    uint[] memory matchingIds = statusToEntityIds[status];
    
    // Handle edge case: offset exceeds total matching entities
    if (offset >= matchingIds.length) {
        return new uint[](0);
    }
    
    // Calculate end index, cap at total matching entities
    uint endIndex = offset + limit;
    if (endIndex > matchingIds.length) {
        endIndex = matchingIds.length;
    }
    
    // Extract the page of IDs
    pageEntityIds = new uint[](endIndex - offset);
    for (uint i = offset; i < endIndex; i++) {
        pageEntityIds[i - offset] = matchingIds[i];
    }
    return pageEntityIds;
}

2. Optimize Data Retrieval (No More Split Calls) #

Instead of splitting entity data across three functions, create a single function to fetch full entities by ID (or batch fetch multiple entities at once). This reduces frontend calls and simplifies logic.

Example batch fetch function:

function getEntitiesByIds(uint[] calldata entityIds) external view returns (EntityA[] memory entities) {
    entities = new EntityA[](entityIds.length);
    for (uint i = 0; i < entityIds.length; i++) {
        entities[i] = items[entityIds[i]];
    }
    return entities;
}

How to Use This Flow:

1. Frontend calls getFilteredPageByStatus with the desired status, offset, and limit to get a list of entity IDs.
2. Frontend uses a multicall (like EIP-1193 batch calls) to fetch all full entities via getEntitiesByIds in one go.

This cuts down on gas costs and frontend complexity significantly.

3. Add Flexible Filtering for Ad-Hoc Queries #

If you need to support combinations of filters (e.g., proposalStatus + flag + var1 range), use a structured filter parameter instead of a single filterAsUint. This makes the contract extensible without breaking function signatures.

Example:

struct FilterParams {
    ProposalStatus status; // Use a special value like `ProposalStatus.All` to ignore this filter
    bool flagEnabled; // Toggle whether to apply the flag filter
    bool flag;
    uint var1Min;
    uint var1Max;
}

// For ad-hoc filters (note: this involves traversal, use sparingly for large datasets)
function getFilteredPage(FilterParams calldata filters, uint offset, uint limit) external view returns (uint[] memory pageEntityIds) {
    uint[] memory matchingIds = new uint[](0);
    uint matchCount = 0;

    // Traverse full array (only do this if pre-indexes aren't feasible)
    for (uint i = 0; i < items.length; i++) {
        EntityA storage entity = items[i];
        
        // Apply filter logic
        bool matches = true;
        if (filters.status != ProposalStatus.All) {
            matches = matches && (entity.proposalStatus == filters.status);
        }
        if (filters.flagEnabled) {
            matches = matches && (entity.flag == filters.flag);
        }
        matches = matches && (entity.var1 >= filters.var1Min) && (entity.var1 <= filters.var1Max);

        if (matches) {
            // Skip until we reach the offset
            if (matchCount >= offset) {
                matchingIds = append(matchingIds, i);
                // Stop once we hit the limit
                if (matchingIds.length == limit) break;
            }
            matchCount++;
        }
    }
    return matchingIds;
}

// Helper to append to a memory array (Solidity doesn't have built-in append)
function append(uint[] memory arr, uint value) internal pure returns (uint[] memory) {
    uint[] memory newArr = new uint[](arr.length + 1);
    for (uint i = 0; i < arr.length; i++) {
        newArr[i] = arr[i];
    }
    newArr[arr.length] = value;
    return newArr;
}

Note: This ad-hoc filter function will be gas-heavy for large datasets, so reserve it for rare, non-critical queries. Prioritize pre-indexes for common filters.

4. Gas Optimization Tips #

• Store short strings as bytes32: For strings under 32 bytes, replace string with bytes32 to save storage and reading gas.
• Limit batch sizes: Cap limit to a reasonable maximum (like 100) to avoid stack overflow or excessive gas costs.
• Use memory arrays: Always use memory for temporary arrays in view functions to avoid unnecessary storage access costs.
• Handle deletions carefully: If you ever need to delete entities, use a "soft delete" approach (mark as inactive) and maintain your indexes accordingly—rebuilding indexes from scratch is prohibitively expensive.

Summary #

Your best bet is to:

1. Pre-index common filters (like proposalStatus) to enable fast, gas-efficient pagination.
2. Batch entity retrieval to eliminate split function calls.
3. Use structured filter parameters for ad-hoc queries (when pre-indexes aren’t feasible).

This setup balances flexibility, gas efficiency, and maintainability—critical for handling large on-chain datasets.

内容的提问来源于stack exchange，提问作者Igor Golovan