HDFql迭代写入HDF5速度极慢，求代码问题排查与优化

This isn't an inherent limitation of HDFql—your code has several inefficiencies that are causing the slowdown. Let's break down the issues and fix them step by step:

Key Issues in Your Current Code #

• Repeatedly activating the HDF5 file: Every call to writeData runs USE FILE, which incurs unnecessary overhead. You only need to activate the file once after creating it.
• Individual dimension alterations and inserts: You're running separate ALTER DIMENSION and INSERT commands for each dataset (xs, ys, ts, ps) every iteration. Each command triggers HDF5 metadata operations and round-trips between your code and the HDFql engine, adding up to massive overhead.
• Redundant variable registration/unregistration: Registering and unregistering variables for each dataset in every iteration creates unnecessary processing overhead. You can register these variables once upfront.
• Overly specific insert indexing: The INSERT INTO events/xs(-%d:1:1:%d) syntax is unnecessary for appending to an unlimited dataset—HDFql can handle appending automatically without specifying the range.

Optimized Code #

Here's a revised version of your code that addresses these bottlenecks:

#include <HDFql.hpp>
#include <vector>
#include <string>

// Global state (can be encapsulated in a class for cleaner code)
std::string HDF5_path_;
int total_events_added_ = 0;
int events_idx_ = 0;
// Reusable variable handles
int xs_handle_, ys_handle_, ts_handle_, ps_handle_;

void createHDF(const std::string &filepath) {
    HDF5_path_ = filepath;
    char script_[1024];
    
    // Create and activate the file once
    sprintf(script_, "CREATE TRUNCATE FILE %s", filepath.c_str());
    HDFql::execute(script_);
    sprintf(script_, "USE FILE %s", filepath.c_str());
    HDFql::execute(script_);
    
    // Batch group creation commands to reduce round-trips
    sprintf(script_, "CREATE GROUP events; CREATE GROUP frames; CREATE GROUP optic_flow");
    HDFql::execute(script_);
    
    // Create chunked datasets (add custom chunk size if needed)
    HDFql::execute("CREATE CHUNKED DATASET events/xs AS SMALLINT(UNLIMITED)");
    HDFql::execute("CREATE CHUNKED DATASET events/ys AS SMALLINT(UNLIMITED)");
    HDFql::execute("CREATE CHUNKED DATASET events/ts AS DOUBLE(UNLIMITED)");
    HDFql::execute("CREATE CHUNKED DATASET events/ps AS TINYINT(UNLIMITED)");
}

void initWriteVariables(std::vector<double>& ts, std::vector<int16_t>& xs, std::vector<int16_t>& ys, std::vector<int8_t>& ps) {
    // Register variables once upfront instead of every iteration
    xs_handle_ = HDFql::variableRegister(&xs[0]);
    ys_handle_ = HDFql::variableRegister(&ys[0]);
    ts_handle_ = HDFql::variableRegister(&ts[0]);
    ps_handle_ = HDFql::variableRegister(&ps[0]);
}

void writeData(const std::vector<double>& ts, std::vector<int16_t>& xs, std::vector<int16_t>& ys, std::vector<int8_t>& ps) {
    const int data_size = ts.size();
    char script_[1024];
    
    // Batch all dimension alterations into a single command
    sprintf(script_, "ALTER DIMENSION events/xs TO +%d; ALTER DIMENSION events/ys TO +%d; ALTER DIMENSION events/ts TO +%d; ALTER DIMENSION events/ps TO +%d", 
            data_size, data_size, data_size, data_size);
    HDFql::execute(script_);
    
    // Simplify inserts (auto-append to unlimited datasets)
    sprintf(script_, "INSERT INTO events/xs VALUES FROM MEMORY %d", xs_handle_);
    HDFql::execute(script_);
    
    sprintf(script_, "INSERT INTO events/ys VALUES FROM MEMORY %d", ys_handle_);
    HDFql::execute(script_);
    
    sprintf(script_, "INSERT INTO events/ts VALUES FROM MEMORY %d", ts_handle_);
    HDFql::execute(script_);
    
    sprintf(script_, "INSERT INTO events/ps VALUES FROM MEMORY %d", ps_handle_);
    HDFql::execute(script_);
    
    total_events_added_ += data_size;
    events_idx_++;
}

int main (int argc, const char * argv[]) {
    std::string path = "/tmp/test.h5";
    createHDF(path);
    
    const int data_size = 1000;
    const int iterations = 10000;
    std::vector<double> ts(data_size);
    std::vector<int16_t> xs(data_size);
    std::vector<int16_t> ys(data_size);
    std::vector<int8_t> ps(data_size);
    
    // Fix typo from original code and populate test data
    for(int i=0; i<data_size; i++) {
        ts[i] = i;
        xs[i] = i;
        ys[i] = i;
        ps[i] = 1;
    }
    
    // Initialize variables before starting iterations
    initWriteVariables(ts, xs, ys, ps);
    
    for(int i=0; i<iterations; i++) {
        writeData(ts, xs, ys, ps);
    }
    
    // Clean up registered variables
    HDFql::variableUnregister(xs_handle_);
    HDFql::variableUnregister(ys_handle_);
    HDFql::variableUnregister(ts_handle_);
    HDFql::variableUnregister(ps_handle_);
    
    return 0;
}

Additional Optimizations to Boost Speed #

• Custom chunk sizes: When creating datasets, specify a chunk size that matches your batch size (e.g., CREATE CHUNKED DATASET events/xs AS SMALLINT(UNLIMITED) CHUNK SIZE 1000). This aligns your writes with HDF5's internal storage blocks, reducing fragmentation.
• Enable HDF5 caching: Add HDFql::execute("SET CACHE SIZE 1048576") (1MB cache) after activating the file to speed up metadata and data access.
• Larger batches: Process bigger chunks of data per iteration if possible—fewer iterations mean fewer ALTER/INSERT cycles.

Why This Fixes the Slowdown #

By reducing round-trips between your code and the HDFql engine, reusing variable registrations, and batching commands, you eliminate most of the unnecessary overhead. HDFql is fully capable of matching the performance of other libraries when used efficiently—your original code was just doing redundant work on every iteration.

内容的提问来源于stack exchange，提问作者Mr Squid