寻求将MqlTick数组按列反解析为JSON的高效方法

I get it—dealing with large tick datasets (like an entire day's worth) and converting them into that specific column-based JSON format can be a pain if you don't optimize for speed and memory. Let's break down how to do this efficiently, avoiding the slow string concatenation that can choke performance with thousands of ticks.

First, let's recap what we're working with: the CopyTicksRange() function returns an array of MqlTick structs, and we need to restructure these into a JSON where each field (time, bid, ask, etc.) is its own array under the ticks key.

Step 1: Fetch the Tick Data #

First, grab your tick array using CopyTicksRange. For your example (March 24, 2018 full day), we'll define the time range and populate the array:

MqlTick ticksArray[];
datetime startDate = D'2018.03.24 00:00:00';
datetime endDate = D'2018.03.25 00:00:00';

int tickCount = CopyTicksRange(_Symbol, ticksArray, startDate, endDate, COPY_TICKS_ALL);
if(tickCount <= 0) {
    Print("Failed to copy ticks. Error code: ", GetLastError());
    return;
}

Step 2: Optimize JSON String Building #

The biggest bottleneck with large datasets is repeated string allocation. We'll use StringReserve() to pre-allocate enough memory for our final JSON, which drastically speeds up the process. We'll also build each column array first, then assemble the full JSON.

Helper Function to Convert Datetime to String #

Since the JSON expects a <string> for the time field, we'll need a helper to format the datetime value into a readable string:

string DatetimeToString(datetime dt) {
    return StringFormat("%04d.%02d.%02d %02d:%02d:%02d",
                        Year(dt), Month(dt), Day(dt),
                        Hour(dt), Minute(dt), Second(dt));
}

Step 3: Build Column Arrays and Assemble JSON #

Now we'll construct each column's array string, then wrap them into the final JSON structure. We'll avoid nested loops where possible and use direct string appends with pre-allocated space:

string GenerateColumnBasedJSON(MqlTick ticks[], int tickCount) {
    if(tickCount == 0) return "{\"ticks\": []}";

    // Pre-reserve memory to avoid reallocations (adjust based on your average tick JSON length)
    int estimatedSize = tickCount * 60; // Rough estimate: ~60 chars per tick across all columns
    string json = "";
    StringReserve(json, estimatedSize + 100); // Add buffer for JSON structure

    // Start building the JSON wrapper
    json = "{\"ticks\": [";

    // --- Build Time column (string)
    json += "[\"";
    for(int i = 0; i < tickCount; i++) {
        if(i > 0) json += "\", \"";
        json += DatetimeToString(ticks[i].time);
    }
    json += "\"], ";

    // --- Build Bid column (float)
    json += "[";
    for(int i = 0; i < tickCount; i++) {
        if(i > 0) json += ", ";
        json += DoubleToString(ticks[i].bid, _Digits); // Use symbol's digit count for precision
    }
    json += "], ";

    // --- Build Ask column (float)
    json += "[";
    for(int i = 0; i < tickCount; i++) {
        if(i > 0) json += ", ";
        json += DoubleToString(ticks[i].ask, _Digits);
    }
    json += "], ";

    // --- Build Last column (float)
    json += "[";
    for(int i = 0; i < tickCount; i++) {
        if(i > 0) json += ", ";
        json += DoubleToString(ticks[i].last, _Digits);
    }
    json += "], ";

    // --- Build Volume column (long)
    json += "[";
    for(int i = 0; i < tickCount; i++) {
        if(i > 0) json += ", ";
        json += (string)ticks[i].volume;
    }
    json += "], ";

    // --- Build Time_msc column (long)
    json += "[";
    for(int i = 0; i < tickCount; i++) {
        if(i > 0) json += ", ";
        json += (string)ticks[i].time_msc;
    }
    json += "], ";

    // --- Build Flags column (int)
    json += "[";
    for(int i = 0; i < tickCount; i++) {
        if(i > 0) json += ", ";
        json += (string)ticks[i].flags;
    }
    json += "]";

    // Close the JSON wrapper
    json += "]}";

    return json;
}

Step 4: Use the Function and Handle Output #

Call the function and save or output the JSON. For large datasets, writing directly to a file instead of holding the entire string in memory might be better (we'll add that as an optimization):

// Main usage
string resultJSON = GenerateColumnBasedJSON(ticksArray, tickCount);

// Optional: Write to file to avoid memory issues with huge JSON
int fileHandle = FileOpen("tick_data.json", FILE_WRITE | FILE_TXT | FILE_ANSI);
if(fileHandle != INVALID_HANDLE) {
    FileWriteString(fileHandle, resultJSON);
    FileClose(fileHandle);
    Print("JSON saved to tick_data.json");
} else {
    Print("Failed to open file. Error code: ", GetLastError());
}

Key Optimizations for Large Datasets #

1. Pre-Reserve String Memory: StringReserve cuts down on expensive memory reallocations that happen when appending to a string repeatedly.
2. Minimize String Concatenation: We build each column in a single loop instead of nested loops, which reduces the number of string operations.
3. File Writing Instead of In-Memory Storage: For very large JSON (like your ~9MB example), writing directly to a file as you build the string can save memory. You could modify the function to write chunks to the file instead of building the entire string first.
4. Precision Control: Using DoubleToString(tick.bid, _Digits) ensures we don't add unnecessary decimal places, keeping the JSON size smaller.

Edge Cases to Handle #

• Empty Tick Array: The function returns a valid empty JSON structure if no ticks are found.
• Error Handling: Always check the return value of CopyTicksRange and file operations to catch issues early.

内容的提问来源于stack exchange，提问作者Marcel Boldt