使用MarkLogic CORB2工具时的URI分页及CSV输出扩展名问题

1. Paging URIs to Avoid Memory Issues with 5M+ Documents #

Yes, CORB2 supports URI paging to prevent loading all URIs into memory at once. Here's how to implement it:

• Add the URIS_BATCH_SIZE parameter to your CORB configuration (e.g., URIS_BATCH_SIZE=10000). Adjust the value based on your MarkLogic server's memory capacity—start with 10k-50k and tweak if needed.
• Modify your URIS-MODULE to accept $start and $page-size external variables, then pass them to cts:uris():

  xquery version "1.0-ml";
  declare variable $start as xs:integer external;
  declare variable $page-size as xs:integer external;
  
  cts:uris(
    (),
    (),
    cts:element-query(xs:QName("your-target-element"), cts:your-query-here()),
    $start,
    $page-size
  )
  

• Disable URI sorting if you don't need ordered results by setting URI-SORT=false—this reduces memory overhead significantly for large datasets.

CORB will repeatedly call your URIS module, incrementing the $start offset each time, until all matching URIs are fetched.

2. Adding .csv Extension to Batch Export Files #

To append .csv to all batch output files, use the EXPORT-FILE-NAME-PATTERN parameter in your CORB configuration:

• Set the parameter to a pattern that includes the .csv extension. For example:

  EXPORT-FILE-NAME-PATTERN=export-batch-%d.csv
  

• The %d placeholder is replaced with the sequential batch number (starting at 1), resulting in filenames like export-batch-1.csv, export-batch-2.csv, etc.
• Ensure this parameter is included alongside EXPORT-FILE-DIR and URIS_BATCH_REF=FALSE in your setup.

This works because EXPORT-FILE-NAME-PATTERN overrides the default batch filename format, letting you define custom extensions and naming conventions.

内容的提问来源于stack exchange，提问作者Madoantony