MarkLogic中distinct-values返回重复值问题及去重方法咨询

Let's break down why you're seeing repeated instances of "EARTH RESEARCH LLC" and how to fix it.

The Root Cause #

Your current query applies distinct-values() per document instead of across all documents. Here's what's happening:

• You loop through each URI in your collection, fetch the document, then run distinct-values() on the Name elements within that single document.
• If multiple documents contain "EARTH RESEARCH LLC", each document will return that value (after removing duplicates inside itself), leading to duplicates in the final output.

Fix 1: Global distinct-values() Wrap #

The simplest fix is to collect all Name values from all documents first, then apply distinct-values() to the entire sequence:

distinct-values(
  for $uri in cts:uris((), (), cts:and-query(cts:collection-query("/citation/company")))[1 to 1000]
  return doc($uri)/PerformingOrganizations/Name
)

This way, you're deduplicating across every Name element from all 1000 documents, not just within each individual document.

Fix 2: Use Indexes for Better Performance #

If you have an element range index configured for the Name element (under /PerformingOrganizations), you can skip document traversal entirely and use cts:values() to get unique values directly from the index. This is way faster, especially for large datasets:

cts:values(
  cts:element-reference(xs:QName("Name")),
  (),
  cts:and-query(cts:collection-query("/citation/company"))
)[1 to 1000]

Just make sure the index is enabled in your MarkLogic database configuration (check under "Element Range Indexes" and add Name with the appropriate data type, usually xs:string).

Bonus: Handle Edge Cases #

If you notice duplicates due to whitespace differences (like "EARTH RESEARCH LLC" vs " EARTH RESEARCH LLC ") or casing, normalize the values before deduplication:

distinct-values(
  for $uri in cts:uris((), (), cts:and-query(cts:collection-query("/citation/company")))[1 to 1000]
  return normalize-space(lower-case(doc($uri)/PerformingOrganizations/Name))
)

Or with cts:values() (using a collation that ignores case/whitespace):

cts:values(
  cts:element-reference(xs:QName("Name")),
  ("collation=http://marklogic.com/collation/en/S2"),
  cts:and-query(cts:collection-query("/citation/company"))
)[1 to 1000]

The S2 collation treats case and whitespace variations as identical values, helping catch more subtle duplicates.

内容的提问来源于stack exchange，提问作者thichxai