Python调用Kaltura API出现QUERY_EXCEEDED_MAX_MATCHES_ALLOWED错误求助

问题描述 #

使用Python调用Kaltura API拉取全量条目时触发ApiException，报错信息为Unable to generate list. max matches value was reached，错误码为QUERY_EXCEEDED_MAX_MATCHES_ALLOWED，无法拉取全部条目。

已尝试的实现 #

初始化逻辑（已配置disableentitlement权限） #

class class_chk_integrity():
    client = None
    pagesize = 0

    def __init__(self,worker_num, progress):
        self.pagesize = 30
        self.worker_num = worker_num
        self.progress = progress

        config = KalturaConfiguration(2723521)
        config.serviceUrl = "https://www.kaltura.com/"
        self.client = KalturaClient(config)
        ks = self.client.session.start("KALTURA_ADMIN_SECRET",
            "email@email.com",
            KalturaPluginsCore.KalturaSessionType.ADMIN,
            "KALTURA_PARTNER_ID",
            432000,
            "disableentitlement")
        self.client.setKs(ks)

条目拉取逻辑（idNotIn过滤不生效） #

def get_total_reg(self, cont, lastEntryIds, lastEntryCreatedAt):
    filter = KalturaPluginsCore.KalturaBaseEntryFilter()

    if lastEntryIds != "":
        filter.idNotIn = lastEntryIds

    filter.orderBy = KalturaBaseEntryOrderBy.CREATED_AT_DESC

    pager = KalturaPluginsCore.KalturaFilterPager()
    
    pageIndex = 1
    entriesGot = 0

    pager.pageSize = self.pagesize
    pager.setPageIndex = pageIndex
    result = self.client.baseEntry.list(filter, pager)
    totalCount = result.totalCount

    if totalCount > 10000:
        totalCount = 9970

    if totalCount <= 0:
        cont = False
    
    while entriesGot < totalCount:
        pager.pageSize = self.pagesize
        pageIndex += 1
        pager.pageIndex = pageIndex
        result = self.client.baseEntry.list(filter, pager)
        entriesGot += len(result.objects)

        for e in result.objects:
            if lastEntryIds == "":
                lastEntryIds.append(e.id)
            else:
                lastEntryIds.append(e.id)
            lastEntryCreatedAt = e.createdAt
    
    return result.totalCount, self.pagesize, cont, lastEntryIds, lastEntryCreatedAt

主函数调用逻辑 #

if __name__ == '__main__':
    try:
        log = _ServiceUtils.log()
        log.setup('all', 'integrity')
        cont = True
        lastEntryIds = []
        lastEntryCreatedAt = 0

        while cont is True:
            kmc = class_chk_integrity(0,0)
        
            kmc_total_reg, kmc_page_size, cont, lastEntryIds, lastEntryCreatedAt = kmc.get_total_reg(cont, lastEntryIds, lastEntryCreatedAt)

            interval = 10
            max_threads = math.ceil(kmc_total_reg / (interval * kmc_page_size))
            # max_threads = 1
            threads_list = []

            print('TOTAL REG : %s | PAGE_SIZE : %s | INTERVAL : %s | THREADS : %s' % (kmc_total_reg,kmc_page_size,interval,max_threads))  

            progress = class_progress_thread(max_threads)

            for index in range(0,max_threads):
                page_ini = index * interval
                page_end = index * interval + interval
                progress.add_worker_progress(index,datetime.now())

                threads_list.append(threading.Thread(target=thread_chk_integrity, args=(index, log, index * interval + 1,index * interval + interval,progress)))
        
            threads_list.append(threading.Thread(target=thread_output_progress, args=(progress,max_threads)))    

            for thread in threads_list:
                thread.start()

            for thread in threads_list:
                thread.join()

            while not progress.stop(): time.sleep(30)

    except KeyboardInterrupt:
        try:
            sys.exit(0)
        except SystemExit:
            os._exit(0)

问题根因与修复方案 #

根因说明 #

1. idNotIn传参格式错误：Kaltura接口要求idNotIn字段传入逗号分隔的ID字符串，你直接传入Python列表对象，接口无法识别，过滤逻辑完全不生效。
2. 偏移分页限制：Kaltura公开API默认单次查询的最大偏移量为10000，当pageIndex * pageSize超过10000时会直接返回QUERY_EXCEEDED_MAX_MATCHES_ALLOWED错误，你截断总数为9970只能拉取前10000条，无法拿到全量数据。
3. session权限配置不一致：KalturaConfiguration中硬编码了partner id为2723521，但session.start方法中传入的是变量KALTURA_PARTNER_ID，两者不一致导致disableentitlement权限未生效。
4. 分页参数赋值错误：首次请求分页时错误将方法setPageIndex当做属性赋值，导致首次请求的页索引未生效。

修复步骤 #

1. 修正idNotIn传参格式，将列表转为逗号分隔字符串：

# 原代码
if lastEntryIds != "":
    filter.idNotIn = lastEntryIds
# 替换为
if lastEntryIds:
    filter.idNotIn = ','.join(lastEntryIds)

2. 替换偏移分页为游标分页，避免超过最大偏移限制：使用createdAt作为分页游标，每次查询过滤创建时间小于等于上次拉取的最小创建时间，搭配idNotIn排除同时间已拉取的条目，无需限制总数：

filter = KalturaPluginsCore.KalturaBaseEntryFilter()
# 新增时间游标过滤
if lastEntryCreatedAt > 0:
    filter.createdAtLessThanOrEqual = lastEntryCreatedAt
if lastEntryIds:
    filter.idNotIn = ','.join(lastEntryIds)
filter.orderBy = KalturaBaseEntryOrderBy.CREATED_AT_DESC

3. 统一session初始化的partner id，保证权限生效：

# 原代码
config = KalturaConfiguration(2723521)
# 替换为
config = KalturaConfiguration(KALTURA_PARTNER_ID)

4. 修正分页参数赋值错误：

# 原代码
pager.setPageIndex = pageIndex
# 替换为
pager.pageIndex = pageIndex

5. 移除不必要的总数截断逻辑，当单次拉取返回的条目数小于页大小时直接终止循环，无需依赖总数判断。

内容的提问来源于stack exchange，提问作者Megalitik