Kubernetes集群出现Calico IPAMBlock更新错误及Oracle连接异常问题

1 Calico IPAM冲突错误原因 #

你日志中出现的Operation cannot be fulfilled on ipamblocks.crd.projectcalico.org "10-0-5-64-26": the object has been modified是Calico网络插件的IP地址管理（IPAM）模块的乐观锁冲突：

• 并发批量创建Pod时，多个集群节点同时尝试修改同一段IP地址段（IPBlock）的分配记录
• K8s自定义资源（CRD）自带resourceVersion版本校验机制，某一个节点先提交修改更新了版本号后，其他节点基于旧版本提交的修改请求会直接被拒绝
• 该错误会直接导致Pod的网络配置下发延迟、失败，出现假性连通异常：你可以通过kubelet链路exec进入Pod操作，但Pod的业务网络链路存在间歇性中断、丢包的问题

2 ORA-03113/03114 错误与关联逻辑 #

ORA-03113的官方定义是「通信通道的文件结束」，本质是Oracle客户端和服务端之间的TCP连接被意外中断，客户端读取到EOF报文，和前面的Calico网络异常直接相关：

• 业务Pod和Oracle建立的长连接，因为Calico网络抖动、规则重置被切断
• 业务代码未配置数据库重连逻辑，连接中断后无法写入任务日志到数据库，任务收尾逻辑卡住，最终出现Pod任务实际执行成功但滞留在集群中不退出的问题

• 优化Calico IPAM配置：下调Pod批量创建的并发数，避免高并发下多个节点争抢同一个IPBlock的分配权限；升级Calico到最新稳定版，旧版本IPAM分配逻辑的并发性能缺陷会大幅提升冲突概率；调整IPBlock的分配块大小，给每个节点分配更大的专属IP段，减少跨节点的IP资源争抢
• 业务逻辑适配：给Oracle、Informatica的客户端配置连接池和自动重连、重试机制，网络短暂抖动时可自动恢复连接，避免单次断连导致整个任务流程卡住；给Pod配置本地日志落盘策略，异常时先将日志写入本地存储，避免日志丢失
• 网络规则校验：检查集群到Oracle/Informatica服务之间的网关、防火墙、安全组的会话超时配置，若空闲TCP连接的超时时间短于业务长连接的空闲时间，会被主动切断，可通过配置TCP保活报文规避该问题

INFO process_step:>1<; message:>Execute: Run successful< 

2021-09-30 18:50:37.898 [ERROR][23511] customresource.go 136: Error updating resource Key=IPAMBlock(10-0-5-64-26) Name="10-0-5-64-26" Resource="IPAMBlocks" Value=&v3.IPAMBlock{TypeMeta:v1.TypeMeta{Kind:"IPAMBlock", APIVersion:"crd.projectcalico.org/v1"}, ObjectMeta:v1.ObjectMeta{Name:"10-0-5-64-26", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"425239779", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v3.IPAMBlockSpec{CIDR:"10.0.5.64/26", Affinity:(*string)(0xc000476000), StrictAffinity:false, Allocations:[]*int{(*int)(0xc00068d388), (*int)(0xc00068d450), (*int)(nil), (*int)(0xc00068d490), (*int)(0xc00068d398), (*int)(0xc00068d3f0), (*int)(0xc00068d498), (*int)(0xc00068d390), (*int)(0xc00068d420), (*int)(0xc00068d4a0), (*int)(0xc00068d3d0), (*int)(nil), (*int)(0xc00068d308), (*int)(0xc00068d3b0), (*int)(0xc00068d310), (*int)(0xc00068d320), (*int)(nil), (*int)(0xc00068d4b8), (*int)(nil), (*int)(nil), (*int)(0xc00068d460), (*int)(0xc00068d4a8), (*int)(0xc00068d458), (*int)(0xc00068d3c8), (*int)(0xc00068d440), (*int)(nil), (*int)(0xc00068d428), (*int)(0xc00068d3b8), (*int)(0xc00068d470), (*int)(0xc00068d408), (*int)(0xc00068d418), (*int)(0xc00068d448), (*int)(0xc00068d438), (*int)(0xc00068d4b0), (*int)(0xc00068d3a8), (*int)(0xc00068d318), (*int)(0xc00068d430), (*int)(0xc00068d3d8), (*int)(0xc00068d410), (*int)(0xc00068d478), (*int)(0xc00068d3e0), (*int)(0xc00068d3c0), (*int)(0xc00068d358), (*int)(0xc00068d330), (*int)(0xc00068d340), (*int)(0xc00068d3f8), (*int)(0xc00068d328), (*int)(0xc00068d400), (*int)(0xc00068d338), (*int)(0xc00068d480), (*int)(0xc00068d350), (*int)(0xc00068d488), (*int)(0xc00068d468), (*int)(0xc00068d348), (*int)(0xc00068d360), (*int)(nil), (*int)(0xc00068d368), (*int)(0xc00068d3e8), (*int)(0xc00068d370), (*int)(0xc00068d378), (*int)(0xc00068d380), (*int)(0xc00068d3a0), (*int)
(nil), (*int)(nil)}, Unallocated:[]int{2, 25, 11, 18, 19, 55, 62, 16, 63
****Bunch of node IP's**************** then
error=Operation cannot be fulfilled on ipamblocks.crd.projectcalico.org "10-0-5-64-26": the object has been modified; please apply your changes to the latest version and try again 

DBD::Oracle::db do failed: ORA-03113: end-of-file on communication channel

内容的提问来源于stack exchange，提问作者ROXOR7