关于Solr默认是否生成.cfs段文件及性能与复制的疑问

Hi there, let's break down your questions step by step:

1. Clarifying the <usecompoundfile> default value confusion #

You assumed the Linux Solr 5.2.1 instance uses false by default, but that's where the mismatch comes in. Solr changed the default value of this configuration between versions:

• Solr 4.x (like your Windows 4.2.1 setup): The default for <usecompoundfile> is false, so your explicit setting aligns with the default behavior, hence no .cfs files.
• Solr 5.x (your Linux 5.2.1 setup): The default was switched to true. Since you didn't override it, Solr generates compound file segments (.cfs) automatically.

2. Performance impact of retaining .cfs files #

The pros and cons depend on your deployment scale and workload:

• Advantages: Compound files bundle multiple segment sub-files (like .fdt, .fdx, .tii) into a single .cfs file. This reduces the number of open file handles your system needs to manage, which is helpful for environments with limited file descriptor quotas or memory constraints. It also cuts down on file system metadata overhead.
• Disadvantages: For large indexes or high-concurrency query workloads, .cfs files introduce extra CPU overhead—Solr has to extract the needed sub-files from the compound archive during queries. This hits random-access query performance the hardest. If your cluster handles heavy query traffic or large indexes, disabling compound files will typically yield better query speeds.

3. Impact on Solr replication (and why full copies happen frequently) #

.cfs files are very likely contributing to your frequent full replication events:

• Solr's incremental replication relies on tracking changes to index segments via checkpoints stored in index.properties. When using compound files, any update or merge operation for a segment creates an entirely new .cfs file (since it encapsulates all segment data). In some cases, the from-node may fail to recognize this as an incremental change and instead flags the segment as a major update, triggering a full replication instead of syncing only the changed parts.
• In contrast, with non-compound files, segments are split into smaller individual files. Incremental replication only needs to sync the specific small files that changed, which is far less likely to trigger a full copy.

Recommended fixes #

• Disable compound files: Add <usecompoundfile>false</usecompoundfile> inside the <indexConfig> section of your Linux Solr instance's solrconfig.xml. Restart Solr, and future segment operations will generate non-compound files.
• Convert existing indexes: If you want to eliminate existing .cfs files, run an optimize operation on your index (do this during low-traffic hours—it’s resource-intensive). This will merge existing segments into non-compound files.
• Validate replication config: Double-check your replicationHandler settings on both master and slave nodes. Ensure pollInterval is set appropriately, and that the master’s transaction log (tlog) is configured correctly to support reliable incremental replication.

内容的提问来源于stack exchange，提问作者Adarsh H D Dev