适配Web服务的无单点故障高性能类POSIX分布式文件系统求荐

Hey there, let's break down your requirements and find the right fit for your read-heavy, small-file workload. You've already ruled out several options with good reason, so let's focus on solutions that check all your boxes: write persistence, great read performance, no single point of failure, and (mostly) POSIX compatibility with your "eventual consistency" tolerance.

1. DRBD + High-Availability NFS Cluster (Tweak Your Initial Idea) #

Your thought of combining DRBD with NFS was on the right track—you just need to eliminate the single NFS server bottleneck. Here's how to make it fully HA:

• Sync storage volumes across multiple nodes using DRBD, then set up a Pacemaker/Corosync cluster for NFS. For your read-heavy workload, an active-active NFS setup will let you scale read throughput across nodes, while active-passive works if you want simpler failover logic.
• Why this fits your needs:
  • Full POSIX compliance, so your app servers don't need any code changes—totally transparent to your existing setup.
  • Read performance is nearly on par with local storage, since NFS handles small-file reads efficiently, and active-active lets you spread the read load across multiple nodes.
  • You get guaranteed write persistence by configuring DRBD to use Protocol C (syncs writes to all replicas before acknowledging success), which aligns with your requirement of waiting for storage confirmation.
  • No single point of failure: if one NFS node goes down, Pacemaker automatically fails over the service to another node with the synced DRBD volume.
• Key considerations:
  • Capacity scaling requires adjusting DRBD volumes (you can pair it with LVM for online resizing), but since you're read-heavy, capacity pressure might not be your biggest concern.
  • Since it's block-level replication, all metadata (even the stuff you don't care about like modification time) is synced—which is fine, because you only need filenames and content anyway.

2. BeeGFS (Formerly FraunhoferFS) #

If you want a pure distributed filesystem without the complexity of Ceph, BeeGFS is a solid, reliable pick:

• It's purpose-built for high-performance workloads, especially optimized for small-file reads. Its architecture separates metadata nodes (which you can cluster for redundancy) and storage nodes, so scaling out won't cause the performance drops you saw with GlusterFS.
• Configuration and management are way simpler than Ceph, with comprehensive official docs and an active community. No sketchy code obfuscation here, unlike MooseFS.
• Supports POSIX compliance, and you can configure write persistence by setting up redundant storage pools via beegfs-ctl.
• For your read-heavy use case, adding more storage nodes directly scales read throughput—no unexpected performance hits when expanding your cluster.
• Standout perks:
  • Stable project with commercial backing, so you don't have to worry about a single maintainer abandoning it (unlike SeaweedFS).
  • Built-in small-file optimizations that align perfectly with your workload.
  • No single point of failure: metadata nodes run in a cluster, storage nodes are redundant, and clients automatically fail over if a node goes offline.

3. Optimized GlusterFS (If You Don't Want to Switch Systems) #

Since you're already using GlusterFS, it's worth trying targeted tweaks before jumping ship—most performance issues stem from default configurations, not the system itself:

• For small-file reads, crank up the client cache with gluster volume set <volname> performance.cache-size 4GB (adjust based on your server's available RAM) and enable performance.io-cache on.
• Ditch the default replica 3 setup. Use a distribute+replica 2 combo instead, or if you're okay with eventual consistency, try erasure coding (ec) mode—though replica mode is better for raw read performance.
• Disable unnecessary POSIX features you don't care about: turn off performance.stat-prefetch off to cut down on metadata overhead, since you don't need modification time or other non-critical metadata details.
• When adding new nodes, run gluster volume rebalance <volname> start to evenly distribute data—unbalanced data is a common cause of performance drops after cluster expansion.

• Go with DRBD + HA NFS Cluster if you want the simplest architecture, best read performance, and full POSIX compliance. It's low-effort to set up and maintain, and checks all your boxes.
• Pick BeeGFS if you need a pure distributed system that scales easily for both capacity and read throughput, with stable long-term support.
• If you're attached to GlusterFS, try the optimizations first—you might be surprised how much better it performs with a few tweaks.

内容的提问来源于stack exchange，提问作者Mikko Rantalainen