如何处理Spark独立集群中的共享文件权限问题？

Alright, based on your description, here's a clear breakdown of how your cluster is configured and behaves:

• Standalone Deployment: You’re using Spark’s native Standalone cluster manager—this is a great fit for your on-demand use case because it’s lightweight, avoids the heavy resource footprint of HDFS, and doesn’t require additional cluster management tools like YARN or Kubernetes.
• Shared Network File System as Storage Layer: Instead of HDFS, all master and worker nodes connect to a single shared network file system. This means your Spark apps can directly read and write data from this shared storage without needing to replicate data across cluster nodes, which saves both storage space and setup overhead.
• On-Demand Cluster Lifecycle: The cluster is only started when you need to run large-scale data processing tasks. This is a smart approach for resource efficiency—no wasted resources on idle nodes when there’s no work to do.
• Runtime Permissions: Spark services run under the spark user and spark group. Since you’re part of the spark group, you have the necessary access to initiate Spark sessions and manage applications on the cluster.
• Application File Access: When you start a Spark session, the cluster provisions a dedicated Spark application. This application has full access to any files stored on the shared network file system (as long as the spark user/group has the appropriate file-level permissions), so you can load data directly from paths like /shared-data/your-dataset.csv without extra configuration steps.

内容的提问来源于stack exchange，提问作者wasperen