Solr-Spark索引失败：访问集合URL出错且Java代码抛空指针异常

Hey there, let's break down the java.lang.NullPointerException you're facing when indexing documents with Spark and Solr. Based on the details you shared (ZooKeeper on 2181, 2-shard test collection, and your partial SparkRead class code), here are the most likely culprits and actionable checks:

1. Issues in Your SparkRead Class #

From the partial code you provided, there are a few red flags that could trigger NPE:

• Incomplete constructor assignment: Your constructor line cuts off at this.nbLinesToSki... — if you forgot to fully assign this.nbLinesToSkip = nbLinesToSkip, or if the input nbLinesToSkip parameter is null, this field will stay null. Any subsequent logic that uses nbLinesToSkip (like line-skipping checks) will throw an NPE.
• Uninitialized fields: Fields like fileName, sizeToRead are initialized to null by default. If your code later tries to call methods on these (e.g., fileName.length()) without first assigning a valid value, that's a guaranteed NPE.
• Serialization gaps: Since SparkRead implements Serializable, make sure all non-transient fields are serializable and properly initialized before being sent to Spark executors. If a field is set only on the driver but not propagated to executors, it'll be null when the executor tries to use it.

2. Solr-Spark Integration Specific Issues #

Even if your class code is solid, there are common NPE triggers in the Spark-Solr pipeline:

• Missing/invalid Solr configuration: If you haven't properly set the zkHost parameter in your Spark-Solr connection (e.g., zkHost="localhost:2181"), the Solr client will fail to connect to ZooKeeper and return a null instance. Any subsequent calls to interact with Solr will throw an NPE. Double-check that you've also specified the correct collection name (collection="test").
• Broken shard metadata: Your test collection has two shards — if one of the shards is offline or its metadata in ZooKeeper is corrupted (e.g., null shard URL), Spark's Solr client will hit an NPE when trying to route documents to shards. Verify your shards are healthy via the Solr Admin UI (check the Shards tab for the test collection).
• Null documents in RDD: If your Spark RDD contains null document objects, the Solr client will throw an NPE when trying to index them. Add a filter to your RDD to exclude nulls:

  rdd.filter(doc -> doc != null)
  

3. Quick Troubleshooting Steps #

To narrow down the exact cause fast:

• Get the full stack trace of the NPE — it will show exactly which line of code is triggering the null reference. This is the most critical piece of info!
• Share the complete SparkRead class code, especially the methods that process data and interact with Solr.
• Add debug logs to print values of fields like nbLinesToSkip, fileName, and individual document objects before they're passed to Solr. This will reveal if any key values are unexpectedly null.
• Validate your Solr cluster health first: confirm ZooKeeper is reachable, all shards are active, and the test collection is working via direct Solr API calls.

内容的提问来源于stack exchange，提问作者Dilak