Hey there, let's break down why you're hitting this NoSuchMethodError even though the BigQueryUtils class exists in your jar. This error almost always boils down to version conflicts or environment-specific dependency clashes—here's how to fix it:

1. Check for Dependency Version Mismatches #

The Spotify Spark BigQuery connector relies on a specific version of the Google Cloud BigQuery Hadoop client. If your pom.xml pulls in a different version of bigquery-connector (or related GCP libraries), the method signatures might not align—even if the class exists.

Action Steps: #

• Run mvn dependency:tree in your project directory to generate a dependency tree. Look for multiple entries of com.google.cloud.bigdataoss:bigquery-connector (or similar GCP artifacts) with different version numbers.
• Match your BigQuery dependencies to the version required by your Spotify connector. For example, if you're using com.spotify:spark-bigquery_2.11:0.2.10, it expects bigquery-connector:hadoop2-1.1.5—adjust your pom.xml to explicitly declare this version, or exclude conflicting versions:

  <dependency>
      <groupId>com.spotify</groupId>
      <artifactId>spark-bigquery_2.11</artifactId>
      <version>0.2.10</version>
      <exclusions>
          <exclusion>
              <groupId>com.google.cloud.bigdataoss</groupId>
              <artifactId>bigquery-connector</artifactId>
          </exclusion>
      </exclusions>
  </dependency>
  <dependency>
      <groupId>com.google.cloud.bigdataoss</groupId>
      <artifactId>bigquery-connector</artifactId>
      <version>hadoop2-1.1.5</version>
  </dependency>
  

2. Override Qubole's Preloaded Dependencies #

Qubole often pre-installs common Hadoop/GCP libraries on its Spark clusters. If these preloaded versions clash with your packaged dependencies, your code will pick up the wrong class version.

Action Steps: #

• Add these Spark configuration properties in your Qubole job settings to prioritize your jar's dependencies over the platform's:

  spark.driver.userClassPathFirst=true
  spark.executor.userClassPathFirst=true
  

• Alternatively, use Qubole's spark.jars.packages parameter to pull in the exact dependencies you need, instead of packaging them yourself. For example:

  spark.jars.packages=com.spotify:spark-bigquery_2.11:0.2.10,com.google.cloud.bigdataoss:bigquery-connector:hadoop2-1.1.5
  

  This lets Qubole handle dependency resolution and avoids conflicts with preloaded jars.

3. Verify Method Signature Compatibility #

It’s possible the waitForJobCompletion method was modified between BigQuery client versions (e.g., added/removed parameters, changed return types).

Action Steps: #

• Check the Javadoc for the BigQueryUtils class in your dependency version. Compare the method signature of waitForJobCompletion to what the Spotify connector expects.
• If you’re using a newer BigQuery client version, roll back to the version specified in the Spotify connector’s pom.xml (you can find this by looking up the Spotify artifact on Maven Central).

4. Use Shade Plugin to Avoid Class Conflicts #

If you’re packaging your jar with mvn assembly:single, you might be including conflicting classes. The Maven Shade plugin can rewrite package names for conflicting libraries to isolate them from Qubole’s environment.

Example Shade Plugin Configuration: #

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>com.google.cloud.hadoop.io.bigquery</pattern>
                                <shadedPattern>your.custom.prefix.google.cloud.hadoop.io.bigquery</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Final Notes #

If you share the full content of your pom.xml, we can pinpoint the exact conflict—but these steps should cover the most common causes of this error on Qubole.

内容的提问来源于stack exchange，提问作者edocx