如何在Spark中获取RDD的长度？附代码及报错信息求助

1. How to get the length (number of elements) of an RDD in Spark? #

To get the total number of elements in an RDD (which is what we typically refer to as "RDD length"), you can use the count() method. It's an action operation that triggers a job to tally up all elements across the RDD's partitions.

For your example RDD:

var mark = sc.parallelize(List(1,2,3,4,5,6))
val rddTotalElements = mark.count()
println(rddTotalElements) // Output: 6

2. Fixing the error in your code & getting the RDD length #

Let's break down why your code throws that error first:

:27: error: value length is not a member of Int mark.map(l => l.length).collect

The problem here is that every element l in your RDD is an integer (Int), and Scala's Int type doesn't have a length method. You likely mixed this up with types like String, which do have a length property.

The fix depends on what you're actually trying to do: #

• If your goal is to get the total number of elements in the RDD (the true "length" of the RDD), ditch the map step entirely and use count() as shown in the first section.
• If you accidentally tried to get the digit count of each integer (e.g., how many characters each number has when written out), convert each Int to a String first:

val elementDigitCounts = mark.map(_.toString.length).collect()
println(elementDigitCounts.mkString(", ")) // Output: 1, 1, 1, 1, 1, 1

Just keep in mind this gives you the length of each individual element, not the total number of elements in the RDD.

内容的提问来源于stack exchange，提问作者Abhishek