scala - Operate on neighbor elements in RDD in Spark

Question

As I have a collection:

List(1, 3,-1, 0, 2, -4, 6)

It's easy to make it sorted as:

List(-4, -1, 0, 1, 2, 3, 6)

Then I can construct a new collection by compute 6 - 3, 3 - 2, 2 - 1, 1 - 0, and so on like this:

for(i <- 0 to list.length -2) yield {
    list(i + 1) - list(i)
}

and get a vector:

Vector(3, 1, 1, 1, 1, 3)

That is, I want to make the next element minus the current element.

But how to implement this in RDD on Spark?

I know for the collection:

List(-4, -1, 0, 1, 2, 3, 6)

There will be some partitions of the collection, each partition is ordered, can I do the similar operation on each partition and collect results on each partition together?

Answer 1

The most efficient solution is to use sliding method:

import org.apache.spark.mllib.rdd.RDDFunctions._

val rdd = sc.parallelize(Seq(1, 3,-1, 0, 2, -4, 6))
  .sortBy(identity)
  .sliding(2)
  .map{case Array(x, y) => y - x}