scala - update variables using map function on spark

Question

Here is my code:

val dataRDD = sc.textFile(args(0)).map(line => line.split(" ")).map(x => Array(x(0).toInt, x(1).toInt, x(2).toInt))
var arr = new Array[Int](3)
printArr(arr)
dataRDD.map(x => {arr=x})
printArr(arr)

This code is not working properly. How can i make it work successfully?

Answer 1

Okay, so operations on RDDs are performed in parallel by different workers (usually on different machines in the cluster) and therefore you cannot pass in this type of "global" object arr to be updated. You see, each worker will receive their own copy of arr which they will update, but the driver will never know.

I'm guessing that what you want to do here is to collect all the arrays from the RDD, which you can do with a simple collect action:

val dataRDD = sc.textFile(args(0)).map(line => line.split(" ")).map(x => Array(x(0).toInt, x(1).toInt, x(2).toInt))
val arr = dataRDD.collect()

Where arr has type Array[Array[Int]]. You can then run through arr with normal array operations.