Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
338 views
in Technique[技术] by (71.8m points)

scala - update variables using map function on spark

Here is my code:

val dataRDD = sc.textFile(args(0)).map(line => line.split(" ")).map(x => Array(x(0).toInt, x(1).toInt, x(2).toInt))
var arr = new Array[Int](3)
printArr(arr)
dataRDD.map(x => {arr=x})
printArr(arr)

This code is not working properly. How can i make it work successfully?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

Okay, so operations on RDDs are performed in parallel by different workers (usually on different machines in the cluster) and therefore you cannot pass in this type of "global" object arr to be updated. You see, each worker will receive their own copy of arr which they will update, but the driver will never know.

I'm guessing that what you want to do here is to collect all the arrays from the RDD, which you can do with a simple collect action:

val dataRDD = sc.textFile(args(0)).map(line => line.split(" ")).map(x => Array(x(0).toInt, x(1).toInt, x(2).toInt))
val arr = dataRDD.collect()

Where arr has type Array[Array[Int]]. You can then run through arr with normal array operations.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...