Recent questions tagged Scala

0 votes

740 views

1 answer

scala - Function returns an empty List in Spark

Below is code for getting list of file Names in a zipped file def getListOfFilesInRepo(zipFileRDD : RDD[(String, ... 889 and instead getting 0? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

708 views

1 answer

scala - Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]

Is it possible in Spark to implement '.combinations' function from scala collections? /** Iterates over combinations. ... in RDD are unique. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

651 views

1 answer

scala - Why are singleton objects more object-oriented?

In Programming in Scala: A Comprehensive Step-by-Step Guide, the author said: One way in which Scala is ... members, but singleton objects? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

687 views

1 answer

scala - Spark DataFrame: does groupBy after orderBy maintain that order?

I have a Spark 2.0 dataframe example with the following structure: id, hour, count id1, 0, 12 id1, 1, 55 .. ... not, how can I work around it ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

777 views

1 answer

scala - How to save a spark DataFrame as csv on disk?

For example, the result of this: df.filter("project = 'en'").select("title","count").groupBy("title").sum( ... DataFrame as a csv file on disk ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

749 views

1 answer

scala - Play 2.4: Form: could not find implicit value for parameter messages: play.api.i18n.Messages

I am new to Play framework and tried to mimic the helloworld sample in my local machine but I encountered an ... Idea 14 via activator template. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

672 views

1 answer

scala - How to turn json to case class when case class has only one field

In play 2.1 reads are used to marshall Json to objects. But how can I do this when the case class has only ... read(Reads.list[String]) )(Data) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

613 views

1 answer

scala - Defaults for missing properties in play 2 JSON formats

I have an equivalent of the following model in play scala : case class Foo(id:Int,value:String) object Foo{ ... and accept any better option) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

684 views

1 answer

scala - Spark: Add column to dataframe conditionally

I am trying to take my input data: A B C -------------- 4 blah 2 2 3 56 foo 3 And add a column to ... but I can't get that to do what I want. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

605 views

1 answer

scala - How to create a Source that can receive elements later via a method call?

I would like to create a Source and later push elements on it, like in: val src = ... // create the ... the recommended way to do this? Thanks! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

728 views

1 answer

scala - Why are `private val` and `private final val` different?

I used to think that private val and private final val are same, until I saw section 4.1 in Scala ... ? Is there any underlying reason? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

523 views

1 answer

scala - Memory barriers and coding style over a Java VM

Suppose I have a static complex object that gets periodically updated by a pool of threads, and read more or less ... for a Scala version :) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

555 views

1 answer

scala - How to write spark streaming DF to Kafka topic

I am using Spark Streaming to process data between two Kafka queues but I can not seem to find a good way ... to Kafka from Spark Streaming? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

544 views

1 answer

scala - How do you do dependency injection with the Cake pattern without hardcoding?

I just read and enjoyed the Cake pattern article. However, to my mind, one of the key reasons to use ... involve mixing traits in statically. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

834 views

1 answer

scala - Stream vs Views vs Iterators

What are the differences among Streams, Views (SeqView), and Iterators in scala? This is my understanding: They are ... again) or views? Thanks. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

674 views

1 answer

scala - How to use COGROUP for large datasets

I have two rdd's namely val tab_a: RDD[(String, String)] and val tab_b: RDD[(String, String)] ... spark.yarn.executor.memoryOverhead. Thank you See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - How to get ID of a map task in Spark?

Is there a way to get ID of a map task in Spark? For example if each map task calls a user defined ... from whithin that user defined function? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

671 views

1 answer

scala - By-name parameter vs anonymous function

What is still unclear for is what's the advantage by-name parameters over anonymous functions in terms of lazy evaluation ... ()=>, and Unit=> See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

780 views

1 answer

scala - Type mismatch on abstract type used in pattern matching

This code compiles with an error: def f1[T](e: T): T = e match { case i:Int => i case b: ... first example and succeeded on the second one? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

729 views

1 answer

scala - Filling gaps in timeseries Spark

I have a problem dealing with time-series data. Due power failures some timestamps are missing in the ... + completeMeasurementsDS.count()) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

674 views

1 answer

scala - If an Int can't be null, what does null.asInstanceOf[Int] mean?

As a test, I wrote this code: object Ambig extends App { def f( x:Int ) { println("Int" ) } def f( x:String ... 't be null... What am I missing? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

543 views

1 answer

scala - java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags

I am a beginner in Spark streaming and Scala. For a project requirement I was trying to run TwitterPopularTags ... in advance. Regards, vpv See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

507 views

1 answer

scala - Weird behavior trying to convert case classes to heterogeneous lists recursively with Shapeless

I stayed up way too late last night trying to figure out this Shapeless issue and I'm afraid it's going to eat ... go read a book or something. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

747 views

1 answer

scala - withFilter instead of filter

Is it always more performant to use withFilter instead of filter, when afterwards applying functions like map, ... forall/exists as well) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

858 views

1 answer

scala - Mixing in a trait dynamically

Having a trait trait Persisted { def id: Long } how do I implement a method that accepts an instance of any ... id: Long): T with Persisted See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to find spark RDD/Dataframe size?

I know how to find the file size in scala.But how to find a RDD/dataframe size in spark? Scala: object Main ... . How to find the RDD size? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

765 views

1 answer

scala - Change nullable property of column in spark dataframe

I'm manually creating a dataframe for some testing. The code to create it is: case class input(id:Long, ... dataframe after it's been created? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

480 views

1 answer

scala - How to add "provided" dependencies back to run/test tasks' classpath?

Here's an example build.sbt: import AssemblyKeys._ assemblySettings buildInfoSettings net.virtualvoid.sbt.graph. ... the assembly artifact. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags