Recent questions tagged Scala

0 votes

666 views

1 answer

scala - Why does Spark fail with java.lang.OutOfMemoryError: GC overhead limit exceeded?

I'm trying to implement a Hadoop Map/Reduce job that worked fine before in Spark. The Spark app definition is ... very long. Thanks in advance. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

513 views

1 answer

scala - Sets, Functors and Eq confusion

A discussion came up at work recently about Sets, which in Scala support the zip method and how this can ... the Set instance in particular? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

526 views

1 answer

scala - Spark final task takes 100x times longer than first 199, how to improve

I am seeing some performance issues while running queries using dataframes. I have seen in my research, that long ... I have empty executors? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

573 views

1 answer

scala - Play 2.x : Reactive file upload with Iteratees

I will start with the question: How to use Scala API's Iteratee to upload a file to the cloud storage (Azure Blob ... for me to get the idea. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

523 views

1 answer

scala - How to create correct data frame for classification in Spark ML

I am trying to run random forest classification by using Spark ML api but I am having issues with creating right data ... ) } } Thanks for help! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

535 views

1 answer

scala - Testing an assertion that something must not compile

The problem When I'm working with libraries that support type-level programming, I often find myself writing ... 'll settle for anything. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

808 views

1 answer

scala - reduceByKey: How does it work internally?

I am new to Spark and Scala. I was confused about the way reduceByKey function works in Spark. Suppose we have the ... + b? how are they filled? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

634 views

1 answer

scala - Spark : how to run spark file from spark shell

I am using CDH 5.2. I am able to use spark-shell to run the commands. How can I run the file(file.spark ... 5.2 without sbt? Thanks in advance See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

697 views

1 answer

scala - Implicit conversion vs. type class

In Scala, we can use at least two methods to retrofit existing or new types. Suppose we want to express that ... be nice, even without example. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

709 views

1 answer

scala - ScalaTest in sbt: is there a way to run a single test without tags?

I know that a single test can be ran by running, in sbt, testOnly *class -- -n Tag Is there a way of ... a single test without it having a tag? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

439 views

1 answer

scala - using prepared statement multiple times, giving a warning of Cassandra Querying Reducing Performance

I am getting data from somewhere and inserting it into cassandra daily basis then I need to retrieve the data from cassandra ... ............ See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

581 views

1 answer

scala - How to access sub-entities in JSON file?

I have a json file look like this: { "employeeDetails":{ "name": "xxxx", "num":"415" }, "work":[ ... [_corrupt_record#0] json I am new to Spark. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

651 views

1 answer

scala - Futures / Success race

I'm learning futures, and I'm trying to create a method that, take two futures as parameter (f and g) and ... I know which one completed first ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

654 views

1 answer

scala - dynamically parse json in flink map

I'm using flink to dynamically analyze json type data,to keyby and sum with the given column,in my mapFunction,I ... to case class or tuple? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

796 views

1 answer

scala - Get a TypeTag from a Type?

I can get a Type from a TypeTag[A] by using the tpe method. But can I also recover the type-tag from a ... have the type and dropped the tag. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

828 views

1 answer

scala - NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities while reading s3 Data with spark

I would like to run a simple spark job on my local dev machine (through Intellij) reading data from Amazon ... Thanks in advance for your help See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

560 views

1 answer

scala - Broadcast not happening while joining dataframes in Spark 1.6

Below is the sample code that I am running. when this spark job runs, Dataframe joins are happening using ... datasize of countriesDf is 5mb. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

653 views

1 answer

scala - Scalaz Bind[Seq] typeclass

I'm currently porting some code from traditional Scala to Scalaz style. It's fairly common through most of my ... my own Bind[Seq] typeclass? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

705 views

1 answer

scala - Futures do not run before program termination

I was trying to reproduce the example on new Scala 2.10 futures feature. The code I've used is: import scala. ... compiler is 2.10.0-20120507. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

695 views

1 answer

scala - Spark column string replace when present in other column (row)

I would like to remove strings from col1 that are present in col2: val df = spark.createDataFrame(Seq( ("Hi I ... so that res looks as below: See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

619 views

1 answer

scala - How to create a Dataset of Maps?

I'm using Spark 2.2 and am running into troubles when attempting to call spark.createDataset on a Seq of ... not covered by current encoders? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

736 views

1 answer

scala - Concrete classes with abstract type members

Given the following traits and class. Why does this compile? Can this be actually used for something? trait ... motivation for it: #1753. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

520 views

1 answer

scala - Separating application logs in Logback from Spark Logs in log4j

I have a Scala Maven project using that uses Spark, and I am trying implement logging using Logback. I am ... still be shown at STDOUT. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

634 views

1 answer

scala - What is a Singleton Type exactly?

What is a singleton type? what are the applications, the implications ? Examples are more than welcome and layman terms are even more welcome ! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

582 views

1 answer

scala - "Parameter type in structural refinement may not refer to an abstract type defined outside that refinement"

When I compile: object Test extends App { implicit def pimp[V](xs: Seq[V]) = new { def dummy(x: V) ... refinement" doesn't really answer this.) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to slice and sum elements of array column?

I would like to sum (or perform other aggregate functions too) on the array column using SparkSQL. I have a table ... apart from the UDF one ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

758 views

1 answer

scala - Is there a simple example of how to generate verilog from Chisel3 module?

I'm looking for a simple howto to convert a simple Chisel3 module in Verilog. I take Gcd source code given ... for converting it in Verilog. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

673 views

1 answer

scala - Multiple parameter closure argument type not inferred

I have a piece of code that I can't get to behave in the way I'd like. I have a class defined in the following way ( ... a, b : Int) => a + b) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags