maximum number of columns we can have in dataframe spark scala

Question

I like to know the maximum number of columns I can have in the dataframe,Is there any limitations in maintaining number of columns in dataframes. Thanks.

Answer 1

Sparing you the details, the answer is Yes, there is a limit for the size the number of columns in Apache Spark.

Theoretically speaking, this limit depends on the platform and the size of element in each column.

Don't forget that Java is limited by the size of the JVM and an executor is also limited by that size - Java largest object size in Heap.

I would go back an refer to this Why does Spark RDD partition has 2GB limit for HDFS? which refers to the limitation with HDFS on block/partition size.

So there is actually lots of restriction to take into account.

This means that you can easily find a hard limit (Int.MaxValue par ex.) but what is more important Spark scales well only long and relatively thin data. (like stated by pault).

Finally, you need to remember that fundamentally you cannot split a single record between executors/partitions. And there is a number of practical limitations (GC, disk IO) which make very wide data impractical. Not to mention some known bugs.

Note : I mention @pault and @RameshMaharjan as this answer is actually the fruit of the discussion we had. (And ofc @zero323 for his comment from the other answer).