apache spark - Is there a way to slice dataframe based on index in pyspark?

Question

In python or R, there are ways to slice DataFrame using index.

For example, in pandas:

df.iloc[5:10,:]

Is there a similar way in pyspark to slice data based on location of rows?

Answer 1

Short Answer

If you already have an index column (suppose it was called 'id') you can filter using pyspark.sql.Column.between:

from pyspark.sql.functions import col
df.where(col("id").between(5, 10))

If you don't already have an index column, you can add one yourself and then use the code above. You should have some ordering built in to your data based on some other columns (orderBy("someColumn")).

---

Full Explanation

No it is not easily possible to slice a Spark DataFrame by index, unless the index is already present as a column.

Spark DataFrames are inherently unordered and do not support random access. (There is no concept of a built-in index as there is in pandas). Each row is treated as an independent collection of structured data, and that is what allows for distributed parallel processing. Thus, any executor can take any chunk of the data and process it without regard for the order of the rows.

Now obviously it is possible to perform operations that do involve ordering (lead, lag, etc), but these will be slower because it requires spark to shuffle data between the executors. (The shuffling of data is typically one of the slowest components of a spark job.)

Related/Futher Reading

• PySpark DataFrames - way to enumerate without converting to Pandas?
• PySpark - get row number for each row in a group
• how to add Row id in pySpark dataframes