replace - What is the best PySpark practice to subtract two string columns within a single spark dataframe?

Question

Let's say I have a spark dataframe as below:

data	A	Expected_column= data - A
https://example1.org/path/to/file?param=42#fragment	param=42#fragment	https://example1.org/path/to/file?
https://example2.org/path/to/file	NaN	https://example2.org/path/to/file

Answer 1

the function you're looking for is called replace:

from pyspark.sql import functions as F

sdf.withColumn("data - A", F.expr("replace(data, coalesce(A, ''), '')")).show(
    truncate=False
)
+---------------------------------------------------+-----------------+----------------------------------+
|data                                               |A                |data - A                          |
+---------------------------------------------------+-----------------+----------------------------------+
|https://example1.org/path/to/file?param=42#fragment|param=42#fragment|https://example1.org/path/to/file?|
|https://example2.org/path/to/file                  |null             |https://example2.org/path/to/file |
+---------------------------------------------------+-----------------+----------------------------------+