Replace all numeric values in a pyspark dataframe by a constant value

Question

Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the dataframe by a constant numeric value (for example by the value 1)? Thanks in advance!

Example for the pyspark dataframe: $$ \begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 0.04 & 1 & 1.35 \\ 2 & -1 & null & -1.2 \\ 3 & null & 1.2 & null \end{array} $$

The result should be:

$$ \begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & null & 1 \\ 3 & null & 1 & null \end{array} $$

Answer 1

As per your problem, I think it might be easier to use lit. Try this-

from pyspark.sql.functions import lit
new_df = df.withColumn('column_name', lit(1))

Hope it helps!

Answer 2

Using lit would convert all values of the column to the given value.

To do it only for non-null values of dataframe, you would have to filter non-null values of each column and replace your value. when can help you achieve this.

from pyspark.sql.functions import when   

df.withColumn('c1', when(df.c1.isNotNull(), 1))
  .withColumn('c2', when(df.c2.isNotNull(), 1))
  .withColumn('c3', when(df.c3.isNotNull(), 1))

This would result in:

\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & null & 1 \\ 3 & null & 1 & null \end{array}

Also, if you want to replace those null values with some other value too, you can use otherwise in combination with when. Let's say you want to impute 0 there:

from pyspark.sql.functions import when   

df.withColumn('c1', when(df.c1.isNotNull(), 1).otherwise(0))
  .withColumn('c2', when(df.c2.isNotNull(), 1).otherwise(0))
  .withColumn('c3', when(df.c3.isNotNull(), 1).otherwise(0))

This would result in:

\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & 0 & 1 \\ 3 & 0 & 1 & 0 \end{array}

Answer 3

This would be easier if you have multiple columns:

from pyspark.sql.functions import when   
cols = df.columns # list of all columns
for col in cols:
    df= df.withColumn(col, when(df[col]>0,1).otherwise(0))