# Replace all numeric values in a pyspark dataframe by a constant value

Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the dataframe by a constant numeric value (for example by the value 1)? Thanks in advance!

Example for the pyspark dataframe: $$\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 0.04 & 1 & 1.35 \\ 2 & -1 & null & -1.2 \\ 3 & null & 1.2 & null \end{array}$$

The result should be:

$$\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & null & 1 \\ 3 & null & 1 & null \end{array}$$

• Welcome to SO! Could you post some data and/or code example so that we can better help you? Oct 20, 2016 at 6:46

As per your problem, I think it might be easier to use lit. Try this-

from pyspark.sql.functions import lit
new_df = df.withColumn('column_name', lit(1))


Hope it helps!

Using lit would convert all values of the column to the given value.

To do it only for non-null values of dataframe, you would have to filter non-null values of each column and replace your value. when can help you achieve this.

from pyspark.sql.functions import when

df.withColumn('c1', when(df.c1.isNotNull(), 1))
.withColumn('c2', when(df.c2.isNotNull(), 1))
.withColumn('c3', when(df.c3.isNotNull(), 1))


This would result in:

\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & null & 1 \\ 3 & null & 1 & null \end{array}

Also, if you want to replace those null values with some other value too, you can use otherwise in combination with when. Let's say you want to impute 0 there:

from pyspark.sql.functions import when

df.withColumn('c1', when(df.c1.isNotNull(), 1).otherwise(0))
.withColumn('c2', when(df.c2.isNotNull(), 1).otherwise(0))
.withColumn('c3', when(df.c3.isNotNull(), 1).otherwise(0))


This would result in:

\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & 0 & 1 \\ 3 & 0 & 1 & 0 \end{array}

This would be easier if you have multiple columns:

from pyspark.sql.functions import when
cols = df.columns # list of all columns
for col in cols:
df= df.withColumn(col, when(df[col]>0,1).otherwise(0))

• How I can apply this condition only to the null values? Mar 31, 2021 at 11:47