14
$\begingroup$

Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the dataframe by a constant numeric value (for example by the value 1)? Thanks in advance!

Example for the pyspark dataframe: $$ \begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 0.04 & 1 & 1.35 \\ 2 & -1 & null & -1.2 \\ 3 & null & 1.2 & null \end{array} $$

The result should be:

$$ \begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & null & 1 \\ 3 & null & 1 & null \end{array} $$

$\endgroup$
1
  • $\begingroup$ Welcome to SO! Could you post some data and/or code example so that we can better help you? $\endgroup$
    – Stereo
    Oct 20, 2016 at 6:46

3 Answers 3

9
$\begingroup$

Using lit would convert all values of the column to the given value.

To do it only for non-null values of dataframe, you would have to filter non-null values of each column and replace your value. when can help you achieve this.

from pyspark.sql.functions import when   

df.withColumn('c1', when(df.c1.isNotNull(), 1))
  .withColumn('c2', when(df.c2.isNotNull(), 1))
  .withColumn('c3', when(df.c3.isNotNull(), 1))

This would result in:

\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & null & 1 \\ 3 & null & 1 & null \end{array}

Also, if you want to replace those null values with some other value too, you can use otherwise in combination with when. Let's say you want to impute 0 there:

from pyspark.sql.functions import when   

df.withColumn('c1', when(df.c1.isNotNull(), 1).otherwise(0))
  .withColumn('c2', when(df.c2.isNotNull(), 1).otherwise(0))
  .withColumn('c3', when(df.c3.isNotNull(), 1).otherwise(0))

This would result in:

\begin{array}{c|lcr} & \text{c1} & \text{c2} & \text{c3} \\ \hline 1 & 1 & 1 & 1 \\ 2 & 1 & 0 & 1 \\ 3 & 0 & 1 & 0 \end{array}

$\endgroup$
8
$\begingroup$

As per your problem, I think it might be easier to use lit. Try this-

from pyspark.sql.functions import lit
new_df = df.withColumn('column_name', lit(1))

Hope it helps!

$\endgroup$
2
$\begingroup$

This would be easier if you have multiple columns:

from pyspark.sql.functions import when   
cols = df.columns # list of all columns
for col in cols:
    df= df.withColumn(col, when(df[col]>0,1).otherwise(0))
$\endgroup$
1
  • $\begingroup$ How I can apply this condition only to the null values? $\endgroup$
    – alex3465
    Mar 31, 2021 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.