I was working with a dataset where the features were unknown (encoded names) and, even though all of them already contained numbers, the data type was string or object. Additionally, it was indicated that missing values were marked as “na”. One of the challenges asked me to determine the mean, median, and standard deviation of certain columns, disregarding null values.
What I did was use the to_numeric() function with the parameter error=“coerce” so that the “na” values would automatically be converted to NaN and I could either drop them or use functions that automatically disregard these values (like describe()). However, I couldn't arrive at any of the answers listed as options.
In the example below, I arrived at a value of 0.195 for corr.
train_df['class'] = train_df['class'].map({'neg': 0, 'pos': 1})
for column in train_df.columns:
train_df[column] = pd.to_numeric(train_df[column], errors='coerce')
train_df.dropna(inplace=True, subset='var1')
train_df.dropna(inplace=True, subset='var2')
corr, _ = spearmanr(train_df['var1'], train_df['var2'])
corr
So, I decided to take a different approach to see if I could find the right answer: I decided to manually delete all “na” values using a boolean mask and then convert the entire dataset with to_numeric() and calculate the statistics. This approach worked, and I even used the parameter error=“raise” and no errors were thrown. I finally got the answers I was looking for.
In the example below, I arrived at a value of 0.310 for corr.
mask = ~train_df.apply(lambda x: x.astype(str).str.contains('na')).any(axis=1)
df_cleaned = train_df[mask]
df_cleaned['var1'] = pd.to_numeric(df_cleaned['var1'])
df_cleaned['var2'] = pd.to_numeric(df_cleaned['var2'])
corr, _ = spearmanr(df_cleaned['var1'], df_cleaned['var2'])
corr
Does anyone know why this happened? Why did I get different statistics when using the first approach compared to the second? Theoretically, shouldn’t both result in the same clean dataset? I mean, if there were other null values besides “na” that were automatically converted to NaN in the first approach, these values would have caused an error in the second approach when using error=“raise” since I only deleted the “na” from the dataset. Does anyone have any idea why this happened?
