-1
$\begingroup$

After merging two dataframes, I end up with NaNs in the new dataframe, because one csv does not have all the ID's that the other has (Two dataframes of different sizes for example). So some rows have NaN values in some columns. Should I deal with those block missing values by removing them or replacing them with mean or median ? Should I remove or replace them with ?

$\endgroup$
3
  • $\begingroup$ Can you clarify if you are joining column-wise or row-wise? $\endgroup$
    – hH1sG0n3
    Oct 23, 2020 at 11:38
  • $\begingroup$ Column-wise for my case. $\endgroup$
    – Moez ‌
    Oct 23, 2020 at 13:19
  • $\begingroup$ They are not normal missing values. You might want to get a look at here to deal with values missing by blocks : datascience.stackexchange.com/questions/84066/… $\endgroup$ Oct 24, 2020 at 9:37

2 Answers 2

1
$\begingroup$

You probably should

  1. conduct a missing values analysis to see what is the percentage of missing per column (figure below, from dataprep package)
  2. Decide a threshold according to which you may want to completely drop a column or not (depending on how your analysis or model treats nans as well)
  3. For the columns that are not dropped, you should impute the missing values experimenting with relevant techniques, e.g. average, std etc. (also depends on the type of the data and feature). https://scikit-learn.org/stable/modules/impute.html

Dataprep package

$\endgroup$
5
  • $\begingroup$ Alright so i'm treating those as normal missing values thanks ! $\endgroup$
    – Moez ‌
    Oct 23, 2020 at 14:03
  • $\begingroup$ I am not sure what a non-normal missing value would mean in this context, given a missing value is a value that is missing. $\endgroup$
    – hH1sG0n3
    Oct 23, 2020 at 14:54
  • $\begingroup$ Imagine if i have a csv file loaded in with missing data already that's usual missing data. But if I have two dataframes with different sizes and/or number of columns, the merging would result in rows containing NaN values in the different columns that are present in one df and not present in the other. That's what I mean. $\endgroup$
    – Moez ‌
    Oct 23, 2020 at 14:57
  • $\begingroup$ Sure. Unless there is a condition specifically related to merging the two dataframes, I think you can treat your nans are normal missing values.I hope this is helpful. $\endgroup$
    – hH1sG0n3
    Oct 23, 2020 at 15:20
  • $\begingroup$ Thank you so much for your answer ! $\endgroup$
    – Moez ‌
    Oct 23, 2020 at 15:32
0
$\begingroup$

nan values in pandas and other python packages represent missing data. In other languages they are often called NULL, NA or similar. They can arise when you left join two tables and the right table has no corresponding element in the left table. Or they can be entered manually. The interpretation is just "missing data". So ideally you want to keep them to keep track of what was missing.

Unlike some other languages, python does not have a null element of each type. pandas uses the float nan for missing data which was actually only meant to represent "not a number", floating point results of undefined mathematical operations like 0/0, inf/inf, etc.

This is a frequent cause of trouble (you are processing strings, and once in a while you have these nans of entirely different type). For this reason you might want to use, for example, the isna function to filter them or the fillna function to replace them with some other value like "" for strings.

Pandas itself has a page on dealing with missing data.

$\endgroup$
1
  • $\begingroup$ I like your answer it's very detailed. But it seems like my question has been misunderstood. I edited it now. I'm actually familiar with what you said and the techniques your presented in the link. However i'm asking if those techniques are still applicable if those NaNs result from a combination/join/jmerging of two dataframes. I'm interested to know how data scientists deal with that sort of problem. I'm looking forward to your answer @Valentas and i'm grateful ! $\endgroup$
    – Moez ‌
    Oct 23, 2020 at 13:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.