I have two files : Test_data - contains the features of a dataset to find predictions for Submission_data - contains two columns : The index column for test data and another column for its corresponding predicted value

So , I have to make predictions on the test data and store the predicted values in the submission file.

During preprocessing of the test data , I am dropping rows that do not contain values (NaN) for atleast 50% of the features(columns) :

test_data = test_data.dropna(thresh=math.ceil(test_data.shape[1]/2))

Now , How do I remove the corresponding rows in the submissions dataframe ? Because , If I drop some rows in the test data , I cannot make a prediction for the corresponding row in the submissions dataframe/file.

The problem is , there is an Index column that does NOT HAVE UNIQUE values (In both test data and submissions data)

So , How do I drop the rows in Submissions data that were also dropped in Test data ?

I am new to ML challenges and I find this challenging .


1 Answer 1


When you read the two csv files and store the data in two dataframes, you could then combine it into one dataframe, do the dropna and then split it back. I will give an example using pandas

import pandas as pd df1 = pd.read_csv('test_data.csv') df2 = pd.read_csv('submission_data.csv') df3 = pd.concat([df1, df2], axis=1) # this will combine the two dfs.

reduced_data = df3.dropna(thresh=math.ceil(test_data.shape[1]/2)) predictions = reduced_data.loc[:,['predictions']] reduced_data.drop(columns=['predictions'], inplace=True)

#instead of 'predictions', use whatever column name you have for the predictions in submission_data.csv file.

Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.