I would like to hear some views on a problem I have with my dataset (which I presume to be a common one).
Let's say I have the following dataset
SKUID PRODUCT QTY MFGDT ...... EXPDT SUPPLIERID SUPPLIERPH CUSTOMFIELD FD001 MILK 3 12/01/18 14:12:02 ... 18/01/18 SV01 04053XXXX FD002 CREAM 3 12/01/18 14:12:02 ... 18/01/18 SV01 04053XXXX FD003 CHEESE 5 12/01/18 14:12:02 ... 18/01/18 SV01 04053XXXX FD004 BUTTER 2 12/01/18 14:12:02 ... 18/01/18 SV01 04053XXXX FD005 ICECREAM 1 12/01/18 14:12:02 ... SV01 04053XXXX
The dataframe is of shape (123078, 199) and there are few records where the field values are jumbled.
When reading this csv using pandas, I used the
error_bad_lines=False attribute to skip the lines where there is a mis-match in the field.
However, I was wondering if there is a way to FIX the data by some means (say, pattern matching with the previous items in the column; based on the dtype etc.)
How do we generally handle a scenario where every record is crucial (or say inter-dependent) and there is mismatch in the field?