0
$\begingroup$

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in the neighborhood. In the beginning of the input signal you can see nans embedded in an otherwise continuum 's' episode. My definition as to what characterizes an episode is occurrence of the corresponding symbol at least 5 times in a row. Also, in my interest to be given more weight to 'r' and 'l' when tied with 's'.

iput = ['s','s','s','s','s','s',np.nan,'s',np.nan,'s','s','s','s','r',np.nan,np.nan,'r','r','r','r','s','s','s','s','s',np.nan,np.nan,'s','s','s','l','l','l','l','l',np.nan,'l','l','l']
oput = ['s','s','s','s','s','s','s','s','s','s','s','s','s','r','r','r','r','r','r','r','s','s','s','s','s','s','s','s','s','s','l','l','l','l','l','l','l','l','l']

I tried knn as following but it is rather suitable for numerical column and also imputing nans with zeros. I was hoping for some ideas how to tackle this problem. from fancyimpute import KNN knnimpute = KNN(k=5)

>>>x = np.array([0,np.nan,1,1,1,np.nan,2,2,2,2,np.nan,2,3,3,3,3,np.nan,1,1,2,2,np.nan,1,3,3,3,3])
>>>x2 = knnimpute.fit_transform(x.reshape(-1,1))
>>>x2
>>> 
array([[0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [2.],
           [2.],
           [2.],
           [2.],
           [0.],
           [2.],
           [3.],
           [3.],
           [3.],
           [3.],
           [0.],
           [1.],
           [1.],
           [2.],
           [2.],
           [0.],
           [1.],
           [3.],
           [3.],
           [3.],
           [3.]])
$\endgroup$
1
$\begingroup$

The following script will give the value of the most frequent item to the nan value. It is a list of 7 items, since it checks the three samples before the nan, the nan itself and the three after the nan samples.

iput = ['s','s','s','s','s','s',np.nan,'s',np.nan,'s','s','s','s','r',np.nan,np.nan,'r','r','r','r','s','s','s','s','s',np.nan,np.nan,'s','s','s','l','l','l','l','l',np.nan,'l','l','l']
for i in range(len(iput)):
    if type(iput[i]) is float:
        iput[i]=max(iput[i-3:i+3],key=iput[i-3:i+3].count)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.