I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column having different proportions of Nan values.

I'm using the Iris dataset for this experimentation to see how the algorithms work and which one works the best. The link for the dataset is here.

Thanks in advance for the help.


Randomly replace values in a numpy array

# The dataset
data = pd.read_csv('iris.data')
mat = data.iloc[:,:4].as_matrix()

Set the number of values to replace. For example 20%:

# Edit: changed len(mat) for mat.size
prop = int(mat.size * 0.2)

Randomly choose indices of the numpy array:

i = [random.choice(range(mat.shape[0])) for _ in range(prop)]
j = [random.choice(range(mat.shape[1])) for _ in range(prop)]

Change values with NaN

mat[i,j] = np.NaN

Dropout for any array dimension

Another way to do that with an array of more than 2 dimensions would be to use the numpy.put() function:

import numpy as np
import random
from sklearn import datasets

data = datasets.load_iris()['data']

def dropout(a, percent):
    # create a copy
    mat = a.copy()
    # number of values to replace
    prop = int(mat.size * percent)
    # indices to mask
    mask = random.sample(range(mat.size), prop)
    # replace with NaN
    np.put(mat, mask, [np.NaN]*len(mask))
    return mat

This function returns a modified array:

modified = dropout(data, 0.2)

We can verify that the correct number of values have been modified:



  • $\begingroup$ This worked. But I have a question, out of 600 observations for the 4 columns, only 75 observations were replaced with NaN when i set it to 0.5 (12.5% replaced), how does that work?. $\endgroup$ – uharsha33 Apr 13 '18 at 10:37
  • $\begingroup$ I see the problem. I will update my answer. $\endgroup$ – michaelg Apr 13 '18 at 10:58
  • $\begingroup$ Thanks, your 1st method worked out fine, but 2nd way might reduce the coding needed. I have a question though, didn't you forget to "mat = data.iloc[:, :4].as_matrix()" in the second method? $\endgroup$ – uharsha33 Apr 16 '18 at 10:23
  • $\begingroup$ In the second method I assume that you are passing a numpy array, while in the first it was a pandas data frame. Sorry for the mismatch. $\endgroup$ – michaelg Apr 16 '18 at 10:34

Depending on the data structure you are keeping the values there might be different solutions.

If you are using Numpy arrays, you can employ np.insert method which is referred here:

import numpy as np
a = np.arrray([(122.0, 1.0, -47.0), (123.0, 1.0, -47.0), (125.0, 1.0, -44.0)]))
np.insert(a, 2, np.nan, axis=0)
array([[ 122.,    1.,  -47.],
       [ 123.,    1.,  -47.],
       [  nan,   nan,   nan],
       [ 125.,    1.,  -44.]])

If you are using Pandas you can use instance method replace on the objects of the DataFrames as referred here:

In [106]:

    x    y
0  10   12
1  50   11
2  18  NaN
3  32   13
4  47   15
5  20  NaN

In the code above, the first argument can be your arbitrary input which you want to change.


Edited my previous comment as there was an Syntax error, This happen as I am new in this join recently(01/04/2021) on this platform you can try replace function with NumPy library which will help to speed up the process.

df.replace('^^',np.NaN) or
df.replace('not filled in',np.NaN),
df.replace('&&', np.NaN),
df.replace('values needed', np.NaN),
df.replace('Na', np.NaN)

this are various string replacement

values are taken from my previous comment and above problem statement


x y
0 10 12
1 50 11
2 18 Nan
3 32 13
4 47 15
5 20 Nan

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.