Is there a way to replace existing values with NaN

Question

I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column having different proportions of Nan values.

I'm using the Iris dataset for this experimentation to see how the algorithms work and which one works the best. The link for the dataset is here.

Thanks in advance for the help.

Answer 1

Randomly replace values in a numpy array

# The dataset
data = pd.read_csv('iris.data')
mat = data.iloc[:,:4].as_matrix()

Set the number of values to replace. For example 20%:

# Edit: changed len(mat) for mat.size
prop = int(mat.size * 0.2)

Randomly choose indices of the numpy array:

i = [random.choice(range(mat.shape[0])) for _ in range(prop)]
j = [random.choice(range(mat.shape[1])) for _ in range(prop)]

Change values with NaN

mat[i,j] = np.NaN

Dropout for any array dimension

Another way to do that with an array of more than 2 dimensions would be to use the numpy.put() function:

import numpy as np
import random
from sklearn import datasets

data = datasets.load_iris()['data']

def dropout(a, percent):
    # create a copy
    mat = a.copy()
    # number of values to replace
    prop = int(mat.size * percent)
    # indices to mask
    mask = random.sample(range(mat.size), prop)
    # replace with NaN
    np.put(mat, mask, [np.NaN]*len(mask))
    return mat

This function returns a modified array:

modified = dropout(data, 0.2)

We can verify that the correct number of values have been modified:

np.sum(np.isnan(modified))/float(data.size)

[out]:

0.2

Answer 2

Depending on the data structure you are keeping the values there might be different solutions.

If you are using Numpy arrays, you can employ np.insert method which is referred here:

import numpy as np
a = np.arrray([(122.0, 1.0, -47.0), (123.0, 1.0, -47.0), (125.0, 1.0, -44.0)]))
np.insert(a, 2, np.nan, axis=0)
array([[ 122.,    1.,  -47.],
       [ 123.,    1.,  -47.],
       [  nan,   nan,   nan],
       [ 125.,    1.,  -44.]])

If you are using Pandas you can use instance method replace on the objects of the DataFrames as referred here:

In [106]:
df.replace('N/A',np.NaN)

Out[106]:
    x    y
0  10   12
1  50   11
2  18  NaN
3  32   13
4  47   15
5  20  NaN

In the code above, the first argument can be your arbitrary input which you want to change.

Answer 3

Edited my previous comment as there was an Syntax error, This happen as I am new in this join recently(01/04/2021) on this platform you can try replace function with NumPy library which will help to speed up the process.

df.replace('^^',np.NaN) or
df.replace('not filled in',np.NaN),
df.replace('&&', np.NaN),
df.replace('values needed', np.NaN),
df.replace('Na', np.NaN)

this are various string replacement

values are taken from my previous comment and above problem statement

Out[106]:

	x	y
0	10	12
1	50	11
2	18	Nan
3	32	13
4	47	15
5	20	Nan