3
$\begingroup$

I am working on lung CT images from luna16 dataset, the dataset have a 3d lung image and a label from CSV file, I have a code for constructing 2d list from 3d array 25x25x25 (the 3d image) and a label [0,1] or [1,0] from CSV file, after creating the 2d list I want to save it in numpy file, below is my code for creating the 2d list and saving it in numpy file:

def getIDlist(csv_Dist,Data_Dist):
    # receive marked coords and ID in annotations.csv, and return the distination with coords.
    print('loading')
    data = np.loadtxt(csv_Dist, delimiter = ',', dtype = 'str')
    # delete the header file via 1:0, and receive the ID, x, y, z, r via 0:5 to a list.
    ID_coords = data[1:,0:5][0:10000] # get list of 'seriesuid' 'coordX' 'coordY' 'coordZ' 'class' (without header).
    # define the output file.
    ID_dist = []

    print('strat finding')
    process_bar = ShowProcess(len(ID_coords))

    for ID,x,y,z,label in ID_coords: 
        ID = ID +'.mhd' 
        found = 0       
        for parent, dirnames, filenames in os.walk(Data_Dist):
            for filename in filenames:# loop inside all files                 
                if ID == filename: # ID + .mhd in csv equal to filename in files
                    process_bar.show_process()
                    ID = parent + '\\' + ID# ID gets full path of the founded file
                    ID_dist.append([ID,x,y,z,label])# ID_dist gets info of founded files
                    found = 1
                    #print("found: ", found)
                    break
            if found == 1:
                break
        if found == 1:
            continue

    process_bar.close()                 
    return ID_dist 

def get3Dmatrix(ID_dist):

    print('preparing the 3d matrix')
    matrixlist = []
    for Dist, xcoords, ycoords, zcoords, label in tqdm(ID_dist):
        # read the image
        imagearray,origin,spacing = load_itk_image(Dist)
        # resample in to 1mm*1mm*1mm
        imagearray = resample(imagearray,spacing,(1,1,1))

        # transfer world coordinates to voxel-coordinates, divide new spacing 1mm
        z = int(round((float(zcoords)-float(origin[0]))/1))
        y = int(round((float(ycoords)-float(origin[1]))/1))
        x = int(round((float(xcoords)-float(origin[2]))/1))

        # get the 3D array with shape 25*25*25           
        imagearray = imagearray[z-13:z+12,y-13:y+12,x-13:x+12]

        #converting the label number into a one-hot-encoding
        if int(label) == 1: 
            label=np.array([0,1])
        elif int(label) == 0: 
            label=np.array([1,0])

        # put it into output file
        matrixlist.append([imagearray,label])# 2d list consist of 3d array + label of all cases.
    return matrixlist 

 def main():
    start_time = time.time()
    # get ID_list from the csv and data dist.
    ID_list = getIDlist(candidates_V2_Dist, Data_Dist)# nested list - get file name with dist + x,y,z,class
    # Data_set[i][0] is the 3D array, Data_set[i][1] is the label
    Data_set = get3Dmatrix(ID_list) # 2d list consist of 3d array + label of all cases.
    print("Begin saving in numpy file")
    np.save(output_path+'np_ds(10000)-25-25-25(zyx)_one_hot.npy', Data_set)
    print("%s time takes in seconds" % (time.time() - start_time))

if __name__ == "__main__":
    main()

my problem is:

1- with approximatly 550 samples, the RAM gets fulled and I get memory error, I am working on dell inspiron core i7 with 16 gb ram laptop.

2- it takes 34 seconds for creating each sample, and I see this is huge amount of time for only one sample.

I did a lot of search in google and asked a question in some other forums but didn't get any answer, can anyone help me, please? realy I am confused with that error. image below is the error message: enter image description here

$\endgroup$
  • $\begingroup$ Where in the code does the memory error occur? Try to find that point. $\endgroup$ – n1k31t4 Aug 30 '18 at 12:16
1
$\begingroup$

I would recommend breaking the problem down a little bit to reduce the memory usage at any one time.

The first part of your main function gets the IDs using getIDList. That seems fine, so leave it there.

I would then break that list down into smaller chunks, calling get3Dmatrix on each chunk in turn. Altering your code, it might look something like this:

# Get number of entries in ID list
N = len(ID_list)

# break it down into a number of chunks e.g. 4, based on your progress bar
import numpy as np    # should already be imported

N = len(ID_list)
num_chunks = 4           # you can play with this number, making it larger until you don't get emmory errors
chunks = np.linspace(0, N, num_chunks)

for i in range(len(chunks) - 1):
    this_sublist = ID_list[chunks[i] : chunks[i + 1]]
    sub_data_set = get3Dmatrix(this_sublist)

    # At this point, either save this sub_data_set, or try appending it to another list toi make one final numpy matrix at the end before saving

...

print("Begin saving in numpy file")
np.save(output_path+'np_ds(10000)-25-25-25(zyx)_one_hot.npy', Data_set)
print("%s time takes in seconds" % (time.time() - start_time))

Even from the traceback you added, it is hard to say where exactly in your code that is happening.

Roughly looking at the dimensions you mention, it also doesn't seem plausible the a 16Gb machine is running out of memory - so I must not completely understand just how many images/patches are being saved.

$\endgroup$
  • $\begingroup$ I think the idea if chunks help and I will try it, but this sub_list every time goes to next 4 chunks or each time gets same 4 chunks? are first 4 chunks removed from memory when get to next chunk? in tqdm it says in 556 samples out of 1000 samples the error is raised. $\endgroup$ – Hunar Aug 31 '18 at 6:34
  • $\begingroup$ With num_chunks=4, it should do 250 of the current 1000 in tdqm each time and because it's in a loop using the same variable names, it should overwrite the memory. Try it out! $\endgroup$ – n1k31t4 Aug 31 '18 at 10:42
  • $\begingroup$ I try it, nothing is changed, sample problem. $\endgroup$ – Hunar Sep 1 '18 at 12:44
  • $\begingroup$ added each sub_data_set to a new dataset inside chunks for loop, but I get the same error. $\endgroup$ – Hunar Sep 1 '18 at 13:27
  • $\begingroup$ Instead of adding each sub_data_set to a master dataset in the loop, try saving the results in chunks to disk and so each loop should overwrite that current chunk: sub_data_set. Set num_chunks larger and larger until that works. If there is still a problem, I think there must be a bug elsewhere, given you have 16Gb of RAM. $\endgroup$ – n1k31t4 Sep 1 '18 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.