0
$\begingroup$

I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are input into a BLSTM model. I am using an AzureML server GPU-N6 for training.

machine specs : STANDARD_NV6 (6 Cores, 56 GB RAM, 380 GB Disk) GPU - 1 x NVIDIA Tesla M60

It exits with code 137, which is an out of memory error. I see the data is downloaded into the local node and NumPy array is created. the OOM happens when the batch processing starts.

I have tried to run the model with only one input (eGeMaps) but run into memory issues.

My question is, is there a way to batch load the NumPy file, similar to an image dataset? I have tried to split the input into individual data files and use tensor dataset or slices, but it doesn't work when loading a NumPy file. Ideally, I would like an equivalent of ImageDataGenerator for NumPy files.

EGEMAPS={'FEATURES':100,'TIMESTEPS':14050} MFCC={'FEATURES':100,'TIMESTEPS':14050} DENSENET201={'FEATURES':1920,'TIMESTEPS':1415}

The training dataset consists of 136 participants each with an array of the sizes mentioned above. I created a single large NumPy array, each row representing a participant and array of the features.

This is the model I am training.

#Create Model :
inputs_eGeMAPS = Input(EGEMAPS_INPUT_SHAPE)
inputs_MFCC = Input(MFCC_INPUT_SHAPE)
inputs_Densenet = Input(DENSENET201_INPUT_SHAPE)

blstm1_eGeMAPS = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")

blstm2_eGeMAPS= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")

blstm1_MFCC = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")

blstm2_MFCC= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")

blstm1_Densenet = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")

blstm2_Densenet= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")

audio_lstm_output = concatenate([blstm2_eGeMAPS,blstm2_MFCC,blstm2_Densenet])

dense1= Dense(500,activation='relu')(audio_lstm_output)

print(f"dense:{dense1}")

dense2= Dense(100,activation='relu')(dense1)
print(f"dense:{dense2}")

dense3= Dense(60,activation='relu')(dense2)
print(f"dense:{dense3}")

output = Dense(1)(dense3)
print(f"output:{output}")

model = Model(inputs=[inputs_eGeMAPS,inputs_MFCC,inputs_Densenet], outputs=output)
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])

print(model.summary())

epochs = 1
hist = model.fit([BoAW_eGeMAPS_Dev_tf,BoAW_MFCC_Dev_tf,Densenet_Dev_tf],Y_train_tf,epochs=epochs, batch_size=batch_size,validation_split=0.2,shuffle=False)
$\endgroup$
0
$\begingroup$

Looking at your code, it looks like you're using Keras w/ a Tensorflow backend -- or Tensorflow functions using its Keras submodule. In that case, you can train your model using its fit_generator function.

As the name implies, if you convert your training data into a generator, it will substantially reduce the memory cost needed to train your model. Here's an example of training a Keras model using fit_generator and how you can preprocess your data to utilize it.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.