I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are input into a BLSTM model. I am using an AzureML server GPU-N6 for training.
machine specs : STANDARD_NV6 (6 Cores, 56 GB RAM, 380 GB Disk) GPU - 1 x NVIDIA Tesla M60
It exits with code 137, which is an out of memory error. I see the data is downloaded into the local node and NumPy array is created. the OOM happens when the batch processing starts.
I have tried to run the model with only one input (eGeMaps) but run into memory issues.
My question is, is there a way to batch load the NumPy file, similar to an image dataset? I have tried to split the input into individual data files and use tensor dataset or slices, but it doesn't work when loading a NumPy file. Ideally, I would like an equivalent of ImageDataGenerator for NumPy files.
EGEMAPS={'FEATURES':100,'TIMESTEPS':14050} MFCC={'FEATURES':100,'TIMESTEPS':14050} DENSENET201={'FEATURES':1920,'TIMESTEPS':1415}
The training dataset consists of 136 participants each with an array of the sizes mentioned above. I created a single large NumPy array, each row representing a participant and array of the features.
This is the model I am training.
#Create Model :
inputs_eGeMAPS = Input(EGEMAPS_INPUT_SHAPE)
inputs_MFCC = Input(MFCC_INPUT_SHAPE)
inputs_Densenet = Input(DENSENET201_INPUT_SHAPE)
blstm1_eGeMAPS = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")
blstm2_eGeMAPS= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")
blstm1_MFCC = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")
blstm2_MFCC= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")
blstm1_Densenet = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")
blstm2_Densenet= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")
audio_lstm_output = concatenate([blstm2_eGeMAPS,blstm2_MFCC,blstm2_Densenet])
dense1= Dense(500,activation='relu')(audio_lstm_output)
print(f"dense:{dense1}")
dense2= Dense(100,activation='relu')(dense1)
print(f"dense:{dense2}")
dense3= Dense(60,activation='relu')(dense2)
print(f"dense:{dense3}")
output = Dense(1)(dense3)
print(f"output:{output}")
model = Model(inputs=[inputs_eGeMAPS,inputs_MFCC,inputs_Densenet], outputs=output)
model.compile(optimizer='adam',
loss='mean_squared_error',
metrics=['accuracy'])
print(model.summary())
epochs = 1
hist = model.fit([BoAW_eGeMAPS_Dev_tf,BoAW_MFCC_Dev_tf,Densenet_Dev_tf],Y_train_tf,epochs=epochs, batch_size=batch_size,validation_split=0.2,shuffle=False)
