the follwoing code is a generator I am using to pass the training, for this generator i am passing questions and answers. the first 200 Q and A I am reserving for testing. The relevant part of the code is after the padding line for ans_pad and ques_pad. Should I also one hot this data. I am trying to build a chatbot, but so far I ma getting very poor results and I feel like the issue lies in this part. I one hoted the output data, is it logical to do the same for the input as well? I tried doing it but it led to errors in my input layer of the model
def generator(questions,answers, batch_size=32): num_samples = len(questions) while True: for offset in range(200, num_samples, batch_size): question_samples = questions[offset:offset+batch_size] answer_samples = answers[offset:offset+batch_size] ques_train =  ans_train =  for i in question_samples: ques_train.append(i) for i in question_samples: ans_train.append(i) ques_train = np.array(ques_train) ans_train = np.array(ans_train) ans_pad = sequence.pad_sequences(ans_train, maxlen = maxLen, dtype = 'int32', padding = 'post', truncating = 'post') ques_pad = sequence.pad_sequences(ques_train, maxlen = maxLen, dtype = 'int32', padding = 'post', truncating = 'post') encoder_input_data = np.array( ques_pad) decoder_input_data = np.array( ans_pad) for i in range(len(ans_train)) : ans_train[i] = ans_train[i][1:] padded_answers = sequence.pad_sequences( ans_train , maxlen=maxLen , padding='post' ) onehot_answers = utils.to_categorical( padded_answers , vocab_size ) decoder_output_data = np.array( onehot_answers ) yield(encoder_input_data,decoder_input_data),decoder_output_data
should I one hot the input question and answers, or just the output?