0
$\begingroup$

the follwoing code is a generator I am using to pass the training, for this generator i am passing questions and answers. the first 200 Q and A I am reserving for testing. The relevant part of the code is after the padding line for ans_pad and ques_pad. Should I also one hot this data. I am trying to build a chatbot, but so far I ma getting very poor results and I feel like the issue lies in this part. I one hoted the output data, is it logical to do the same for the input as well? I tried doing it but it led to errors in my input layer of the model

def generator(questions,answers, batch_size=32):
num_samples = len(questions)
while True:
    for offset in range(200, num_samples, batch_size):
        question_samples = questions[offset:offset+batch_size]
        answer_samples = answers[offset:offset+batch_size]
        ques_train = []
        ans_train = []
        for i in question_samples:
            ques_train.append(i)
        for i in question_samples:
            ans_train.append(i)
        ques_train = np.array(ques_train)
        ans_train = np.array(ans_train)
        ans_pad = sequence.pad_sequences(ans_train, maxlen = maxLen, dtype = 'int32', padding = 'post', truncating = 'post')
        ques_pad = sequence.pad_sequences(ques_train, maxlen = maxLen, dtype = 'int32', padding = 'post', truncating = 'post')
        encoder_input_data = np.array( ques_pad)
        decoder_input_data = np.array( ans_pad)
        for i in range(len(ans_train)) :
            ans_train[i] = ans_train[i][1:]
        padded_answers = sequence.pad_sequences( ans_train , maxlen=maxLen , padding='post' )
        onehot_answers = utils.to_categorical( padded_answers , vocab_size )
        decoder_output_data = np.array( onehot_answers )
        yield(encoder_input_data,decoder_input_data),decoder_output_data

should I one hot the input question and answers, or just the output?

$\endgroup$
  • $\begingroup$ Welcome to DS StackExchange. Please update your question adding some explanation of your problem. At the moment it's just a code block that no one understands. Please explain your dataset, your task, the problems encountered so far, what you tried to solve them up to now. As of now, it's not possible to help you. Thanks $\endgroup$ – Leevo Jun 4 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.