I am trying to perform Hindi To English translation using s2s model, following - https://chunml.github.io/ChunML.github.io/project/Sequence-To-Sequence/

I have used https://github.com/karimkhanp/Seq2Seq/tree/master/data dataset which contains 37726 sentences for training.

Training took around 44 hours on my 8GB machine. I considered 3 layers, 10 epoch, 20000 vocap size

Other parameters

ap.add_argument('-max_len', type=int, default=200)
ap.add_argument('-vocab_size', type=int, default=20000)
ap.add_argument('-batch_size', type=int, default=100)
ap.add_argument('-layer_num', type=int, default=3)
ap.add_argument('-hidden_dim', type=int, default=1000)
ap.add_argument('-nb_epoch', type=int, default=10)
ap.add_argument('-mode', default='train') 

But now when I test using hindi sentence, it give UNK for all words. Though I take same sentence for testing as in training even it says UNK in results.

Test sentences:

डेली हिन्दी न्यूज - बुंदेलखंड का प्रथम अन्तरजालीय स्थल
मैं भारत से प्यार करता हूँ
ज़बूर जो कि दाउद को प्रदान की गयी
प्रशासनिक विभाजन
वे इस देश के प्रथम UNK -LRB- अफ्रीकी UNK -RRB-
नेपाली विदेश
वेल्श खिलाड़ी इंग्लैंड के लिए खेलने के लिए पात्र हैं
उनके बड़े भाई अजीत तेंडुलकर ने उन्हें खेलने के लिये प्रोत्साहित किया था ।
फिर एक मिनट के बाद किताब छत की ओर उछालकर उन्होंने कहा - चलो ।

Result :

of of of
of of of
the of of of

lrb lrb rrb rrb

the the of of of
the the of of of of
the the the of of of of

I could not understand the issue behind this result. I have used some sentences which are same as training sentences. Atleast answer should be correct for them.

Did I do anything wrong? I really appreciate any help

Answer -

if MODE == 'train':
    k_start = 1
    t1 = time()
    # If any trained weight was found, then load them into the model
    if len(saved_weights) != 0:
        print('[INFO] Saved weights found, loading...')
        epoch = saved_weights[saved_weights.rfind('_')+1:saved_weights.rfind('.')]
        k_start = int(epoch) + 1

    i_end = 0
    for k in range(k_start, NB_EPOCH+1):
        # Shuffling the training data every epoch to avoid local minima
        indices = np.arange(len(X))
        X = X[indices]
        y = y[indices]

        # Training 1000 sequences at a time
        for i in range(0, len(X), 1000):
            if i + 1000 >= len(X):
                i_end = len(X)
                i_end = i + 1000
            y_sequences = process_data(y[i:i_end], y_max_len, y_word_to_ix)

            print('[INFO] Training model: epoch {}th {}/{} samples'.format(k, i, len(X)))
            model.fit(X[i:i_end], y_sequences, batch_size=BATCH_SIZE, nb_epoch=1, verbose=2)
    print("Time taken to train the data in hour=>", (time()-t1)/3600)

# Performing test if we chose test mode
    # Only performing test if there is any saved weights
    if len(saved_weights) == 0:
        print("The network hasn't been trained! Program will exit...")
        # import pdb
        # pdb.set_trace()
        X_test = load_test_data('test', X_word_to_ix, MAX_LEN)
        X_test = pad_sequences(X_test, maxlen=X_max_len, dtype='int32')
        # print(X_test)
        # print(model.predict(X_test))
        predictions = np.argmax(model.predict(X_test), axis=2)
        # print(predictions)
        # print(y_ix_to_word)
        sequences = []
        for prediction in predictions:
            sequence = ' '.join([y_ix_to_word[index] for index in prediction if index > 0])

Clearly, the optimization of your loss function led to it only predicting the most common words in English. The probability of other words are obviously small. I am no expert, but I would check three things. 1. How long does it take for the loss to converge? With the output you show it is likely it does not take a long time. 2. Maybe try to filter out so you have different words in the sentences. 3. Check if it can help to optimize another loss function which would have a bias towards more diverse sentences. I hope this can give you a few ideas at least.

  • $\begingroup$ Thanks for giving these hints. predicting the most common words in English. Correct, I was also thinking same. I think your points can improve the result, but Sorry to say, I could not find how to test any of them. Will providing the code will help you to give more hints on your answer. Let me update the code $\endgroup$
    – user123
    Aug 28 '17 at 11:53
  • $\begingroup$ As the other person suggests, it could be you need more time for training. Check how the loss changes. I've had okay results with seq2seq models in about 10 hours training on a Titan GPU. Still however, biased towards simple sentences. Azure charges around 1.6 $ an hour for a machine with that kind of graphics card. $\endgroup$ Aug 28 '17 at 12:40
  • $\begingroup$ Thanks for your reply. waiting for the result is not an issue if it is going to converge. Currently for three layers and 10 epochs it took 43 hours on my system. I will increase my config, but just increasing the epochs and layers will improve the result? Rest of the techniques used in the code is correct to translation? $\endgroup$
    – user123
    Aug 28 '17 at 16:38
  • $\begingroup$ It should be clear when your loss has converged. Print it, or use tensorboard for example. I havn't really looked at your code, but adding layers will increase the training time for convergence and you don't know if it will succeed. Running more epochs, could help, or not, it all depends if your loss is still lowering. It is very important you check if your loss converged, that will tell if you are optimizing the wrong loss function/having a bad architecture, or simply need more time for training. $\endgroup$ Aug 31 '17 at 14:57

This is kind of a common issue in Seq2Seq model. I haven't tried machine translation, but I have tried text generation and got stuck with this problem of repeating words. The problem is in learning weights I believe. Even with GPUs it takes almost a week to get a considerable good result. I guess, you have used CPUs. for sure, CPUs takes much more time than 44 hours.

  • $\begingroup$ Yeah, I used CPU, taking time is fine, do you have any idea regarding improving the result? $\endgroup$
    – user123
    Aug 28 '17 at 11:55
  • 1
    $\begingroup$ yea. As I said. more time. I know its frustating that you have to wait for 10 days to get some results. In trainining my text generation model, from chunml.github.io he used a while loop and the results keep on getting better over time and you could stop when you feel its sensible. So, just keep on test the dataset for each epochs and see how long do you need to train for a better model. $\endgroup$
    – chmodsss
    Aug 28 '17 at 11:57
  • $\begingroup$ Thanks for your reply. waiting for the result is not an issue if it is going to converge. Currently for three layers and 10 epochs it took 43 hours on my system. I will increase my config, but just increasing the epochs and layers will improve the result? Rest of the techniques used in the code is correct to translation? $\endgroup$
    – user123
    Aug 28 '17 at 16:38
  • $\begingroup$ I'm afraid i'm no expert, I just shared my experience. As per the blog entry mentioned, the results get better with time. I'm too following the question for a better answer. Good luck. $\endgroup$
    – chmodsss
    Aug 29 '17 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.