1
$\begingroup$

After analyzing questions on a forum, a human support team has created a set of general answers, that can be used to provide basic answers on the forum.

I am trying to build a system that:

  1. Selects best answer from this set of answers for a given question. How to do this?

  2. Estimates acceptability of such an answer. Which metrics to use?

Using document embeddings, such as doc2vec to find similarity between question and answer does not solve the problem, I think. Other ideas?

Update 1

In my case I don't have labeled data set with good answers to train my model. My problem is unsupervised learning problem.

$\endgroup$

1 Answer 1

2
$\begingroup$

This problem is multiple choice answering question. I can see you have already tried gensim, doc2vec etc. You can try pytorch based transformer solution. Here is the link: multiple-choice . You can create your data in swag format and remove --do_train in below code for prediction on your dataset.

It has been trained on swag dataset and has given decent accuracy.

If it works fine for you then good, else you would like to finetune it. For finetuning --do_train should be mentioned below:

#training on 4 tesla V100(16GB) GPUS
export SWAG_DIR=/path/to/swag_data_dir
python ./examples/run_multiple_choice.py \
--model_type roberta \
--task_name swag \
--model_name_or_path roberta-base \
--do_train \
--do_eval \
--do_lower_case \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir models_bert/swag_base \
--per_gpu_eval_batch_size=16 \
--per_gpu_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output
$\endgroup$
2
  • $\begingroup$ Thanks, unfortunately in my case I don't have labeled data set with good answers to train my model. My problem is unsupervised learning problem. $\endgroup$
    – dokondr
    Mar 11, 2020 at 10:26
  • 1
    $\begingroup$ BERT is a pretrained model. I think you should give it a try on your data without even training. I had got good results even without training $\endgroup$
    – Sonu
    Mar 11, 2020 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.