I have around 40K question and answers. How can I build a machine learning model so that if any new question comes it has to detect as duplicate or not?
A simple method is to compare similarity based on words. TF-IDF can be used considering the importance of each word. The similarity of words can be measured in various ways, such as cosine similarity. There are several recent algorithms:
- W. E. Zhang, Q. Z. Sheng, J. H. Lau, and E. Abebe, “Detecting duplicate posts in programming qa communities via latent semantics and association rules,” in 26th International Conference on World Wide Web (WWW), Geneva, Switzerland, 2017, pp. 1221–1229.
- B. Xu, D. Ye, Z. Xing, X. Xia, G. Chen, and S. Li, “Predicting semantically linkable knowledge in developer online forums via convolutional neural network,” in 31st International Conference on Automated Software Engineering (ASE), 2016, pp. 51–62.
- Y. Zhang, D. Lo, X. Xia, and J.-L. Sun, “Multi-factor duplicate question detection in Stack Overflow,” Journal of Computer Science and Technology, vol. 30, no. 5, pp. 981–997, Sep 2015.
- M. Ahasanuzzaman, M. Asaduzzaman, C. K. Roy, and K. A. Schneider, “Mining duplicate questions in Stack Overflow,” in 13th International Conference on Mining Software Repositories (MSR), 2016, pp. 402–412.
- Y. Mizobuchi and K. Takayama, “Two improvements to detect duplicates in Stack Overflow,” in 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 563–564.