I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. Nearly $50$ posts with tags are available for testing the results.

My approach: Remove stopwords, punctuations. Find the similarity(cosine) between word vector of each word of the post and the vectors of all the tags.

Problem: Context sensitive tags like fired(as in from a job) are shown for irrelevant post e.g. 'car back-fired) and only on average $3$ out $5$ most similar tags are relevant.

Gathered more posts($~200$ with average word length $40$) from other websites. Tried preprocessing of the posts: lemmatization and stemming, created dictionary, made bow corpus then used Topic modeling (Latent Dirichlet Allocation)

Used gensim.models.LdaMulticore tried both BOW and tf-idf models but the topics produced had low confidence(of the order $0.07$) for the words in them. Found the relevant tags (using vector similarity) considering only top $10$ words of each topic. But the performance degraded even more as now at most $2$ tags were relevant.

Conditions:The tags are diverse and finding posts/text related to each tag is difficult. Tags are not be modified.

Does anyone have a better approach? Any help would be appreciated.

  • $\begingroup$ Yes, vector similarity is a very non-representative way to find relevant tags. Instead, try using a many-to-many RNN in keras. Further you can also use the word2vec embeddings for better outputs. $\endgroup$ – Tirth Patel Jun 17 at 9:36
  • $\begingroup$ Thanks but the main problem is the scarcity of relevant posts. On training word2vec on my data, the relevant tags are missing because of the fact that the posts related to each tag are scarce and difficult to find. $\endgroup$ – Shivam Sharma Jun 18 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.