I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. Nearly $50$ posts with tags are available for testing the results.
My approach: Remove stopwords, punctuations. Find the similarity(cosine) between word vector of each word of the post and the vectors of all the tags.
Problem: Context sensitive tags like fired(as in from a job) are shown for irrelevant post e.g. 'car back-fired) and only on average $3$ out $5$ most similar tags are relevant.
Gathered more posts($~200$ with average word length $40$) from other websites. Tried preprocessing of the posts:
stemming, created dictionary, made bow corpus then used Topic modeling (Latent Dirichlet Allocation)
Used gensim.models.LdaMulticore tried both BOW and tf-idf models but the topics produced had low confidence(of the order $0.07$) for the words in them. Found the relevant tags (using vector similarity) considering only top $10$ words of each topic. But the performance degraded even more as now at most $2$ tags were relevant.
Conditions:The tags are diverse and finding posts/text related to each tag is difficult. Tags are not be modified.
Does anyone have a better approach? Any help would be appreciated.