I'm relatively new to data science / machine learning (yes, I know) and am experimenting with text analysis. I only want a relatively naive approach and am looking to know whether my approach is valid procedurally even if I may not get perfect results. I want to compare my model to a fulltext tf-idf search of specific skills/keywords against an index of resumes.

The task: Classify job descriptions, rather cluster them in an unsupervised way. Eventually I want to match resumes against them as a basic implementation of my idea. I was thinking that a naive implementation of word embeddings and kmeans would be fine. Basically, extract document_vectors from specific job descriptions, then cluster the job descriptions based on these vectors. This would create a ghetto "labeling" system so that I could then apply a classification algorithm Then I would use resume raw_text and extract a new vector, applying a simple classification it into one of these clusters (regression).

To summarize

1. Use a large database of job descriptions
2. Tokenize the text, and apply doc2vec and generate document vectors
3. Cluster the documents and group similar job descriptions
    NOTE: as to how many clusters to use, that will have to be tuned
4. Tokenize the raw_text of a resume
5. Use regression (or some technique) to find the closest cluster based on the resume vector compared to the cluster vector.

Now, I realize there are most likely some huge holes in the way I'm doing this (I'm using intuition based on no training). For example, my assumption that I can even match resume to job description is based on the idea that these are even comparable text structures - and how good the matching is is also dependent on how good the clustering of documents is.

I would like to know if this approach is good for something naive or whether I'm missing anything critical. I'm using nltk, scikit-learn, and gensim Word2Vec


What you say seems reasonable.

You can also try to create the document embeddings as you mention and use k-Nearest Neighbors to find the nearest job descriptions to a given resume based on some distance, without any clustering of job descriptions. There are some approximate implementations of k-Nearest Neighbors that work very fast like https://github.com/nmslib/nmslib.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.