I am trying to do a small project on my own to find out job openings using twitter data. I saved data using flume and converted it to .csv for analysis. My problem is i don't know how to classify tweets, whether it is a job vacancy or just some news on say machine-learning.I read online about neural networks and word2vec but i am not sure if it will solve my problem. Can anyone suggest some ways to do this based on tweet text and hashtags.I don't have training and test data, i just have tweets stored using flume.Also what kind of analysis will i be able to do with it.I am new to data science :/


1 Answer 1


You should be reading about text classification with supervised learning technique. You could chose a neural network such as CNN implementation and use it to your data. But you have to first prepare your data by cleaning it up and labelling it with respect to your desired category. For example,

  • You should collect the tweet data and label each tweet or set of words as job or news with respect to their original category.
  • Then you should use the prepared data and labels to train the chosen network.
  • Read about k fold cross validation and segregate data in such a way so that systematic iterative training and validation can be done and tweaks to the network could be made.
  • Finally once the training is completed you could use the model to test it on unseen test data.

These are common steps involved in text classification. It should be customized according to your need. You could google profanity filter which is mostly developed with a NLP type classifier. That would help you a lot.

  • $\begingroup$ I think RNNs would be better for NLP, instead of CNN. Tweets as sentences are sequences, a sequence model would be better. I agree with the rest. Upvoted. $\endgroup$
    – Ugur MULUK
    Commented Nov 22, 2018 at 11:40
  • 2
    $\begingroup$ Yes, RNN or even RCNN makes much more sense. $\endgroup$ Commented Nov 22, 2018 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.