2
$\begingroup$

My objective is to classify objects that all belong to a certain category, based on a textual description of these objects by humans. My problem is not specific to a certain category of objects, but for sake of clarity I am going to give examples as if the objects I wanted to classify were movies.

To be precise:

  • the description contains both a judgement of the object, and a more objective description of the various parts of the object. For example: “This movie has great lines, and the scenario is well-though. It counterbalances the poor actor performance. Still, overall I think it's a very good movie”. This both contains information about different aspects of the movie, and provides a subjective review.
  • what I want is:
    1. a score for each object (like a movie rating), based on how appreciated it is;
    2. for a given object, "similar" objects (ie. if you liked this movie, you might also enjoy these), based on similar "features" each object has. For instance, a movie which was also well-written might be considered "similar" to the former example.
  • I also have access to a pre-existing classification of these objects. For instance, a movie might be labeled "action/thriller". This classification is too broad for my purposes (ie. not all "action/thriller" movies are similar), but it might be a good start.

I have though that to solve my problem, I could use sentiment analysis to give each object a score, and that natural language processing coupled with a feature space could do the trick for classifying objects.

The point is that I am unsure on how to proceed, because I am new to machine learning, natural language processing, and data sciences in general. I have nonetheless a CS and mathematical background.

Could you provide some insight on where to start? Are there libraries that already provide this kind of features?

This is a repost of this question, since it was not focused enough, and this forum seems more appropriate. It has been rewritten.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Search for 'Huggingface transformers' for a package that already does a lot of this. $\endgroup$
    – Brady Gilg
    Commented Jul 26, 2021 at 18:49

1 Answer 1

Reset to default
2
$\begingroup$

Text vectorisation is a good way to have a reliable classification.

You have several libraries like doc2vec that you can use together with logistic regression or dimensional reduction technique like tSNE or UMAP. https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.html

On the other hand, you can also use libraries like BERT or TF-IDF:

https://pypi.org/project/bert-document-classification/

https://medium.com/swlh/text-classification-using-tf-idf-7404e75565b8

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the quick reply, but could you please provide a bit more details please, I am really a newbie in statistical and data science techniques. Also, I'll wait a bit to see if there are other leads proposed, if not I'll probably accept your answer. $\endgroup$
    – jthulhu
    Commented Jul 26, 2021 at 9:30
  • $\begingroup$ Also, I would like some details about how to get data to train models, if required, because some of the articles you mentioned require having a dataset that is already labeled, and it is not my case. $\endgroup$
    – jthulhu
    Commented Jul 26, 2021 at 9:50
  • $\begingroup$ I can't explain you every solution in detail, first because I don't know them all completely. All I can say as an intuition starting point, is that you need to transform your text data into probabilistic vectors, i.e. a matrix where words proximity are compared to each other. For instance "Cup" is 70% related to "tea" and 60% related to "coffee", but 0.5% to "dinosaur", etc. This is done through the learning process. Doc contents are processed the same way. Then you can classify them with many different techniques. Here is one quite well explained: youtube.com/watch?v=wvsE8jm1GzE $\endgroup$
    – Nicolas Martin
    Commented Jul 26, 2021 at 9:55
  • $\begingroup$ Indeed, that is broadly what I though of too (I explained it in more details in the other post, but it was quite long and tedious). The problem is I don't really know how to do that in practice. $\endgroup$
    – jthulhu
    Commented Jul 26, 2021 at 10:03
  • 1
    $\begingroup$ In that case, my best recommendation is to start by a simple recommendation system using the predifined labels and extraction of key words from text (using NLTK) that you can evaluate with a basic sentiment analysis score, and then classify with a dimensional reduction technique or a logistic regression. Then increase complexity with text vectorisation/BERT/IDF. $\endgroup$
    – Nicolas Martin
    Commented Jul 26, 2021 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.