3
$\begingroup$

I am working on a business problem where I have a movie description dataset. In this dataset I've columns as - Movie title, Movie plot summary, Date of Release. Now based on this information and using machine learning I want to predict which category the movie falls into. For example The Conjuring should fall into Horror and Thriller i.e a multiclass classification problem. Now the problem is I don't have a label column besides the movie description and other info. Now I want my model to predict which categories a movie(unseen to model) should fall into. I have decided 5 labels that I want to consider - Horror, Thriller, Comedy, Romantic and Emotional. So, I want the dataset to look like this -


Conjuring| Description | Title | Horror,Thriller


The notebook| Description| Title | Romantic,Emotional


I believe if I want to proceed this problem as a classification problem then I have to think of some way to create labels to existing dataset by some script and logic. If not supervised then maybe if I can do clustering first and then based on where the data point lies I can do classification later on.

What I have tried ?

Once I decided what my 5 labels should be, I made 50 synonyms for each and then iterated the description of the movies and based on the number of occurrence of words I made frequency and based on majority of the occurrence I decided which category a movie should fall into. Very bad results from this approach.

I used K means clusters from the data and tried to extract information from the clusters. Could not get very meaningful information though.

To be very honest I am pretty clueless and just want a direction how to approach this problem.

$\endgroup$

2 Answers 2

3
$\begingroup$

Your specific problem can be solved by Googling.

Here is a solution that

  1. Searches the "imdb [year] [movie name]" in Google,
  2. Finds its IMDb address and fetches the IMDb page, and then
  3. Searches for the genres inside the IMDb page.

I have changed "romantic" to "romance", and "emotional" to "drama" to match the IMDb vocabulary.

from requests import get
import re

titles=["2013+Conjuring", "2004+The+notebook"]
genres = ['horror', 'thriller', 'comedy', 'romance', 'drama']
matched_genres = {}
for title in titles:
  query = "https://www.google.com/search?q=imdb+" + title
  print(query)
  search_result = get(query).text.lower()
  imdb_id = re.findall("https://www.imdb.com/title/(tt\d+)/", search_result)[0]
  imdb_address = "https://www.imdb.com/title/%s/" % imdb_id
  print(imdb_address)
  imdb_result = get(imdb_address).text.lower()
  matched_genres[title] = []
  for genre in genres:
    # find ">genre<" inside tags
    if imdb_result.find(">%s<" % genre) > -1:  
      matched_genres[title].append(genre)

print(matched_genres)

Output

https://www.google.com/search?q=imdb+2013+Conjuring
https://www.imdb.com/title/tt1457767/
https://www.google.com/search?q=imdb+2004+The+notebook
https://www.imdb.com/title/tt0332280/
{'2013+Conjuring': ['horror', 'thriller'], '2004+The+notebook': ['romance', 'drama']}

This solution could be improved by

  1. Querying movie titles in parallel,
  2. Directly querying IMDb API,
  3. Handling edge cases (for example when the first IMDb url is irrelevant or no IMDb page is found, etc.),

etc.

$\endgroup$
3
  • $\begingroup$ Thanks for your respose but I was looking more machine learning way to solve the problem as this is not the only problem the company I am working has, they have a dozen problem which quite are the same as this one but cannot be solved by googling. Thanks a ton for your time though. $\endgroup$
    – Pankaj
    Apr 24, 2019 at 9:52
  • 1
    $\begingroup$ @Pankaj My pleasure. I think If you place a separate question for each of those problems you may find more accurate and specialized solutions for each. $\endgroup$
    – Esmailian
    Apr 24, 2019 at 9:58
  • 1
    $\begingroup$ @Pankaj, the only general way to solve unlabeled multi-class classification is by doing the actual labeling yourself. Otherwise you are left with domain specific solutions or praying that your classes exist as neat clusters in your data. $\endgroup$ Apr 24, 2019 at 10:08
0
$\begingroup$

Since your problem is not precisely what you ask (seeing the comments you did), I might suggest that your problem has nothing to do with multi label classification, but more with summary extraction (summaries can be single keywords such as "science fiction". You could look for topic modelling algorithms to start. In any case, you'll need some task specific heuristics to achieve what you want, since your problem is not well formalized as a machine learning task.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.