1
$\begingroup$

Suppose we have two data sets of movie reviews; one from IMDB and one from Rotten Tomatoes (RT). Each entry has a written-review and a score attached to it. The concatenated datasets might look like

Movie|Score|Review|Site

The Lion King|9.8|"This is one of the best movies ever!"|IMDB

The Lion King|4.5|"This move absolutely made my childhood"|RT

etc.

Our task is to predict scores from reviews. This is similar to the Kaggle IMDB sentiment analysis challenge, except we're trying to predict a continuous variable from two data sources. For a single data set, we might train a model in Keras with an embedding and lstm layer. In R:

model <- keras_model_sequential() %>%
   layer_embedding(input_dim = features, output_dim = 32) %>%
   layer_lstm(units = 32) %>%
   layer_dense(units = 1, activation = "linear")

where features is the number of words in our corpus.

How do we combine the IMDB and Rotten Tomatoes data sets to maximize our predictions?

My first thought is to include the site column as a categorical variable and then let the model learn how to best combine them. Would this cause a skew in the predictions?

We could also try multi-task learning, but most of these methods seem to be designed to predict repeated observations. Such as 5 different people reviewing the same 100 movies.

Some things to consider:

  1. IMDB and RT don't have all the same movies (i.e. not a repeated observation problem)
  2. There are words local to each dataset. i.e. the word "rotten" might be a frequent word on RT, but never appears in the IMDB dataset.
  3. The outcome measures are different. RT uses a 5 star scale, whereas IMDB uses a 10 star one.

How do we deal with this heterogeneity?

$\endgroup$
1
$\begingroup$

Following your considerations:

IMDB and RT don't have all the same movies

I would build a larger dataset of movies, aggregate all observations available, and run a model on that.


There are words local to each dataset. i.e. the word "rotten" might be a frequent word on RT, but never appears in the IMDB dataset.

I think this doesn't constitute a problem once you aggregate data in a larger dataset. At that point you can run some NLP method on the whole aggregated corpus. Let me know what you think.


The outcome measures are different. RT uses a 5 star scale, whereas IMDB uses a 10 star one. How do we deal with this heterogeneity?

You just need to uniform the scales on a given interval like [0, 1]. Then all votes could be compared.

$\endgroup$
  • $\begingroup$ Aggregating the datasets does seem like the simplest solution. Could we improve accuracy by training two separate models and letting them share information for certain layers? Something like cross-stitch networks? The movie example I provided is a simple case, but I'm more interested in the general question of combining heterogenous datasets that have a similar underlying structure, but also require their own fine tuning. $\endgroup$ – Peter DeWeirdt Jul 8 '19 at 14:43
  • $\begingroup$ I don't know if training two separate models would provide better results (it could be), but it would certainly be much more complicated and time consuming. Personally, I am all for aggregation, that would make your life easier and will let two separate pieces of information to complete/enhance each other. For example, NLP techniques could find associations between words that two separate models could never capture. $\endgroup$ – Leevo Jul 8 '19 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.