I got this as an assignment from a company recruiter and I've successfully scraped a dataset of about 650 movies with their 'Plot', 'Music' and 'Marketing' sections and gross. I've tried tfidf and count vectorizers and performed LSA/PCA to reduce the dimensions which originally are around 20k terms.
This is really boggling me, due to less instances(650) I guess the no. features should be around 100 or atleast < 600 but that is a drastic reduction of dimensions using PCA or LSA, is it the right thing to do?
Also my regression model is way off target or in some cases overfitting excessively. I've also tried normalizing and de-normalizing the target(gross) for no improvements. I know I should try NN and other models too but the results of linear regression should be atleast comparable too, right?
This makes me ponder if its even possible to model such a problem or am I doing something wrong with latter being more probable as I'm no expert in the field.
PLEASE HELP !