3
$\begingroup$

I'm quite new to Deep Learning and trying to solve the problem of Multi-Class, multi-label text classification using Deep Learning.

https://github.com/fchollet/keras/blob/master/examples/imdb_cnn_lstm.py . I've another dataset. int form of a csv file ("text","classifier"), on which i want to perform text classification task. I've tried a few ways to pass my training text to keras but couldn't so I'm stuck at this point. Can anyone suggest me how should I pass my "train.csv" and "test.csv" file to the X_train, y_train and X_test, y_test?

Typically stuck in this line.

(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)

'train.csv' has this format:

"Job Description:An ideal fitment would apply his/ her advanced analytics expertise at a cutting edge Industrial Analytics specialized Data Science organization; primarily, in any of the following areas- automotive/ energy/ oil & gas/ aerospace/ marine/ chemical. Experience in statistical modeling, predictive modeling, Random forests, Decision tree, Linear Regression, Correlation, Time- series. BE / MS/ PhD in Mechanical/ OR/ IE/ computer science/ EE/ chemical. Mentor/ Lead a small team of data scientist",Business Analytics

'test.csv' has same format that is "job_description","category"

$\endgroup$
  • $\begingroup$ Can you provide few lines(3-5 lines) of your train.csv and test.csv? $\endgroup$ – Icyblade Feb 14 '17 at 3:33
  • $\begingroup$ @Icyblade sure. Here are them.Updated in the question. $\endgroup$ – Kajal Puri Feb 14 '17 at 5:21
3
$\begingroup$

After I read the source code, I find out that keras.datasets.imdb.load_data doesn't actually load the plain text data and convert them into vector, it just loads the vector which has been converted before.

As for your problem, I assume you want to convert your job_description into vector. Maybe you can try sklearn.feature_extraction.text.CountVectorizer.

I'm no expert on NLP, but I've encountered problems (e.g. Two Sigma Connect: Rental Listing Inquiries) which need such technique.

In the meanwhile, there are other word2vec/embedding techniques you may try. Here is a tutorial from keras which shows a detailed example on word embedding.

References

https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py

http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html

https://www.kaggle.com/sudalairajkumar/two-sigma-connect-rental-listing-inquiries/xgb-starter-in-python

| improve this answer | |
$\endgroup$
1
$\begingroup$

Something like this:

nb_classes = 3 # the number of categories you have
x_train = []
y_train = []

with open('train.csv', 'r') as train_file:
    reader = csv.reader(train_file)
    for row in reader:
       sentence = row[0]
       category = row[1]

       x_train.append(sentence)
       y_train.append(category)

Y_train = np_utils.to_categorical(y_train, nb_classes)
X_train = ? # your choice of tokanization

You should do the same with your test dataset.

You should also change the loss to categorical_crossentropy.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.