I'm trying to classify if a book is fiction/nonfiction based on title and summary.

This is 2 distinct types of information - is there a way to segment title and summary before feeding it to a model, rather than concatenating the information?

For example:

Title: "such a long journey"

Summary: "it is bombay in 1971, the year india went to..."

Label: "fiction" (where fiction =1)

Current procedure:

What I've been doing until now is concatenating the information, so the above becomes,

example = "such a long journey it is bombay in 1971, the year india went to..."
label = 1

Then the usual setup, something like,

X = lemmatize(X)
X_train, X_test, y_train, y_test = split_data(X,y)

vectorizer = TfidfVectorizer(...)
X_train = vectorizer.fit_transform(X_train)
X_test = vectorizer.transform(X_test)

classifier.fit(X_train, y_train)
y_predict = classifier.predict(X_test)

But feeding the data concatenated feels intuitively wrong. Is there a better way to do this?

If for some reason its possible with a library other than sklearn (keras, tensorflow) I'd be also open to hearing about that.


Going from,

X = ['two'],['two'],['four'],['two'],['four'],['four']]
y = ['human','human','dog','human','dog','dog']


X = [['two','hello'],['two','hello'],['four','bark'],['two','hi'],['four','bark'],['four','woof']]
y = ['human','human','dog','human','dog','dog']

causes errors to be thrown.

'list' object has no attribute 'lower' is X is a list, and 'numpy.ndarray' object has no attribute 'lower' if X is an array.

The error is thrown when I call,

X_train = vectorizer.fit_transform(X_train)

Is it possible to pass in a vector of features?

  • $\begingroup$ There's no reason to concatenate; pretty much any scikit-learn algorithm handles vector features. $\endgroup$ – Adrian Keister Sep 13 '18 at 16:05
  • $\begingroup$ @AdrianKeister I've updated my question. $\endgroup$ – tim_xyz Sep 21 '18 at 22:34
  • $\begingroup$ Try creating a model.with multiple inputs and then merge them after few layers $\endgroup$ – Aditya Sep 22 '18 at 12:34

You could just apply two independent vectorization steps on your input X (one vectorizer for description and another for summary) and then concatenate the obtained feature matrices into a single feature matrix.

Doing it this way you will have features such as description_"such", description_"long", ..., summary_"bombay", summary_"1971", ..., so any model you apply will be able to:

  1. Use all the features from the description and summary altogether
  2. Give a different weight to desctiption tokens and summary tokens

In neural network, it is possible to create a model with multiple inputs. Rather than concatenating, you can create separate input for the title and another for the summary where the input size of the title is smaller than that of the summary. In keras, you can do:


input_title = Input(shape=(MAX_TITLE_LENGTH,), name='input_title')
input_summary = Input(shape=(MAX_SUMMARY_LENGTH,), name='input_summary')

emb_title = Embedding(VOCAB_SIZE, EMBEDDING_DIM, input_length=MAX_TITLE_LENGTH)(input_title)
emb_summary = Embedding(VOCAB_SIZE, EMBEDDING_DIM, input_length=MAX_SUMMARY_LENGTH)(input_summary)

a = LSTM(128)(emb_title)
a = Dense(128, activation='relu')(a)

b = LSTM(512)(emb_summary)
b = Dense(128, activation='relu')(b)

z = Concatenate()([a, b])
z = Dense(128, activation='relu')(z)
output = Dense(CLASS_SIZE, activation='softmax')(z)

model = Model(inputs=[input_title , input_summary], outputs=output)

Also, regarding your error:

'list' object has no attribute 'lower'

Are you perhaps trying to convert the string in a list to lowercase? If so, check your code. It's trying to convert the list rather than the string, thus, giving this error.


The error you see is because the input you pass to vectorizer is a list of lists (or array of arrays), while, as stated in the TfidfVectorizer syntax you should pass a sequence of strings


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.