Am struggling to find the exact way to preprocess the text data with multiple text columns on the dataframe of features input and single output text label. Below is the sample dataset,

name    |   big_text_phrase     |   action      | action_type
test_name_1 | some bag of words |   action_1    | action_type_1
test_name_2 | some different bag of words   |   action_2    | action_type_2

When I was trying to do the text classification using just one feature big_text_phrase as input and output label as name it works fine and able to predict. Below is the model details with the single text feature input.

import numpy as np
import pandas as pd
import tensorflow as tf

from sklearn.preprocessing import LabelBinarizer, LabelEncoder
from sklearn.metrics import confusion_matrix

from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.preprocessing import text, sequence
from keras import utils
train_size = int(len(data) * .8)
train_posts = data['big_text_phrase'][:train_size]
train_tags = data['name'][:train_size]
test_posts = data['big_text_phrase'][train_size:]
test_tags = data['name'][train_size:]
max_words = 10000
tokenize = text.Tokenizer(num_words=max_words, char_level=False)
x_train = tokenize.texts_to_matrix(train_posts)
x_test = tokenize.texts_to_matrix(test_posts)
encoder = LabelEncoder()
y_train = encoder.transform(train_tags)
y_test = encoder.transform(test_tags)
num_classes = np.max(y_train) + 1
y_train = utils.to_categorical(y_train, num_classes)
y_test = utils.to_categorical(y_test, num_classes)
batch_size = 32
model = Sequential()
model.add(Dense(512, input_shape=(max_words,)))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(x_train, y_train, batch_size=batch_size, epochs=5, verbose=1, validation_split=0.1)
score = model.evaluate(x_test, y_test, batch_size=batch_size, verbose=1)

But am in full of confusion as how to implement the same with multiple input text features and single output text label, Googling doesn't get much details and heared that one hot encoding may suitable. So I tried with one hot encding where its kind of categorical method. Since I have huge bag of words in big_text_phrase column and the categorical columns are growing like anything.

Update 1:

I tried something different (might be stupid attempt) as below where the samples counts are going wrong,

train_posts = data[['big_text_phrase', 'action', 'action_type']][:train_size]
train_tags = data['name'][:train_size]
test_posts = data[['big_text_phrase', 'action', 'action_type']][train_size:]
test_tags = data['name'][train_size:]

print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)
print('y_train shape:', y_train.shape)
print('y_test shape:', y_test.shape)
x_train shape: (3, 10000)
x_test shape: (3, 10000)
y_train shape: (3587, 47)
y_test shape: (897, 47)

The model returned the error Input arrays should have the same number of samples as target arrays. Found 3 input samples and 3587 target samples..


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.