Use a dataframe of word vectors as input feature for SVM

Question

I have a dataframe with a bunch of columns (words).

df

        arg1 predicate
    0   PERSON        be
    1       it      Pick
    2  details      Edit
    3    title   Display
    4    title   Display

I used a pretrained word2vec model to create a new df with all words replaced by vectors (1-D numpy arrays).

 get updated_df

    updated_df = df.applymap(lambda x: self.filterWords(x))
    def filterWords(self, x):
        model = gensim.models.KeyedVectors.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)
        if x in model.vocab:
            return model[x]
        else:
            return model['xxxxx']

updated_df print:

             arg1  \
        0  [0.16992188, -0.48632812, 0.080566406, 0.33593...   
        1  [0.084472656, -0.0003528595, 0.053222656, 0.09...   
        2  [0.06347656, -0.067871094, 0.07714844, -0.2197...   
        3  [0.06640625, -0.032714844, -0.060791016, -0.19...   
        4  [0.06640625, -0.032714844, -0.060791016, -0.19...   

                                                   predicate  
        0  [-0.22851562, -0.088378906, 0.12792969, 0.1503...  
        1  [0.018676758, 0.28515625, 0.08886719, 0.213867...  
        2  [-0.032714844, 0.18066406, -0.140625, 0.115722...  
        3  [0.265625, -0.036865234, -0.17285156, -0.07128...  
        4  [0.265625, -0.036865234, -0.17285156, -0.07128...

I need to train a SVM(sklearn Linear SVC) with this data. When I pass the updated_df as X_Train, I get

clf.fit(updated_df, out_df.values.ravel())    
array = np.array(array, dtype=dtype, order=order, copy=copy)
ValueError: setting an array element with a sequence

What is the right way of passing this as the input data to the classifier? My y_train is fine. If I get a hash of the words to create the updated_df like below, it works fine.

updated_df = df.applymap(lambda x: hash(x))

But I need to pass the word2vec vectors to establish a relationship between the words. I am new to python/ML and appreciate the guidance.

Editing with the current status based on Theudbald's suggestion:

class ConcatVectorizer(object):
def __init__(self, word2vec):
    self.word2vec = word2vec
    # if a text is empty we should return a vector of zeros
    # with the same dimensionality as all the other vectors
    self.dim = len(word2vec.itervalues().next())
    print "self.dim = ", self.dim

def fit(self, X, y):
    print "entering concat embedding fit"
    print "fit X.shape = ", X.shape
    return self

def transform(self, X):
    print "entering concat embedding transform"
    print "transform X.shape = ", X.shape
    dictionary = {':': 'None', '?': 'None', '': 'None', ' ': 'None'}
    X = X.replace(to_replace=[':','?','',' '], value=['None','None','None','None'])
    X = X.fillna('None')
    print "X = ", X
    X_array = X.values
    print "X_array = ", X_array

    vectorized_array = np.array([
        np.concatenate([self.word2vec[w] for w in words if w in self.word2vec]
                or [np.zeros(self.dim)], axis=0)
        for words in X_array
    ])

    print "vectorized array", vectorized_array
    print "vectorized array.shape", vectorized_array.shape
    return vectorized_array


model = gensim.models.KeyedVectors.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)
    w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}
etree_w2v_concat = Pipeline([
    ("word2vec vectorizer", ConcatVectorizer(w2v)),
    ("extra trees", ExtraTreesClassifier(n_estimators=200))])
rf.testWordEmbClassifier(etree_w2v_concat)

       def testWordEmbClassifier(self, pipe_obj):
    kb_fname = 'kb_data_3.csv'
    test_fname = 'kb_test_data_3.csv'
    kb_data = pd.read_csv(path + kb_fname, usecols=['arg1',
                                                        'feature_word_0',
                                                        'feature_word_1',
                                                        'feature_word_2',
                                                        'predicate'])
    kb_data_small = kb_data.iloc[0:5]
    kb_data_out = pd.read_csv(path + kb_fname, usecols=['output'])
    kb_data_out_small = kb_data_out.iloc[0:5]
    print kb_data_small
    pipe_obj.fit(kb_data_small, kb_data_out_small.values.ravel())
    print pipe_obj.predict(kb_data_small)
    self.wordemb_predictResult(pipe_obj, test_fname, report=True)

Answer 1

In my opinion, scikit-learn raises an error because updated_df is composed of 2 features (columns) with list formats. Therefore, for a given observation x_i :

x_i = [arg1_i, predicate_i] = [[vector_arg1_i], [vector_predicate_i]].

Scikit-learn can't handle this format of input features.

There are mutiple ways to train a suprevised machine learning model after Word2Vec text processing. A common one is to sum or to average columns arg1 and predicate in order to have following observation x_i structure :

x_i = [(arg1_i + predicate_i) / 2] = [(vector_arg_i + vector_predicate_i) / 2]

More explanations and a gentle comparison between Word2Vec and CountVectorizer features engineering approaches for text classification :

http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/