I have a dataframe with a bunch of columns (words).


        arg1 predicate
    0   PERSON        be
    1       it      Pick
    2  details      Edit
    3    title   Display
    4    title   Display

I used a pretrained word2vec model to create a new df with all words replaced by vectors (1-D numpy arrays).

 get updated_df

    updated_df = df.applymap(lambda x: self.filterWords(x))
    def filterWords(self, x):
        model = gensim.models.KeyedVectors.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)
        if x in model.vocab:
            return model[x]
            return model['xxxxx']

updated_df print:

             arg1  \
        0  [0.16992188, -0.48632812, 0.080566406, 0.33593...   
        1  [0.084472656, -0.0003528595, 0.053222656, 0.09...   
        2  [0.06347656, -0.067871094, 0.07714844, -0.2197...   
        3  [0.06640625, -0.032714844, -0.060791016, -0.19...   
        4  [0.06640625, -0.032714844, -0.060791016, -0.19...   

        0  [-0.22851562, -0.088378906, 0.12792969, 0.1503...  
        1  [0.018676758, 0.28515625, 0.08886719, 0.213867...  
        2  [-0.032714844, 0.18066406, -0.140625, 0.115722...  
        3  [0.265625, -0.036865234, -0.17285156, -0.07128...  
        4  [0.265625, -0.036865234, -0.17285156, -0.07128...

I need to train a SVM(sklearn Linear SVC) with this data. When I pass the updated_df as X_Train, I get

clf.fit(updated_df, out_df.values.ravel())    
array = np.array(array, dtype=dtype, order=order, copy=copy)
ValueError: setting an array element with a sequence

What is the right way of passing this as the input data to the classifier? My y_train is fine. If I get a hash of the words to create the updated_df like below, it works fine.

updated_df = df.applymap(lambda x: hash(x))

But I need to pass the word2vec vectors to establish a relationship between the words. I am new to python/ML and appreciate the guidance.

Editing with the current status based on Theudbald's suggestion:

class ConcatVectorizer(object):
def __init__(self, word2vec):
    self.word2vec = word2vec
    # if a text is empty we should return a vector of zeros
    # with the same dimensionality as all the other vectors
    self.dim = len(word2vec.itervalues().next())
    print "self.dim = ", self.dim

def fit(self, X, y):
    print "entering concat embedding fit"
    print "fit X.shape = ", X.shape
    return self

def transform(self, X):
    print "entering concat embedding transform"
    print "transform X.shape = ", X.shape
    dictionary = {':': 'None', '?': 'None', '': 'None', ' ': 'None'}
    X = X.replace(to_replace=[':','?','',' '], value=['None','None','None','None'])
    X = X.fillna('None')
    print "X = ", X
    X_array = X.values
    print "X_array = ", X_array

    vectorized_array = np.array([
        np.concatenate([self.word2vec[w] for w in words if w in self.word2vec]
                or [np.zeros(self.dim)], axis=0)
        for words in X_array

    print "vectorized array", vectorized_array
    print "vectorized array.shape", vectorized_array.shape
    return vectorized_array

model = gensim.models.KeyedVectors.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)
    w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}
etree_w2v_concat = Pipeline([
    ("word2vec vectorizer", ConcatVectorizer(w2v)),
    ("extra trees", ExtraTreesClassifier(n_estimators=200))])

       def testWordEmbClassifier(self, pipe_obj):
    kb_fname = 'kb_data_3.csv'
    test_fname = 'kb_test_data_3.csv'
    kb_data = pd.read_csv(path + kb_fname, usecols=['arg1',
    kb_data_small = kb_data.iloc[0:5]
    kb_data_out = pd.read_csv(path + kb_fname, usecols=['output'])
    kb_data_out_small = kb_data_out.iloc[0:5]
    print kb_data_small
    pipe_obj.fit(kb_data_small, kb_data_out_small.values.ravel())
    print pipe_obj.predict(kb_data_small)
    self.wordemb_predictResult(pipe_obj, test_fname, report=True)
  • $\begingroup$ Is my question clear? Do you need more details? $\endgroup$ – iHavADoubt Mar 16 '18 at 21:56

In my opinion, scikit-learn raises an error because updated_df is composed of 2 features (columns) with list formats. Therefore, for a given observation x_i :

x_i = [arg1_i, predicate_i] = [[vector_arg1_i], [vector_predicate_i]].

Scikit-learn can't handle this format of input features.

There are mutiple ways to train a suprevised machine learning model after Word2Vec text processing. A common one is to sum or to average columns arg1 and predicate in order to have following observation x_i structure :

x_i = [(arg1_i + predicate_i) / 2] = [(vector_arg_i + vector_predicate_i) / 2]

More explanations and a gentle comparison between Word2Vec and CountVectorizer features engineering approaches for text classification :


| improve this answer | |
  • $\begingroup$ Thank you. This perfectly addresses my question. I will accept this but I have a followup. I have extracted the arg1 and predicate from a sentence using syntactic parsing and I do not want to lose the information of which word is a predicate and which is arg1. In this case, does it make logical sense to concatenate arg1 and predicate and other arguments into a single column instead of averaging them? I have updated the question with the code I am using for concatenation. $\endgroup$ – iHavADoubt Mar 20 '18 at 4:53
  • $\begingroup$ In my opinion, if you simply concatenate arg1 and predicate1, you will multiply by two your dimensionality + introducing multicollinearity in your input features. The purpose of Word2Vec algorithm in text classification is to reduce dimensionality in your input features. This is particularly usefull when you don't have a lot of observations (and text) in training set. In case you have enough observations (and text), you should maybe consider more standard approaches such as CountVectorizer or TfIdf Vectorizer. It would yield better results such as indicated in upper link. $\endgroup$ – Theudbald Mar 24 '18 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.