0
$\begingroup$

enter image description hereHi, I have been seeing this problem for quite some time. Whenever I tried vectorizing input text data though avgw2v vectorization technique. The size of vectorized data is less than the size of the input data. Is there any statistical reason behind this? In my case 100K is the size of the input and it gives 999,98 sized output I'm wondering what is causing this problem. Thanks in advance

Code:

listofsentences=[]
for sent in x_train:
    listofsentences.append(sent.split())

training_model = Word2Vec(sentences=listofsentences, workers=-1,min_count=5)
modelwords = list(training_model.wv.vocab)

std_avgw2v_x_train = []
for everysentence in tqdm(listofsentences):
    count = 0
    sentence = np.zeros(100)
    for everyword in everysentence:
        if everyword in modelwords:
            w2v = training_model.wv[everyword]
            count += 1
            sentence += w2v

    if count != 0:
        sentence/=count
        std_avgw2v_x_train.append(sentence)

len(std_avgw2v_x_train)
>99998

len(x_train)
>100000

EDIT1: I'd like to mention that I Just started learning ML, Its been 55 days since I started. Also, the same code gives our 100K output samples While I vectorize with TFIDFW2V

I have attached the image of the same. Kindly look into it

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I think the issue can be one of the two :

A . You have missing value in x_train

B . One of the values in x_train has no word that is there in modelwords.

In both the cases ,

if everyword in modelwords: w2v = training_model.wv[everyword] count += 1 sentence += w2v

condition doesn't get satisfied and you end up not addding any new value to sentence

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks! I'll check the same and update the question. BTW, The same input when I vectorize via TFIDFW2V Gives out 100K samples as output $\endgroup$
    – karthikeyan mg
    Jan 19, 2019 at 17:33
  • $\begingroup$ Since , tf-idf creates a vector based on the corpus only , it accounts for all the words unlike here where you are comparing if the word is present in the w2v vocab or not. $\endgroup$
    – Gyan Ranjan
    Jan 19, 2019 at 17:36
  • $\begingroup$ A: Thanks, To Check for any missing data, I tried converting this to a dataframe and sorted indices wtih attribute na_position='first' and I couldn't find any missing values interpreted as Na or any empty spaces and also I performed df.dropna(inplace=True) and the size of the dataset still remains (100000,1) $\endgroup$
    – karthikeyan mg
    Jan 19, 2019 at 17:54
  • $\begingroup$ B: I tried some words that are not in the vocabulary. For example, When I tried w2v = training_model.wv['hi'] and this gave me KeyError: "word 'hi' not in vocabulary" So this means, Your Suggestion B also is not the problem here right?? Correct me if I'm wrong. Also if you can suggest anyother robust methods for checking the problem, I would try that. Thanks $\endgroup$
    – karthikeyan mg
    Jan 19, 2019 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.