2
$\begingroup$

I am trying to do some analysis on my data set with PCA so I can effectively cluster it with kmeans.

My preprocessed data is tokenized, filtered (stopwords, punctuation, etc.), POS tagged, and lemmatized

I create a data set of about 1.2 million tweet vectors (300 features each) by taking the averaged word vectors multiplied by their tfidf scores, like so:

# trained with same corpus as tfidf
# size=300, epochs=5, and min_count=10
tweet_w2v = Word2Vec.load('./models/tweet2vec_lemmatized_trained.model')

tweet_tfidf = TfidfVectorizer()
with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    tweet_tfidf.fit(infile)

tweet_tfidf_dict = dict(zip(tweet_tfidf.get_feature_names(), list(tweet_tfidf.idf_)))

tfidf_tweet_vectors = []

with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    for line in infile:
        word_vecs = []
        
        words = line.replace('\n', '').split(' ')
        
        if len(words) == 0:
            continue
            
        for word in words:
            try:
                word_vec = tweet_w2v.wv[word]
                word_weight = tweet_tfidf_dict[word]
                word_vecs.append(word_vec * word_weight)
            except KeyError:
                continue
                
        if len(word_vecs) != 0:
            tweet_vec = np.average(np.array(word_vecs), axis=0)
        else:
            continue
        tfidf_tweet_vectors.append(tweet_vec)

I also tried the above code with just average tweet vectors (no tfidf), and my problem still ended up happening.

I am starting to think that maybe my data set just isn't big enough or I am not training my word2vec model properly? I have somewhere around 100 million tweets I can use, but after filtering out retweets and only getting english language, it comes to around 1.3 million.

I'm not sure what's happening and what step I should take next. Any explanation is appreciated.

# Load in the data
df = pd.read_csv('./models/tfidf_weighted_tweet_vectors.csv')
df.drop(df.columns[0], axis=1, inplace=True)

# Standardize the data to have a mean of ~0 and a variance of 1
X_std = StandardScaler().fit_transform(df)

# Create a PCA instance: pca
pca = PCA(n_components=20)
principalComponents = pca.fit_transform(X_std)

# Plot the explained variances
features = range(pca.n_components_)
plt.bar(features, pca.explained_variance_ratio_, color='black')
plt.xlabel('PCA features')
plt.ylabel('variance %')
plt.xticks(features)

PCA feature variance

$\endgroup$
1
$\begingroup$

So the question is asking why the first two principal components of your encoded text data is encapsulating all of the variation in the data.

One potential issue could be the averaging over word vectors.

Suppose for a particular feature across word vectors for a particular post f, there could be an array of positive and negative values. When we then apply an average over f we could zero out the dimension and thus cause greater data sparsity, which could explain for what you are seeing (this zero value will exist regardless of whether you multiply this average with the td-idf or not). It could be the case that this sort of thing is happening across multiple dimensions in your text embeddings / feature vectors.

With this, you might need to think of another way of deriving a text embedding, maybe used Doc2Vec instead, which follows the same principles as Word2Vec, but instead derives document embeddings, which encapsulates the meaning of a section of text instead of word embeddings, which encapsulates the meaning of an individual word within a section of text.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I normalized my data instead of standardizing it and received this: i.stack.imgur.com/Mngvh.png would this make more sense? or should i still switch to doc2vec $\endgroup$ – Aiden Jul 11 at 21:24
  • $\begingroup$ This obviously looks a lot more promising than before. In terms of using doc2vec, maybe this could be something you experiment on to see which representation produces higher generalisation performance in the model. When you have an answer to this, report back, because I would really like to see the results. $\endgroup$ – shepan6 Jul 12 at 7:13
  • $\begingroup$ Will do. I am almost done with the project, I will report back with my findings. Currently testing at 77% f1 score with XGBoost on the Sentiment140 dataset, so im hopeful i can squeeze an 80 after some tuning and some other techniques im about to try. Can i ask you a couple questions about tfidf modelling in the meantime? $\endgroup$ – Aiden Jul 16 at 21:04
  • $\begingroup$ Brilliant, look forward to hearing more. If you have further questions, I recommend you post a separate question. $\endgroup$ – shepan6 Jul 23 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.