First two principal components explain 100% variance of data set with 300 features

Question

I am trying to do some analysis on my data set with PCA so I can effectively cluster it with kmeans.

My preprocessed data is tokenized, filtered (stopwords, punctuation, etc.), POS tagged, and lemmatized

I create a data set of about 1.2 million tweet vectors (300 features each) by taking the averaged word vectors multiplied by their tfidf scores, like so:

# trained with same corpus as tfidf
# size=300, epochs=5, and min_count=10
tweet_w2v = Word2Vec.load('./models/tweet2vec_lemmatized_trained.model')

tweet_tfidf = TfidfVectorizer()
with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    tweet_tfidf.fit(infile)

tweet_tfidf_dict = dict(zip(tweet_tfidf.get_feature_names(), list(tweet_tfidf.idf_)))

tfidf_tweet_vectors = []

with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    for line in infile:
        word_vecs = []
        
        words = line.replace('\n', '').split(' ')
        
        if len(words) == 0:
            continue
            
        for word in words:
            try:
                word_vec = tweet_w2v.wv[word]
                word_weight = tweet_tfidf_dict[word]
                word_vecs.append(word_vec * word_weight)
            except KeyError:
                continue
                
        if len(word_vecs) != 0:
            tweet_vec = np.average(np.array(word_vecs), axis=0)
        else:
            continue
        tfidf_tweet_vectors.append(tweet_vec)

I also tried the above code with just average tweet vectors (no tfidf), and my problem still ended up happening.

I am starting to think that maybe my data set just isn't big enough or I am not training my word2vec model properly? I have somewhere around 100 million tweets I can use, but after filtering out retweets and only getting english language, it comes to around 1.3 million.

I'm not sure what's happening and what step I should take next. Any explanation is appreciated.

# Load in the data
df = pd.read_csv('./models/tfidf_weighted_tweet_vectors.csv')
df.drop(df.columns[0], axis=1, inplace=True)

# Standardize the data to have a mean of ~0 and a variance of 1
X_std = StandardScaler().fit_transform(df)

# Create a PCA instance: pca
pca = PCA(n_components=20)
principalComponents = pca.fit_transform(X_std)

# Plot the explained variances
features = range(pca.n_components_)
plt.bar(features, pca.explained_variance_ratio_, color='black')
plt.xlabel('PCA features')
plt.ylabel('variance %')
plt.xticks(features)

Answer 1

So the question is asking why the first two principal components of your encoded text data is encapsulating all of the variation in the data.

One potential issue could be the averaging over word vectors.

Suppose for a particular feature across word vectors for a particular post f, there could be an array of positive and negative values. When we then apply an average over f we could zero out the dimension and thus cause greater data sparsity, which could explain for what you are seeing (this zero value will exist regardless of whether you multiply this average with the td-idf or not). It could be the case that this sort of thing is happening across multiple dimensions in your text embeddings / feature vectors.

With this, you might need to think of another way of deriving a text embedding, maybe used Doc2Vec instead, which follows the same principles as Word2Vec, but instead derives document embeddings, which encapsulates the meaning of a section of text instead of word embeddings, which encapsulates the meaning of an individual word within a section of text.