# First two principal components explain 100% variance of data set with 300 features

I am trying to do some analysis on my data set with PCA so I can effectively cluster it with kmeans.

My preprocessed data is tokenized, filtered (stopwords, punctuation, etc.), POS tagged, and lemmatized

I create a data set of about 1.2 million tweet vectors (300 features each) by taking the averaged word vectors multiplied by their tfidf scores, like so:

# trained with same corpus as tfidf
# size=300, epochs=5, and min_count=10

tweet_tfidf = TfidfVectorizer()
with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
tweet_tfidf.fit(infile)

tweet_tfidf_dict = dict(zip(tweet_tfidf.get_feature_names(), list(tweet_tfidf.idf_)))

tfidf_tweet_vectors = []

with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
for line in infile:
word_vecs = []

words = line.replace('\n', '').split(' ')

if len(words) == 0:
continue

for word in words:
try:
word_vec = tweet_w2v.wv[word]
word_weight = tweet_tfidf_dict[word]
word_vecs.append(word_vec * word_weight)
except KeyError:
continue

if len(word_vecs) != 0:
tweet_vec = np.average(np.array(word_vecs), axis=0)
else:
continue
tfidf_tweet_vectors.append(tweet_vec)


I also tried the above code with just average tweet vectors (no tfidf), and my problem still ended up happening.

I am starting to think that maybe my data set just isn't big enough or I am not training my word2vec model properly? I have somewhere around 100 million tweets I can use, but after filtering out retweets and only getting english language, it comes to around 1.3 million.

I'm not sure what's happening and what step I should take next. Any explanation is appreciated.

# Load in the data
df.drop(df.columns[0], axis=1, inplace=True)

# Standardize the data to have a mean of ~0 and a variance of 1
X_std = StandardScaler().fit_transform(df)

# Create a PCA instance: pca
pca = PCA(n_components=20)
principalComponents = pca.fit_transform(X_std)

# Plot the explained variances
features = range(pca.n_components_)
plt.bar(features, pca.explained_variance_ratio_, color='black')
plt.xlabel('PCA features')
plt.ylabel('variance %')
plt.xticks(features)


So the question is asking why the first two principal components of your encoded text data is encapsulating all of the variation in the data.

One potential issue could be the averaging over word vectors.

Suppose for a particular feature across word vectors for a particular post f, there could be an array of positive and negative values. When we then apply an average over f we could zero out the dimension and thus cause greater data sparsity, which could explain for what you are seeing (this zero value will exist regardless of whether you multiply this average with the td-idf or not). It could be the case that this sort of thing is happening across multiple dimensions in your text embeddings / feature vectors.

With this, you might need to think of another way of deriving a text embedding, maybe used Doc2Vec instead, which follows the same principles as Word2Vec, but instead derives document embeddings, which encapsulates the meaning of a section of text instead of word embeddings, which encapsulates the meaning of an individual word within a section of text.

• I normalized my data instead of standardizing it and received this: i.stack.imgur.com/Mngvh.png would this make more sense? or should i still switch to doc2vec Jul 11 '20 at 21:24
• This obviously looks a lot more promising than before. In terms of using doc2vec, maybe this could be something you experiment on to see which representation produces higher generalisation performance in the model. When you have an answer to this, report back, because I would really like to see the results. Jul 12 '20 at 7:13
• Will do. I am almost done with the project, I will report back with my findings. Currently testing at 77% f1 score with XGBoost on the Sentiment140 dataset, so im hopeful i can squeeze an 80 after some tuning and some other techniques im about to try. Can i ask you a couple questions about tfidf modelling in the meantime? Jul 16 '20 at 21:04
• Brilliant, look forward to hearing more. If you have further questions, I recommend you post a separate question. Jul 23 '20 at 7:43
• Unfortunately because of the scope of the internship they wouldn't really allow me to take any information home. It wasn't anything crazy, but i did learn a lot and had a bit of fun. I ended up using a concatenated CNN and LSTM NN model created with keras at around 80% test accuracy. We put it on a website for further use (but again, they couldn't really let us host it on their HPC server and we couldn't figure out how to get our own backend hosting set up before the internship ended.) If you have any other questions though, id love to answer them Aug 18 '20 at 4:45