I am trying to do some analysis on my data set with PCA so I can effectively cluster it with kmeans.
My preprocessed data is tokenized, filtered (stopwords, punctuation, etc.), POS tagged, and lemmatized
I create a data set of about 1.2 million tweet vectors (300 features each) by taking the averaged word vectors multiplied by their tfidf scores, like so:
# trained with same corpus as tfidf # size=300, epochs=5, and min_count=10 tweet_w2v = Word2Vec.load('./models/tweet2vec_lemmatized_trained.model') tweet_tfidf = TfidfVectorizer() with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile: tweet_tfidf.fit(infile) tweet_tfidf_dict = dict(zip(tweet_tfidf.get_feature_names(), list(tweet_tfidf.idf_))) tfidf_tweet_vectors =  with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile: for line in infile: word_vecs =  words = line.replace('\n', '').split(' ') if len(words) == 0: continue for word in words: try: word_vec = tweet_w2v.wv[word] word_weight = tweet_tfidf_dict[word] word_vecs.append(word_vec * word_weight) except KeyError: continue if len(word_vecs) != 0: tweet_vec = np.average(np.array(word_vecs), axis=0) else: continue tfidf_tweet_vectors.append(tweet_vec)
I also tried the above code with just average tweet vectors (no tfidf), and my problem still ended up happening.
I am starting to think that maybe my data set just isn't big enough or I am not training my word2vec model properly? I have somewhere around 100 million tweets I can use, but after filtering out retweets and only getting english language, it comes to around 1.3 million.
I'm not sure what's happening and what step I should take next. Any explanation is appreciated.
# Load in the data df = pd.read_csv('./models/tfidf_weighted_tweet_vectors.csv') df.drop(df.columns, axis=1, inplace=True) # Standardize the data to have a mean of ~0 and a variance of 1 X_std = StandardScaler().fit_transform(df) # Create a PCA instance: pca pca = PCA(n_components=20) principalComponents = pca.fit_transform(X_std) # Plot the explained variances features = range(pca.n_components_) plt.bar(features, pca.explained_variance_ratio_, color='black') plt.xlabel('PCA features') plt.ylabel('variance %') plt.xticks(features)