Sparse matrix after vectorization giving size = 1

Question

I am working on a NLP problem https://www.kaggle.com/c/nlp-getting-started. I want to perform vectorization after train_test_split but when I do that, the resulting sparse matrix has size = 1 which cannot be right. Below is my code:

def clean_text(text):
    tokens = nltk.word_tokenize(text)    #tokenizing the words
    lower = [word.lower() for word in tokens]  #converting words to lowercase
    remove_stopwords = [word for word in lower if word not in set(stopwords.words('english'))]  
    remove_char = [word for word in remove_stopwords if word.isalpha()]
    lemm_text = [ps.stem(word) for word in remove_char]     #lemmatizing the words
    cleaned_data = " ".join([str(word) for word in lemm_text])
    return cleaned_data

x['clean_text']= x["text"].map(clean_text)

x.drop(['text'], axis = 1, inplace = True)

from sklearn.model_selection import train_test_split
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size = 0.2, random_state = 69, 
stratify = y)

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
tfidf = TfidfVectorizer()
train_x_vect = tfidf.fit_transform(train_x)
test_x1 = tfidf.transform(test_x)

pd.DataFrame.sparse.from_spmatrix(train_x_vect,
                              index=train_x.index,
                              columns=tfidf.get_feature_names())    

When I try to convert the sparse matrix (with size = 1) into a dataframe, it gives me error

ValueError: Index length mismatch: 4064 vs. 1

The dataframe x has size = 4064 and my sparse matrix has size = 1 which is why it is giving me error. Any help will be aprreciated!

Answer 1

You can convert the train_x to a list and pass that to your tfidf.fit_transform.

Approach 1

The updated code is:

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
tfidf = TfidfVectorizer()
train_x_vect = tfidf.fit_transform(train_x['text'].tolist())

Approach 2

Rather than converting this to a list you can simply pass the training data along with the column name like this train_x['text'] inside your fit_transform