0
$\begingroup$

I am working on a NLP problem https://www.kaggle.com/c/nlp-getting-started. I want to perform vectorization after train_test_split but when I do that, the resulting sparse matrix has size = 1 which cannot be right. Below is my code:

def clean_text(text):
    tokens = nltk.word_tokenize(text)    #tokenizing the words
    lower = [word.lower() for word in tokens]  #converting words to lowercase
    remove_stopwords = [word for word in lower if word not in set(stopwords.words('english'))]  
    remove_char = [word for word in remove_stopwords if word.isalpha()]
    lemm_text = [ps.stem(word) for word in remove_char]     #lemmatizing the words
    cleaned_data = " ".join([str(word) for word in lemm_text])
    return cleaned_data

x['clean_text']= x["text"].map(clean_text)

x.drop(['text'], axis = 1, inplace = True)

from sklearn.model_selection import train_test_split
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size = 0.2, random_state = 69, 
stratify = y)

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
tfidf = TfidfVectorizer()
train_x_vect = tfidf.fit_transform(train_x)
test_x1 = tfidf.transform(test_x)

pd.DataFrame.sparse.from_spmatrix(train_x_vect,
                              index=train_x.index,
                              columns=tfidf.get_feature_names())    

When I try to convert the sparse matrix (with size = 1) into a dataframe, it gives me error

ValueError: Index length mismatch: 4064 vs. 1

The dataframe x has size = 4064 and my sparse matrix has size = 1 which is why it is giving me error. Any help will be aprreciated!

$\endgroup$
3
  • $\begingroup$ What is the size of train_x_vect ? If it not same as train_x, then there is the problem. If it is same, then where you are converting it to dataframe is the problem. $\endgroup$
    – Ankit Seth
    Aug 23, 2021 at 16:24
  • 1
    $\begingroup$ The size of train_x_vect is (1,1). $\endgroup$
    – spectre
    Aug 24, 2021 at 7:40
  • $\begingroup$ Check your train_x size before using tfidf. Given tfidf is working properly, train_x may not be what you think it is. $\endgroup$
    – Ankit Seth
    Aug 24, 2021 at 12:27

1 Answer 1

0
$\begingroup$

You can convert the train_x to a list and pass that to your tfidf.fit_transform.

Approach 1

The updated code is:

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
tfidf = TfidfVectorizer()
train_x_vect = tfidf.fit_transform(train_x['text'].tolist())

Approach 2

Rather than converting this to a list you can simply pass the training data along with the column name like this train_x['text'] inside your fit_transform

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.