1
$\begingroup$

I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using keras.text_to_sequences function.

But now I want to use TfIdf with the LSTM can anyone tell me or share the code how to do it. Please also guide me if it is possible and good approach or not.

If you are wondering why I would like to do this there are two reasons:

  1. I want to see if this improves the results.
  2. Second, my professor asked me to perform Latent Dirichlet Allocation, and use same features for both of the tasks.
$\endgroup$
1
  • $\begingroup$ Did you know how to combine tf-idf with LSTM? $\endgroup$ May 7, 2020 at 19:17

2 Answers 2

1
$\begingroup$

Using TfIdf with LSTM is not a common approach, as LSTM networks are generally more suitable for handling sequential data like text sequences. TfIdf, on the other hand, is a technique commonly used for converting text data into fixed-length vectors based on the importance of words in a document.

However, if you want to experiment with combining both techniques, you can follow a two-step process:

  1. Convert Text to TfIdf Vectors: Use TfIdf vectorization to convert your text data into vectors. You can use the TfidfVectorizer from scikit-learn for this purpose. Here's a simple example:

    from sklearn.feature_extraction.text import TfidfVectorizer
    
    # Assuming 'texts' is a list of your email documents
    texts = ['email 1 text', 'email 2 text', ...]
    
    tfidf_vectorizer = TfidfVectorizer(max_features=5000)  # Adjust max_features as needed
    tfidf_matrix = tfidf_vectorizer.fit_transform(texts)
    
    # Convert the sparse matrix to a dense array
    tfidf_vectors = tfidf_matrix.toarray()
    

    Now, tfidf_vectors contains the TfIdf vectors for your documents.

  2. Combine TfIdf Vectors with LSTM: You can concatenate the TfIdf vectors with the LSTM output for each document. You'll need to reshape the TfIdf vectors to have the same sequence length as your LSTM output. Here's a high-level example:

    from keras.models import Sequential
    from keras.layers import LSTM, Dense, concatenate
    import numpy as np
    
    # Assuming lstm_model is your LSTM model
    lstm_model = ...
    
    # Assuming lstm_output is the output tensor of the LSTM layer
    lstm_output = ...
    
    # Reshape TfIdf vectors to match the LSTM output length
    tfidf_vectors_reshaped = np.repeat(tfidf_vectors[:, np.newaxis, :], lstm_output.shape[1], axis=1)
    
    # Build a simple model to concatenate LSTM output and TfIdf vectors
    combined_model = Sequential()
    combined_model.add(concatenate([lstm_output, tfidf_vectors_reshaped]))
    combined_model.add(Dense(1, activation='sigmoid'))  # Adjust output layer as needed
    
    # Compile and train the combined model as usual
    

    This is a basic example, and you may need to adapt it based on your specific model architecture and requirements.

However, it's important to note that combining these techniques might not always lead to improved results. LSTM networks are powerful in capturing sequential dependencies in data, and combining them with TfIdf vectors might introduce noise or redundant information. It's recommended to experiment carefully and evaluate the performance thoroughly.

Additionally, for Latent Dirichlet Allocation (LDA), it is a topic modeling technique that identifies topics present in a text corpus. You can use libraries like gensim in Python for LDA. The topics identified by LDA can be treated as additional features for your classification task. You can concatenate these LDA features with your LSTM output or TfIdf vectors to create a combined feature set for classification. Again, thorough experimentation and evaluation are crucial.

$\endgroup$
0
$\begingroup$

The goal of text_to_sequence + embedding in traditional LSTM is to transform text to word vectors.

If you already have the tfidf transformation, the idea is usually to get rid of the embedding layer in your LSTM when you are constructing the model, and directly connect the input (i.e., tfidf matrix) to the layer followed by the embedding layer.

Not sure if it's a good approach but that's for you to figure it out :P.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.