In my dataset I have a 'text' column and a 'followers' column containing lists of follower IDs, i.e. '1093777852477116417, 936194589043683328,...'. Some of the 'followers' values contain thousands of IDs.

I am preprocessing the data for LSTM, and I will do word embedding on the text column.

My question is, should I add the follower IDs to the word embedding of the text column, or should I hash the follower IDs and add an extra LSTM input layer for the IDs?

Thanks in advance!

  • 1
    $\begingroup$ What are you trying to accomplish? A classification task? Difficult to help without more context $\endgroup$ – Valentin Calomme Jan 28 at 12:56
  • $\begingroup$ I am trying to label articles as either fake or real, so yes its a classification task $\endgroup$ – kneki Jan 28 at 15:08

It depends…

The general rule of thumb is that there should be at least 40 occurrences of an item to train an embedding model to find a robust representation. If most follower IDs repeat then an embedding model can learn which ones co-occur. If follower IDs are sparse then hashing (which randomly assigns numbers) is a better choice.

Which method is better is an empirical question. You can create both models, benchmark, and then choose the data processing pipeline that is best for your task.

| improve this answer | |
  • $\begingroup$ Allright, I will try both methods and see what gives the best result :) Thank you $\endgroup$ – kneki Jan 29 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.