I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of documents to different labels. I got categorical, numerical and text data. The text data is already tokenised. So in feature 'text' I have single words. Meaning that, there is one word per row. E.g., 'downtown road', '5678-VB', '3552', 'product#3'. The data frame looks like the following:

label                               316511 non-null object
num_feature_1                       316511 non-null float16
num_feature_2                       316511 non-null float16
num_feature_3                       316511 non-null float16
num_feature_4                       316511 non-null float16
text                                316511 non-null object
length                              316511 non-null float16
start_with_capital                  316511 non-null uint8
is_upper                            316511 non-null uint8
ending_with_colon                   316511 non-null uint8

I already encode every-feature but the words (I got current word, previous word and next word as three different features). I have thought about using sklearn.preprocessing.LabelBinarizer for one-hot-encode the words. However, I got 316511 instances/words and 161880 unique words so I created a total of 161920 features. This increased the weight of the pandas data-frame up to 47GB. This is not feasible for my laptop/cloud instance so I had to find another way to handle this large dataset. I found that HashingVectorizer as another way to encode large dataset in a more optimised way But still, this uses frequency of words and for this problem case it does not make sense to apply frequency.

Anyone got some other idea on how to encode text/strings on this fashion to input sklearn-models?

****EDIT: Just to have a small view about 5 entrances with few features of the dataset:

|        | actual_label          |num_feature_1 | text        |   is_upper |num_feature_2  |
|  22337 | Sold-to-party Address | 0.989258     | 6B          |          1 | 0.0400085     | 
| 199218 | Material Description  | 0.956543     | 32X10       |          1 | 0.0999756     |
|  33579 | Sold-to-party Address | 0.989258     | Hatfield    |          0 | 0.160034      |
|   6506 | Sold-to-party Address | 0.956543     | 26          |          0 | 0.0400085     |
| 199066 | Material Description  | 0.93457      | scanalatura |          0 | 0.219971      |
  • $\begingroup$ Your setting is not very clear to me: every instance represents a word, and you have a text feature containing a word as well? Anyway, depending on what kind of classification you're after, a common and easy thing to do is to filter out the least frequent words, at least the ones which appear only once (since they won't help anyway). This will greatly decrease the number of feature thanks to the Zipf's law. $\endgroup$ – Erwan Jul 25 at 16:57
  • $\begingroup$ @Erwan please, see edit (I added some little example so you can check). The thing is that I got 161880 unique words showing up in the dataset. I can filter out those words showing only once but that's about half the dataset... $\endgroup$ – Javiss Jul 30 at 11:49
  • $\begingroup$ I still don't completely understand: so the goal is to predict a category for each word? In this case what happens if a word appears in several categories? It feels like maybe the problem is not defined in an optimal way. Anyway, the reason why you can remove low-frequency words is that they are very unlikely to appear again, so in the best case the model learns something useless and in the worst case it overfits, because with a low-frequency word it doesn't have enough cases to generalize. $\endgroup$ – Erwan Jul 30 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.