I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, Embeddings is of categorical-set type.


Index  NumberOfFollowers                  UserName                    Embeddings        Target Variable

0        15                                name1                      [0.5 0.3 0.2]       0
1        4                                 name2                      [0.4 0.2 0.4]       1
2        8                                 name3                      [0.5 0.5 0.0]       0
3        10                                name1                      [0.1 0.0 0.9]       0
...      ...                               ....                       ...                 ..

I would like to convert this pandas data frame into the LibSVM input format.

Desired Output:

0 0:15 4:1 1:0.5 2:0.3 3:0.2
1 0:4 5:1 1:0.4 2:0.2 3:0.4
0 0:8 6:1 1:0.5 2:0.5 3:0.0
0 0:10 4:1 1:0.1 2:0.0 3:0.9

One solution I found was using:


It takes input as a NumPy array or a sparse matrix.

The UserName has a million unique values, so calling pd.get_dummies on this column and storing it as a dense NumPy array is not a solution and will not fit in the memory.

So, I know it may be done using sparse matrices, but, then I don't know how to convert the above data with mixed data types into a sparse matrix and then use sklearn.datasets.dump_svmlight_file.html.

In reality, I have many columns with mixed data types and I need to convert them into libSVM format. But, all the columns fall in one of the above three types.

Thanks in advance for any thoughts on how to solve the above problem.


As you've mentioned you can use the function by sklearn, I don't see the problem using it (perhaps I'm missing something)

import pandas as pd
from sklearn.datasets import dump_svmlight_file

def df_to_libsvm(df: pd.DataFrame):
    x = df.drop('label', axis=1)
    y = df['label']
    dump_svmlight_file(X=x, y=y, f='libsvm.dat', zero_based=True)

Regarding the categorical features with 10^6 unique categories, you can use a simple embedding for it into a binary vector. One way to do it will be to map each username to a unique integer number. Then you can convert the number to binary representation; that way you'll have a simple embedding of approximately size 20 (2^20=1,048,576) i.e. this feature is represented by 20 binary features.

Of course if the usernames are all unique, they probably shouldn't be a feature (identical to id).

| improve this answer | |
  • $\begingroup$ Hi Oleg, sure, all the categorical features can be converted into unique integers using label encoder. If I use one-hot-encoder after that, I get 10^6 length long bit vector (10^6 Pandas columns). If I use binary representation, sure, I get 20 bit vector. I am not sure how I can convert it into a sparse matrix after that. Perhaps you can explain over chat. As for whether usernames should be used as features, they are very often used as features in applications such as recommender systems. $\endgroup$ – Sumit Sidana Jul 16 at 16:41
  • $\begingroup$ By now, I figured for "not very massive data" solution is to use DictVectorizer. For very massive data, one has to avoid usage of pandas. $\endgroup$ – Sumit Sidana Jul 16 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.