Skip to main content
added 493 characters in body
Source Link
Oleg
  • 190
  • 6

As you've mentioned you can use the function by sklearn, I don't see the problem using it (perhaps I'm missing something)

import pandas as pd
from sklearn.datasets import dump_svmlight_file

def df_to_libsvm(df: pd.DataFrame):
    x = df.drop('label', axis=1)
    y = df['label']
    dump_svmlight_file(X=x, y=y, f='libsvm.dat', zero_based=True)

Regarding the categorical features with 10^6 unique categories, you can use a simple embedding for it into a binary vector. One way to do it will be to map each username to a unique integer number. Then you can convert the number to binary representation; that way you'll have a simple embedding of approximately size 20 (2^20=1,048,576) i.e. this feature is represented by 20 binary features.

Of course if the usernames are all unique, they probably shouldn't be a feature (identical to id).

As you've mentioned you can use the function by sklearn, I don't see the problem using it (perhaps I'm missing something)

import pandas as pd
from sklearn.datasets import dump_svmlight_file

def df_to_libsvm(df: pd.DataFrame):
    x = df.drop('label', axis=1)
    y = df['label']
    dump_svmlight_file(X=x, y=y, f='libsvm.dat', zero_based=True)

As you've mentioned you can use the function by sklearn, I don't see the problem using it (perhaps I'm missing something)

import pandas as pd
from sklearn.datasets import dump_svmlight_file

def df_to_libsvm(df: pd.DataFrame):
    x = df.drop('label', axis=1)
    y = df['label']
    dump_svmlight_file(X=x, y=y, f='libsvm.dat', zero_based=True)

Regarding the categorical features with 10^6 unique categories, you can use a simple embedding for it into a binary vector. One way to do it will be to map each username to a unique integer number. Then you can convert the number to binary representation; that way you'll have a simple embedding of approximately size 20 (2^20=1,048,576) i.e. this feature is represented by 20 binary features.

Of course if the usernames are all unique, they probably shouldn't be a feature (identical to id).

Source Link
Oleg
  • 190
  • 6

As you've mentioned you can use the function by sklearn, I don't see the problem using it (perhaps I'm missing something)

import pandas as pd
from sklearn.datasets import dump_svmlight_file

def df_to_libsvm(df: pd.DataFrame):
    x = df.drop('label', axis=1)
    y = df['label']
    dump_svmlight_file(X=x, y=y, f='libsvm.dat', zero_based=True)