I have a dataset with 6 categorical variables ( nominal variables), each of which have 10 categories. The dataset include 10 independent variables and 1 dependent variable. There are 500K observations and I need to use linear regression model. Do I need to one hot encoding all of the columns as this will generated huge new data given the number of column and rows, or is their another efficient way to handle this ?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Does this answer your question? Enocding of months for machine learning project $\endgroup$– dx2-66Commented Sep 29, 2022 at 6:50
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
500K does not sound like a huge dataset. This can be fitted with scikit-learn in a few seconds. You can use the example below to test how long this would run on your machine.
from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=int(5e5), n_features=100, n_informative=50)
model = LinearRegression()
model.fit(X, y)
