I have a dataset with 6 categorical variables ( nominal variables), each of which have 10 categories. The dataset include 10 independent variables and 1 dependent variable. There are 500K observations and I need to use linear regression model. Do I need to one hot encoding all of the columns as this will generated huge new data given the number of column and rows, or is their another efficient way to handle this ?
500K does not sound like a huge dataset. This can be fitted with
scikit-learn in a few seconds. You can use the example below to test how long this would run on your machine.
from sklearn.linear_model import LinearRegression from sklearn.datasets import make_regression X, y = make_regression(n_samples=int(5e5), n_features=100, n_informative=50) model = LinearRegression() model.fit(X, y)