I'm relatively new to data science and I'm working with a large dataset. It has lots of rows and around 270 features after removing features with a lot of nan valuew and encoding categorical features. And when I run logistic regression using sklearn, my computer runs out of ram and crashes. How do I handle huge datasets like these?
I'm assuming you already have made feature selection so all your ~200 features are the ones that describe your target
So particularly for models that use SGD you can train your model in batches i.e adding new observations each time
In your case if using python you can make usage of SGDClassfiier with
loss = log to optimize logistic regression cost function, and use the method
You may need to do something like this:
chunksize = 5 clf = SGDClassifier(loss='log', penalty="l2", random_state = 42) for train in pd.read_csv("train.csv", chunksize=chunksize, iterator=True): X = train[features_columns] Y = train["target"] clf.partial_fit(X, Y)
Running models on all your available variables may not be the deal, since you'd have too much variables not directly linked to your result, and you might lose performance (you flood your model with unnecessary info, so it's lost and can't find direct and necessary one). You should start trying to produce the model starting with only a few variables you think show the most interest in your problem. Then, you'll have to make feature selection (Select the variables you want to put in your model) so you know which set would bring the best models. There are a lot of feature selection algorithms, you'll easily find some more info about them elsewhere.
Run a PCA on or LDA your data set. Here is some sample code to start with.
import numpy as np import pandas as pd import seaborn as sns import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer cancer = load_breast_cancer() df = pd.DataFrame(data=cancer.data, columns=cancer.feature_names) df.head() X = df.values X.shape from sklearn.preprocessing import StandardScaler # It is essential to perform feature scaling before running PCA if there is a significant difference in # the scale between the features of the dataset; for example, one feature ranges in values between 0 and 1 # and another between 100 and 1,000. PCA is very sensitive to the relative ranges of the original features. # We can apply z-score standardization to get all features into the same scale by using Scikit-learn # StandardScaler() class which is in the preprocessing submodule in Scikit-learn. scaler = StandardScaler() scaler.fit(X) X_scaled = scaler.transform(X) from sklearn.decomposition import PCA pca_30 = PCA(n_components=30, random_state=2020) pca_30.fit(X_scaled) X_pca_30 = pca_30.transform(X_scaled) print('variance explained by all 30 components = ', sum(pca_30.explained_variance_ratio_ * 100)) # The first component alone captures about 44.27% of the variability in the dataset and the second component alone captures about 18.97% of the variability in the dataset and so on. pca_30.explained_variance_ratio_ * 100 np.cumsum(pca_30.explained_variance_ratio_ * 100) plt.plot(np.cumsum(pca_30.explained_variance_ratio_)) plt.xlabel('number of components') plt.ylabel('explained variance') print(np.cumsum(pca_30.explained_variance_ratio_ * 100)) print(np.cumsum(pca_30.explained_variance_ratio_ * 100)) print(np.cumsum(pca_30.explained_variance_ratio_ * 100)) # You can see that the first 10 principal components keep about 95.1% of the variability in the # dataset while reducing 20 (30–10) features in the dataset. That’s great. The remaining 20 features # only contain less than 5% of the variability in data. # two principal components pca_2 = PCA(n_components=2, random_state=2020) pca_2.fit(X_scaled) X_pca_2 = pca_2.transform(X_scaled) plt.figure(figsize=(10,10)) sns.scatterplot(x=X_pca_2[:,0], y=X_pca_2[:,1], s=70, hue=cancer.target, palette=['blue','red']) plt.title('2D Scatterplot of 63% of Variability Captured') plt.xlabel('First Principal Component') plt.ylabel('Second Principal Component') # three principal components pca_3 = PCA(n_components=3, random_state=2020) pca_3.fit(X_scaled) X_pca_3 = pca_3.transform(X_scaled) from mpl_toolkits import mplot3d fig = plt.figure(figsize= (12,9)) ax = plt.axes(projection='3d') sctt = ax.scatter3D(X_pca_3[:,0], X_pca_3[:,1], X_pca_3[:,2], c=cancer.target, s=50, alpha=0.6) plt.title('3D Scatterplot of 72% of Variability Captured') plt.xlabel('First Principal Component') plt.ylabel('Second Principal Component') pca_95 = PCA(n_components=.95, random_state=2020) pca_95.fit(X_scaled) X_pca_95 = pca_95.transform(X_scaled) # This means that the algorithm has found 10 principal components to preserve 95% of the variability in # the data. The X_pca_95 array holds the values of all 10 principal components. X_pca_95.shape df_new = pd.DataFrame(X_pca_95, columns=['PC1','PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10']) df_new['label'] = cancer.target df_new