# Work with large number of features for machine learning with pandas and sklearn

I'm relatively new to data science and I'm working with a large dataset. It has lots of rows and around 270 features after removing features with a lot of nan valuew and encoding categorical features. And when I run logistic regression using sklearn, my computer runs out of ram and crashes. How do I handle huge datasets like these?

• Can you post the dimension of your dataset? Sep 20 at 4:36

So particularly for models that use SGD you can train your model in batches i.e adding new observations each time

In your case if using python you can make usage of SGDClassfiier with loss = log to optimize logistic regression cost function, and use the method partial_fit.

You may need to do something like this:

chunksize = 5
clf = SGDClassifier(loss='log', penalty="l2", random_state = 42)

for train in pd.read_csv("train.csv", chunksize=chunksize, iterator=True):
X = train[features_columns]
Y = train["target"]
clf.partial_fit(X, Y)


Running models on all your available variables may not be the deal, since you'd have too much variables not directly linked to your result, and you might lose performance (you flood your model with unnecessary info, so it's lost and can't find direct and necessary one). You should start trying to produce the model starting with only a few variables you think show the most interest in your problem. Then, you'll have to make feature selection (Select the variables you want to put in your model) so you know which set would bring the best models. There are a lot of feature selection algorithms, you'll easily find some more info about them elsewhere.

Run a PCA on or LDA your data set. Here is some sample code to start with.

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = pd.DataFrame(data=cancer.data, columns=cancer.feature_names)

X = df.values
X.shape

from sklearn.preprocessing import StandardScaler

# It is essential to perform feature scaling before running PCA if there is a significant difference in
# the scale between the features of the dataset; for example, one feature ranges in values between 0 and 1
# and another between 100 and 1,000. PCA is very sensitive to the relative ranges of the original features.
# We can apply z-score standardization to get all features into the same scale by using Scikit-learn
# StandardScaler() class which is in the preprocessing submodule in Scikit-learn.
scaler = StandardScaler()
scaler.fit(X)
X_scaled = scaler.transform(X)

from sklearn.decomposition import PCA
pca_30 = PCA(n_components=30, random_state=2020)
pca_30.fit(X_scaled)
X_pca_30 = pca_30.transform(X_scaled)

print('variance explained by all 30 components = ', sum(pca_30.explained_variance_ratio_ * 100))

# The first component alone captures about 44.27% of the variability in the dataset and the second component alone captures about 18.97% of the variability in the dataset and so on.
pca_30.explained_variance_ratio_ * 100

np.cumsum(pca_30.explained_variance_ratio_ * 100)

plt.plot(np.cumsum(pca_30.explained_variance_ratio_))
plt.xlabel('number of components')
plt.ylabel('explained variance')

print(np.cumsum(pca_30.explained_variance_ratio_ * 100)[0])
print(np.cumsum(pca_30.explained_variance_ratio_ * 100)[1])
print(np.cumsum(pca_30.explained_variance_ratio_ * 100)[2])

# You can see that the first 10 principal components keep about 95.1% of the variability in the
# dataset while reducing 20 (30–10) features in the dataset. That’s great. The remaining 20 features
# only contain less than 5% of the variability in data.

# two principal components
pca_2 = PCA(n_components=2, random_state=2020)
pca_2.fit(X_scaled)
X_pca_2 = pca_2.transform(X_scaled)

plt.figure(figsize=(10,10))
sns.scatterplot(x=X_pca_2[:,0], y=X_pca_2[:,1], s=70, hue=cancer.target, palette=['blue','red'])
plt.title('2D Scatterplot of 63% of Variability Captured')
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')

# three principal components
pca_3 = PCA(n_components=3, random_state=2020)
pca_3.fit(X_scaled)
X_pca_3 = pca_3.transform(X_scaled)

from mpl_toolkits import mplot3d
fig = plt.figure(figsize= (12,9))
ax = plt.axes(projection='3d')
sctt = ax.scatter3D(X_pca_3[:,0], X_pca_3[:,1], X_pca_3[:,2], c=cancer.target, s=50, alpha=0.6)
plt.title('3D Scatterplot of 72% of Variability Captured')
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')

pca_95 = PCA(n_components=.95, random_state=2020)
pca_95.fit(X_scaled)
X_pca_95 = pca_95.transform(X_scaled)

# This means that the algorithm has found 10 principal components to preserve 95% of the variability in
# the data. The X_pca_95 array holds the values of all 10 principal components.
X_pca_95.shape

df_new = pd.DataFrame(X_pca_95, columns=['PC1','PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10'])
df_new['label'] = cancer.target
df_new