improving accuracy of classification

Question

I have data with 95 numeric variables and 5 categorical variables. My Y has 2 values. I built a decision tree and my accuracy was 81.8%. Then I created 3 new variables as follows. It improved accuracy to 84.3%

1. Normalize numeric variables and for training data, find mean vector for Y=1 and Y=0
2. for each data point, find euclidean distance from each mean vector - distance0 and distance1
3. third variable will be 0 if distance0 is <= distance1

I was wondering if there is any other new variables that i could create to improve the accuracy

I used a decision tree as it is fast to build and gives me indication whether a newly created variable is useful or not.

Please let me know if you have any thoughts

Answer 1

You could cluster your data with something like k-means and use the assigned cluster as a new feature.

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

data = load_iris()

X = pd.DataFrame(data['data'], columns=data['feature_names'])
y = data['target']

X_train, X_test, y_train, y_test = train_test_split(X.copy(), y, test_size=0.2)

km = KMeans(n_clusters=3)
km.fit(X_train)
training_dist = km.transform(X_train)

X_train['dist1'] = training_dist[:,0]
X_train['dist2'] = training_dist[:,1]
X_train['dist3'] = training_dist[:,2]

clf = DecisionTreeClassifier()
clf.fit(X_train, y_train)

# new data
test_dist = km.transform(X_test)

X_test['dist1'] = test_dist[:,0]
X_test['dist2'] = test_dist[:,1]
X_test['dist3'] = test_dist[:,2]

clf.predict(X_test)