implementing an algorithm that mixes data clustering and linear regression

Question

i have the following dataframe available in the link as a csv, it conveys information about stars.

more specifically - column ID represents arbitrary ID of sample. column z represents my target variable (response). the other columns represent the attributes available for each sample (predictors) and their corresponding measurement errors.

i used the following code to reduce the 11D data to 3 principal components and plotted the scatter of the data in principal space (with color indicating of the target variable Z)

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

#first we remove the target z and ID from the dataset and standredize it (mean=0 and std=1)
pca = PCA(n_components=3)
data_for_pca=data_clean.iloc[:,2:13]
data_for_pca=StandardScaler().fit_transform(data_for_pca)
#now we perform the pca and get the amount of variance, or relative information that each new component holds. 
principal_c=pca.fit_transform(data_for_pca)
pd.DataFrame(pca.explained_variance_ratio_).transpose()

import matplotlib.cm as cmx
from mpl_toolkits.mplot3d import Axes3D
def scatter3d(x,y,z, cs, colorsMap='jet'):
    cm = plt.get_cmap(colorsMap)
    cNorm = matplotlib.colors.Normalize(vmin=min(cs), vmax=max(cs))
    scalarMap = cmx.ScalarMappable(norm=cNorm, cmap=cm)
    fig6 = plt.figure()
    ax6 = Axes3D(fig6)
    ax6.scatter(x, y, z, c=scalarMap.to_rgba(cs))
    ax6.set_xlabel('pc1',fontweight='bold')
    ax6.set_ylabel('pc2',fontweight='bold')
    ax6.set_zlabel('pc3',fontweight='bold')

    scalarMap.set_array(cs)
    fig6.colorbar(scalarMap)

    plt.show()
scatter3d(principal_c[:,0],principal_c[:,1], principal_c[:,2],np.array(data.iloc[:,1]))

i attached the dataframe and code so anyone can reproduce and observe the 3d plot from all directions, my main intention with this question is to check if my intuitive analysis of the results is good and how to implement my idea on predicting z from the predictor data.

i see that the dots have somewhat of a smooth spherical gradient centered - roughly speaking - at (-2.5,-1,-0.25). perhaps i should implement some kind of gaussian kernel? if that's a good idea, how can i implement it?

another observation is that the data is slightly clustered in "plates" (as can be seen .

maybe i should perform seperate linear regression for each cluster/plate. and let the algorithm classify each point to a cluster/plate, then for each cluster/plate i can infer the target with the more sensitive linear regression coefficients.

if you think that might work, how should i implement it?

maybe there's a more rigorous approach to the further analysis of the PCA? (i mean that i'm kinda using my eyes to decide what is best to do but i'm sure there's a computational approach to the task).

would love to hear opinion and advice, this is a solo project that is part of an attempt to get a better understanding of data science after a BSc in physics.

thanks in advance!

Answer 1

A lot of the values from your data seem to be of the same size, or very small difference, also it seems it is important to deal with noise. The same problem is true for image compression. I think a good approach would be to use Haar-Wavelet Transform which compresses the information by a very huge amount and reduces the noisy data, like it does for image compression. After the transofrm, you need to apply quantization on the resulted data, here you can simply set all values to zero which have the smallest absolute value up until some bound, the remaining values you can also quntize into groups, if their difference is less than some bound. By doing this, you get rid of all noise. From here you would have to specify more precisely, what you want to achieve next. You want to predict z? For this task you can simply use Decision Tree learning methods, using the maximum information gain to decide on which attribute to make the first splits. Random Forests, which use a lot of different Decision Trees, where each one can be seen as an expert on some concrete task, should give very good results on this data, a good quantization method is very important for trees. You can also implement them all by yourself, without using any black box libraries, you learn much more from implementier it yourself. "McGrawHill_-_Machine_Learning_-Tom_Mitchell" Has very well explanations on those algorithms, explaining the mechanics behind all those black box like approaches.