how to confirm a correlation between features

Question

i have created a matrix to show the correlation between different features in this data. i do not believe my findings. surely the blp (price to buy the car), should correlate with the rental price more than the number of seats does.

how can i confirm the correlation is correct or disprove it?

here is my code:

from matplotlib import pyplot
import pandas as pd
import numpy as np

from sklearn import *

def scale_this_data(data, col_names):

    print("scalling data now")
    new_df = pd.DataFrame(columns = col_names)
    for col in data.columns:
        wanted_col = False
        for the_col in col_names:
            if the_col == col:
                wanted_col = True
        if wanted_col == True:
            np_arr = data[col].values
            np_arr = np_arr.reshape(-1, 1)
            min_max_scaler = preprocessing.MinMaxScaler()
            
            np_arr = min_max_scaler.fit_transform(np_arr)
            
            #for n in range(len(data[col])):
            old = data[col].iloc[3]
            data[col] = np_arr
            print(str(data[col].iloc[3])+ "   this   became    this    = "+ str(data[col]))
    return data
    
Path = "new_ratebook.csv"
col_names = ['Net Rental2','Doors2', 'Seats2', 'BHP2', 'Eng CC2', 'CO22',  'blp2']
data = pd.read_csv(Path , dtype = str , index_col=False, low_memory=False)
data = scale_this_data(data, col_names)
data.to_csv("scaleddata.csv")
correlations = data.corr()
fig = pyplot.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(correlations, vmin=0, vmax=1)
fig.colorbar(cax)
ticks = np.arange(0,7,1)
ax.set_xticks(ticks)
ax.set_yticks(ticks)
ax.set_xticklabels(col_names)
ax.set_yticklabels(col_names)
pyplot.savefig('correlations.png')
pyplot.show()

    enter code here

Answer 1

I'm guessing that your problem is in this line:

cax = ax.matshow(correlations, vmin=0, vmax=1)

Correlations generally range from -1 to 1; because you set vmin=0, all negative correlations will show with the same deep blue hue of 0 correlation.

Either set vmin=-1 to see both positive and negative correlations or change the line

correlations = data.corr()

to

correlations = data.corr().abs()

to have the negative correlation use the same color scale of positive correlation of equal magnitude.

Answer 2

Investigation of the scatter plot between all pairs of variables is a good first pass approach to determine any correlation between variables. I'd suggest pairplot() from seaborn. Documentation here: https://seaborn.pydata.org/generated/seaborn.pairplot.html

Second order of business is principal components, and looking at the loadings of each variable on PC1 as to what is the main driver of cost is. This kind of analysis can be illuminating, but can be equally difficult to digest if there are multiple variables contributing equally. Looking at your correlation matrix (a standardized covariance matrix) there seems to be three "important" variables, #seats, eng CC2, CO22.