35
$\begingroup$

I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of area of activity (theater, cloth stores, food ...) and other data.

I tried to create a new data frame and insert a column with the income of all kinds of stores that belong to the same category, and the returning data frame has only the first column filled and the rest is full of NaN's. The code that I tired: 

corr = pd.DataFrame()
for at in activity:
    stores.loc[stores['Activity']==at]['income']

I want to do so, so I can use .corr() to gave the correlation matrix between the category of stores.

After that, I would like to know how I can plot the matrix values (-1 to 1, since I want to use Pearson's correlation) with matplolib.

|improve this question
$\endgroup$
24
$\begingroup$

I suggest some sort of play on the following:

Using the UCI Abalone data for this example...

import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Read file into a Pandas dataframe
from pandas import DataFrame, read_csv
f = 'https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data'
df = read_csv(f)
df=df[0:10]
df

enter image description here

Correlation matrix plotting function:

# Correlation matric plotting function 

def correlation_matrix(df):
    from matplotlib import pyplot as plt
    from matplotlib import cm as cm

    fig = plt.figure()
    ax1 = fig.add_subplot(111)
    cmap = cm.get_cmap('jet', 30)
    cax = ax1.imshow(df.corr(), interpolation="nearest", cmap=cmap)
    ax1.grid(True)
    plt.title('Abalone Feature Correlation')
    labels=['Sex','Length','Diam','Height','Whole','Shucked','Viscera','Shell','Rings',]
    ax1.set_xticklabels(labels,fontsize=6)
    ax1.set_yticklabels(labels,fontsize=6)
    # Add colorbar, make sure to specify tick locations to match desired ticklabels
    fig.colorbar(cax, ticks=[.75,.8,.85,.90,.95,1])
    plt.show()

correlation_matrix(df)

enter image description here

Hope this helps!

|improve this answer
$\endgroup$
  • $\begingroup$ The second part was really very helpful, but I still have the first problem and I need to solve it before going to the second part $\endgroup$ – gdlm Mar 1 '16 at 18:10
  • $\begingroup$ Its very hard to understand what you want in the first part without some data. Can you add some data to illustrate the other piece that you have a question about. I believe this is trivially solved based on what you've mentioned. Just write 10 rows of the dataframe and the before and after of what you have and want. $\endgroup$ – AN6U5 Mar 2 '16 at 0:16
  • 1
    $\begingroup$ The line import numpy as np is not necessary, is it? $\endgroup$ – Martin Thoma Jan 23 '17 at 9:28
  • 1
    $\begingroup$ You don't use cbar, so why do you assign it? $\endgroup$ – Martin Thoma Jan 23 '17 at 9:29
  • 1
    $\begingroup$ @Martin Thoma - You are correct that numpy is not used. I was thinking that .corr() was a numpy function but it is pandas. I do use the colorbar, but you are correct that I didn't need to assign it to cbar. I've edited the reponse based on your comments. Thanks! $\endgroup$ – AN6U5 Jan 24 '17 at 0:35
29
$\begingroup$

Another alternative is to use the heatmap function in seaborn to plot the covariance. This example uses the Auto data set from the ISLR package in R (the same as in the example you showed).

import pandas.rpy.common as com
import seaborn as sns
%matplotlib inline

# load the R package ISLR
infert = com.importr("ISLR")

# load the Auto dataset
auto_df = com.load_data('Auto')

# calculate the correlation matrix
corr = auto_df.corr()

# plot the heatmap
sns.heatmap(corr, 
        xticklabels=corr.columns,
        yticklabels=corr.columns)

enter image description here

If you wanted to be even more fancy, you can use Pandas Style, for example:

cmap = cmap=sns.diverging_palette(5, 250, as_cmap=True)

def magnify():
    return [dict(selector="th",
                 props=[("font-size", "7pt")]),
            dict(selector="td",
                 props=[('padding', "0em 0em")]),
            dict(selector="th:hover",
                 props=[("font-size", "12pt")]),
            dict(selector="tr:hover td:hover",
                 props=[('max-width', '200px'),
                        ('font-size', '12pt')])
]

corr.style.background_gradient(cmap, axis=1)\
    .set_properties(**{'max-width': '80px', 'font-size': '10pt'})\
    .set_caption("Hover to magify")\
    .set_precision(2)\
    .set_table_styles(magnify())

enter image description here

|improve this answer
$\endgroup$
  • $\begingroup$ first time see using R package in python. A lot of R function can be used now. Great $\endgroup$ – Diansheng Apr 4 '18 at 6:18
  • $\begingroup$ Versions of Pandas > 0.19 don't contain the rpy module. You need to use the standalone project rpy2. See the warning from Pandas here. $\endgroup$ – n1k31t4 Aug 30 '18 at 12:38
7
$\begingroup$

Why not simply do this:

import seaborn as sns
import pandas as pd

data = pd.read_csv('Dataset.csv')

plt.figure(figsize=(40,40)) 
# play with the figsize until the plot is big enough to plot all the columns
# of your dataset, or the way you desire it to look like otherwise

sns.heatmap(data.corr())

You can change the color palette by using the cmap parameter:

sns.heatmap(data.corr(), cmap='BuGn')
|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.