0
$\begingroup$

please tell me, I'm trying to do a training competition on Kaggle, I want to build a heatmap based on a dataset, as one author of the guide did, but I complain that there are categorical signs, but the chatgpt says that pandas automatically ignores categorical signs and builds a heatmap based on numerical ones, and the author of the guide himself did not do any preliminary processing

hp_train.corr()

Error:

Cell In[17], line 1
----> 1 hp_train.corr()

File /opt/conda/lib/python3.10/site-packages/pandas/core/frame.py:10054, in DataFrame.corr(self, method, min_periods, numeric_only)
  10052 cols = data.columns
  10053 idx = cols.copy()
> 10054 mat = data.to_numpy(dtype=float, na_value=np.nan, copy=False)
  10056 if method == "pearson":
  10057     correl = libalgos.nancorr(mat, minp=min_periods)

File /opt/conda/lib/python3.10/site-packages/pandas/core/frame.py:1838, in DataFrame.to_numpy(self, dtype, copy, na_value)
   1836 if dtype is not None:
   1837     dtype = np.dtype(dtype)
-> 1838 result = self._mgr.as_array(dtype=dtype, copy=copy, na_value=na_value)
   1839 if result.dtype is not dtype:
   1840     result = np.array(result, dtype=dtype, copy=False)

File /opt/conda/lib/python3.10/site-packages/pandas/core/internals/managers.py:1732, in BlockManager.as_array(self, dtype, copy, na_value)
   1730         arr.flags.writeable = False
   1731 else:
-> 1732     arr = self._interleave(dtype=dtype, na_value=na_value)
   1733     # The underlying data was copied within _interleave, so no need
   1734     # to further copy if copy=True or setting na_value
   1736 if na_value is not lib.no_default:

File /opt/conda/lib/python3.10/site-packages/pandas/core/internals/managers.py:1794, in BlockManager._interleave(self, dtype, na_value)
   1792     else:
   1793         arr = blk.get_values(dtype)
-> 1794     result[rl.indexer] = arr
   1795     itemmask[rl.indexer] = 1
   1797 if not itemmask.all():

ValueError: could not convert string to float: 'RL'

https://www.kaggle.com/code/pmarcelino/comprehensive-data-exploration-with-python/notebook#2.-First-things-first:-analysing-'SalePrice'

$\endgroup$
2
  • 1
    $\begingroup$ Hi @lemintare welcome to the site. I think you might have modified the dataframe somehow, as "RL" is not in the original dataset for the kaggle competition. When I load the data from a .csv into a dataframe, and take the correlation matrix and create a heatmap, everything works fine. $\endgroup$ Oct 15, 2023 at 12:46
  • $\begingroup$ idk bro im tilted nothing is working $\endgroup$
    – lemintare
    Oct 16, 2023 at 0:27

3 Answers 3

0
$\begingroup$

You can check column dtypes using hp_train.dtypes. Subset the dataframe for only the desired columns before calling corr.

For example if you only want float64 cols

dtype_df = hp_train.dtypes
float_cols = dtype_df.iloc[(dtype_df=='float64').values].index
hp_train[float_cols].corr()
$\endgroup$
1
  • $\begingroup$ ye, but why didn't the author of the guide do this $\endgroup$
    – lemintare
    Oct 16, 2023 at 0:27
0
$\begingroup$

We have got 'tips' dataframe with 6 columns

total_bill  tip sex smoker  day time    size

Here total_bill,tip are of float ; size is int;rest are String

Suppose you have string column,float column,int column in your dataframe.For tips.corr() we can use only float column but int columns too (if first we convert them from int to float)

Here we can use total_bill(float),tip(float),size(but first we need to convert size from int to float) for making correlational matrix

First convert size from int to float

convert_dict = {'size': float }

tips=tips.astype(convert_dict)

If you have any other int columns other than size, than add that column name as key and float as value in above dictionary convert_dict

Now all required columns are in float ,now use below code

dtype_df = tips.dtypes

float_cols = dtype_df.iloc[(dtype_df=='float64').values].index

tips_tips=tips[float_cols].corr()

sns.heatmap(tips_tips)

using this 6 Bold line as code you can convert any dataframe (float columns+int (converted to float) columns) into correlational matrix to form heatmap

$\endgroup$
0
$\begingroup$

I've faced the same error while trying to create a heatmap. Following code solved my problem.

    hp_train.corr(numeric_only=True)
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.