1
$\begingroup$

I used to do some analysis on Excel but my company want to use Python to increase the analysis. I am novice on python and overwhelmed by all of this knowledge I have to learn :'(

So for my first project, I have imported my csv file and I had this error message "C:\Users\xxx\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py:2785: DtypeWarning: Columns (8,9,31,32,35,36,37,38,39,40) have mixed types. Specify dtype option on import or set low_memory=False.interactivity=interactivity, compiler=compiler, result=result) "

Sadly I have no clue about all of this so I looked on the web and guess what?it's still not clear for me. So my questions are

  1. What should I do to avoid this error message ?
  2. What are the best practices/ formatting when you work with dataset ?

Thanks

$\endgroup$
1
$\begingroup$

This warning is trying to let you know that some of the columns of your DataFrame failed to resolve to a single data type (at least from pandas' perspective, it didn't know what to interpret the columns as!). Since it's a warning, pandas is happy to give you the DataFrame after finishing the read_csv but it's not advisable to proceed without addressing your mixed type columns in some fashion.

1) You can avoid this warning by explicitly identifying the dtype. The parameter in read_csv is conveniently called dtype. The usage specified by the doc is either single data type (e.g. dtype = str) or a dictionary of columns and data types (e.g. dtype = {0:str, 1:int}). Some options for the data type are:

  • the pandas specific data types (object)
  • the built in Python types (str, int, float)
  • or numpy data types (numpy.int32, numpy.float64)

The nice thing about specifying per column data types is that Python will throw a hard error if something breaks that type convention. It's a great, immediate check that your data is in a format you expect and it's always nice for Python to break loudly. However, it's possible you want to just read all your data in and then fix it in the DataFrame. You generally can get away with specifying dtype=str and pandas will (usually) perform the read_csv without complaint.

2) Since this question is unfortunately best answered as "it depends," I would advise looking into data cleaning methodologies (since that's what this problem largely is). Some specific (but generic) advice is: think hard about how you want to deal with your missing/malformed data, be explicit about the transformations you'll apply to the data to handle those cases, and keep track of the fact you've made those changes.

I hope this was helpful!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.