0
$\begingroup$

I have a csv file where a column of data is a 19 digit number of which the last three digits are always 163, this is specific to identify that this is the correct id. If i open the file in sublime, the column is correct, however, if I import it into pandas, these last three digits are different, and I can't work out what is happening. I have set the following option to prevent scientific notation as this should be a string anyway not a numerical number of any sort as they are id numbers not values and then imported the file. This is a file that I compiled so I know this data, it is for a database upload, but this issue is causing doubt as to what is in this file:

pd.set_option('display.float_format', lambda x: '%.0f' % x)
df_api_data = pd.read_csv('api_manipulated_data.csv')
df_api_data['stream_id']

Here is what one of these values actually is, compared to what pandas is importing:

This is the actual correct id: 
8533312285629196163
7593893060854955163
9075665575245639163

Pandas is importing:
8533312285629196288
7593893060854955008
9075665575245638656

What can be causing the last 3 or 4 digits to be ransomised and how can I make sure these values are getting read correctly from the csv?

for clarity, I got the line:

pd.set_option('display.float_format', lambda x: '%.0f' % x)

from another stackoverflow answer on how to suppress scientific notation, and I had to modify it to produce no decimal point, as this is for a database upload so needs to be correct and as a result I can't confirm it is correctly implimented.

$\endgroup$

1 Answer 1

1
$\begingroup$

I have found a solution. It appears that large integars are a bit of a problem, but the data can be imported as string data by using dtype as object as follows:

df_api_data = pd.read_csv('api_manipulated_data.csv', dtype=object)

the data is now displaying correctly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.