0
$\begingroup$

I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward.

I downloaded some Google Play Store data from here, and upon taking a peak, it appears to be primarily numerical, although the numbers stored are not of a recognized type.

dat.info() returns

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10841 entries, 0 to 10840
Data columns (total 13 columns):
App               10841 non-null object
Category          10841 non-null object
Rating            9367 non-null float64
Reviews           10841 non-null object
Size              10841 non-null object
Installs          10841 non-null object
Type              10840 non-null object
Price             10841 non-null object
Content Rating    10840 non-null object
Genres            10841 non-null object
Last Updated      10841 non-null object
Current Ver       10833 non-null object
Android Ver       10838 non-null object
dtypes: float64(1), object(12)
memory usage: 1.1+ MB

A lot of these are obviously numerical anyway when checking with head(), but some entries have something like "3.0M" in place of 3000000, and so I can't have any fun with them yet (I'm trying to first make histograms of them to explore). I wrote this quick loop to try and address this by converting them to a string, taking off the "M", then making them floats, but it doesn't seem to be working because the output still isn't numeric.

k=0
for i in dat["Reviews"]:
    #make the entry a string:
    j=str(i)
    #remove the Ms- I'm assuming these mean million
    if j[-1]=='M':
        j=j[:-1] #cut off last letter
        j=float(j)
        j=j*10**6
    dat[k,"Reviews"]=j
    k=k+1

There is likely a far easier way to do this that I'm missing, but the fact not even this works is strange. Does anyone have any ideas on how to deal with an unknown object type in a dataframe like this?

$\endgroup$
2
$\begingroup$

Try avoid looping when working pandas. Either create a function and apply it to your dataframe/series or you can use a lambda like I do below. I basically took your if/then logic with x*10**6 and applied it to a new column

import pandas as pd
df = pd.DataFrame({'Number':'10M 20M 10.5M 30M 100M 1000000 5000000'.split()})

    Number
0      10M
1      20M
2    10.5M
3      30M
4     100M
5  1000000
6  5000000

df['Num_Transformed'] = df['Number'].apply(lambda x: float(x[:-1])*10**6 if str(x).upper()[-1]=='M' else float(x))

    Number  Num_Transformed
0      10M       10000000.0
1      20M       20000000.0
2    10.5M       10500000.0
3      30M       30000000.0
4     100M      100000000.0
5  1000000        1000000.0
6  5000000        5000000.0
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.