1
$\begingroup$

Suppose I have a data frame like this :

Hospital_name    State    Employees    ......
Fortis           Delhi    5000         ......
AIIMS            Delhi    1000000      ......
SuperSpeciality  Chennai  1000         ......

Now I want to use this data frame to build a machine learning model for predictive analysis. For that, I must convert the strings to float values. Also, some of these columns in Hospital_name and State contains 'NAN' values. In such a case, how should I prepare my data for building a model in Keras?

$\endgroup$
2
  • $\begingroup$ How did you create the dataframe in the first place? Are the "NAN" values literally three character strings. If not, what are they? $\endgroup$
    – Emre
    Jul 29 '16 at 17:52
  • $\begingroup$ The data is provided in form a CSV file by one of my client. And 'NAN' appeared as values in the columns for which no information was provided in the CSV. 'NAN' just represents that no info about that column is available. $\endgroup$
    – enterML
    Jul 29 '16 at 17:56
4
$\begingroup$

To convert from string to float in pandas (assuming you want to convert Employees and you loaded the data frame with df), you can use:

df['Employees'].apply(lambda x:float(x))

You have not given enough information about your input and expected output. So let us assume that hospital name or anything for that matter which is the input for your model is nan. You would like to remove it from the dataset because extracting features from 'nan' wouldn't make sense. Apart from that, if they are just other peripheral features, then it might be alright. In that case, if you wish to convert them into blank, then use:

df.replace(np.nan,' ', regex=True)`

Else, if you wish to remove that frame, you can check for nan using this.

$\endgroup$
2
  • $\begingroup$ I want to convert the Hospital_name and State values (which are strings) to float because the model I am going to define is dependent on Hospital_name too. $\endgroup$
    – enterML
    Jul 29 '16 at 18:09
  • $\begingroup$ You will have to use a mapping. If you want some nlp related advice, maybe you can use some word embeddings like word2vec and convert word to a vector. $\endgroup$ Jul 29 '16 at 18:19
2
$\begingroup$

The best way to deal with types is to specify it when ingesting the file:

pandas.read_csv(file_name, dtype={"Employees": float})

What you do with the missing data in Keras is up to you. You can elaborate further as it actually depends on your plan.

$\endgroup$
1
$\begingroup$

I don't understand why you would map the strings to floats. I would suggest using one hot encoding to categorize the strings with a Boolean 1 or 0.

In pandas this would be:

import pandas as pd
s = pd.Series(list('abca'))
pd.get_dummies(s)

   a  b  c
0  1  0  0
1  0  1  0
2  0  0  1
3  1  0  0

You can also add pd.get_dummies(l, dummy_na=True) to deal with the NaN values.

$\endgroup$
1
$\begingroup$

A more direct way of converting Employees to float.

df.Employees = df.Employees.astype(float)

You didn't specify what you wanted to do with NaNs, but you can replace them with a different value (int or string) using:

df = df.fillna(value_to_fill)

If you want to drop rows with NaN in it, use:

df = df.dropna()
$\endgroup$
2
  • $\begingroup$ I want to convert Hospital_names and States to float values. $\endgroup$
    – enterML
    Jul 29 '16 at 18:08
  • $\begingroup$ There's no standard for converting a sting to a float. You'll have to define your own mapping, for example {'Delhi' : 0, 'Chennai' : 1}, and use [map] (pandas.pydata.org/pandas-docs/stable/generated/…) $\endgroup$
    – user666
    Jul 29 '16 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.