1
$\begingroup$

Suppose I have a dataframe like this :

Hospital_name    State    Employees    ......
Fortis           Delhi    5000         ......
AIIMS            Delhi    1000000      ......
SuperSpeciality  Chennai  1000         ......

Now I want to use this dataframe to build a machine learning model for predictive analysis. For that I must convert the strings to float values. Also some of these columns in Hospital_name and State contains 'NAN' values. In such a case how should I prepare my data for building a model in keras?

$\endgroup$
  • $\begingroup$ How did you create the dataframe in the first place? Are the "NAN" values literally three character strings. If not, what are they? $\endgroup$ – Emre Jul 29 '16 at 17:52
  • $\begingroup$ The data is provided in form a CSV file by one of my client. And 'NAN' appeared as values in the columns for which no information was provided in the CSV. 'NAN' just represents that no info about that column is available. $\endgroup$ – enterML Jul 29 '16 at 17:56
4
$\begingroup$

To convert from string to float in pandas(assuming you want to convert Employees and you loaded the dataframe with df), you can use

df['Employees'].apply(lambda x:float(x))

You have not given your enough information about your input and expected output. So let us assume that hospital name or anything for that matter which is the input for your model is nan, you would like to remove it from the dataset because extracting features from 'nan' wouldn't make sense. Apart from that if they are just other peripheral features, then it might be alright. In that case, if you wish to convert them into blank then use df.replace(np.nan,' ', regex=True). Else if you wish to remove that frame, you can check for nan using this.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I want to convert the Hospital_name and State values (which are strings) to float because the model I am going to define is dependent on Hospital_name too. $\endgroup$ – enterML Jul 29 '16 at 18:09
  • $\begingroup$ You will have to use a mapping. If you want some nlp related advice, maybe you can use some word embeddings like word2vec and convert word to a vector. $\endgroup$ – Hima Varsha Jul 29 '16 at 18:19
2
$\begingroup$

The best way to deal with types is to specify it when ingesting the file:

pandas.read_csv(file_name, dtype={"Employees": float})

What you do with the missing data in Keras is up to you. What's your plan?

|improve this answer|||||
$\endgroup$
1
$\begingroup$

A more direct way of converting Employees to float.

df.Employees = df.Employees.astype(float)

You didn't specify what you wanted to do with NaN's, but you can replace them with a different value (int or string) using:

df = df.fillna(value_to_fill)

If you want to drop rows with NaN in it: df = df.dropna()

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I want to convert Hospital_names and States to float values. $\endgroup$ – enterML Jul 29 '16 at 18:08
  • $\begingroup$ There's no standard for converting a sting to a float. You'll have to define your own mapping, for example {'Delhi' : 0, 'Chennai' : 1}, and use [map] (pandas.pydata.org/pandas-docs/stable/generated/…) $\endgroup$ – user666 Jul 29 '16 at 18:12
1
$\begingroup$

I don't understand why map you would map the strings to floats... I would suggest using one hot encoding to categorize the strings with a Boolean 1 or 0.

In pandas this would be

import pandas as pd
s = pd.Series(list('abca'))
pd.get_dummies(s)

   a  b  c
0  1  0  0
1  0  1  0
2  0  0  1
3  1  0  0

You can also add pd.get_dummies(l, dummy_na=True) to deal with the NaN values.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.