1
$\begingroup$

I have a data frame with 4 features and 1 target. The 4 features are 3 categorical and 1 numerical.

I created X which is a new data frame for the 3 categorical features. I use one hot label encoding but now it is a numpy array. Why?

Should i convert it back to a data frame? why not?

what is the best practice to merge X with my 1 numerical feature now ?

$\endgroup$
1
$\begingroup$

Should i convert it back to a data frame? why not?

If you have some specific requirements like, saving data in a file or want to perform some specific operations which can be run better on DataFrame, then its a good choice to convert it back to dataframe. Otherwise it should be ok to go with numpy array, even Scikit_learn different algo takes numpy array as an input.

what is the best practice to merge X with my 1 numerical feature now ?

I can share my experience and what exactly I did.

  1. Save separately and drop the categorical feature and move rest of the features in to numpy array.
  2. Convert categorical features in to OneHot encoding.
  3. Concatenate OneHot Encoding numpy array with rest of the features and consume this array for model training.
$\endgroup$
  • $\begingroup$ why does one hot label convert df to array while one label encoder does not ? $\endgroup$ – grim_reaper Jun 22 at 0:59
  • $\begingroup$ I tried with sample code, OneHotEncoder and LabelEncoder both methods are returning numpy array. Still if it's returning df(in some cases), for that you have to check the implementation of label encoder. $\endgroup$ – vipin bansal Jun 22 at 1:45
  • $\begingroup$ thanks everyone ! your all right my code had apply back to a DF. $\endgroup$ – grim_reaper Jun 26 at 20:08
  • $\begingroup$ Great and all the best. $\endgroup$ – vipin bansal Jun 27 at 17:56
1
$\begingroup$
  1. Convert to dataframe and merge
  2. Create a new df with array(encoded) and column with continuous value as inputs
  3. Perform inplace operation for one-hot-encoding in the dataframe
$\endgroup$
0
$\begingroup$

Should i convert it back to a data frame? why not?

  • Most of the sklearn transformers like LabelBinarizer outputs numpy array (it is one of design principal of scikit learn) so it is easier to work with ndarray in pipelines. Unless you absolutely need some features of panda it is good idea to work with ndarray

what is the best practice to merge X with my 1 numerical feature now ?

  • I would suggest to use Pipeline with FeatureUnion. FeatureUnion will run each pipeline in parallel and combine results of all the pipelines. Please look at example code below
    class DataFrameSelector(TransformerMixin, BaseEstimator):
        def __init__(self, include=None, exclude=None):
            self.include = include
            self.exclude = exclude

        def fit(self, X, y=None):
            return self

        def transform(self, X, y=None):
            """
            Returns only attributes listed in %include parameter if it is not None else return all attributes except listed
            in %exclude parameter
            """
            if self.include:
                return X[self.include].copy()
            else:
                return X.drop(self.exclude, axis=1)

    """Wrapper for LabelBinarizer as it only takes one parameter for 
    fit and transform methods and is not working with pipeline"""
    class LblBinarizer(TransformerMixin, BaseEstimator):
        def __init__(self):
            self.binarizer = LabelBinarizer()

        def fit(self, X, y=None):
            return self.binarizer.fit(X)

        def transform(self,X,y=None):
            return self.binarizer.transform(X)


    cat_pipeline = Pipeline(
    [
        ("select categorical features",  prepare_data.DataFrameSelector(include=["ocean_proximity"])),
        ("Binarize categorical features", LblBinarizer())
    ])

    num_pipeline = Pipeline(
    [
        ("select numerical features", prepare_data.DataFrameSelector(exclude=["ocean_proximity"]))
    ])

    full_pipeline = FeatureUnion(transformer_list=[
        ("num pipeline", num_pipeline),
        ("cat pipeline", cat_pipeline)
    ])

    prepared_data = full_pipeline.fit_transform(housing_features)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.