0
$\begingroup$

I am building a pipeline. I am downloading a dataset from an online ML repository and generating descriptive stats for it. The link for the dataset is https://archive.ics.uci.edu/dataset/45/heart+disease. I am using the processed.cleveland.data dataset.

I added the column names manually and I am converting numeric to string as required. I converted the DataFrame to a Numpy array to separate the predictor and target variables. After that, I am getting the list of numeric and categorical variables for the pipeline.

I develop the pipeline and then summarize the DataFrame. Why are there extra columns except the columns generated from OneHotEncoder? Ideally, my output would contain the same number of columns from the original dataset with the transformations (simple imputer) and the columns generated by OneHotEncoder for the categorical variables.

Could someone please let me know the issues?

    import pandas as pd
    import numpy as np
    import os
    from pathlib import Path

    from sklearn.compose import ColumnTransformer
    from sklearn.pipeline import Pipeline
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.preprocessing import MinMaxScaler
    from sklearn.impute import SimpleImputer

    url = ...
   names = ['age', 'sex', 'cp', 'trestbps', 'chol', 'fbs', 'restecg', 'thalach', 'exang',   'oldpeak', 'slope', 'ca', 'thal', 'num']

   def getData():
        return pd.read_csv(url, sep=',', names=names)

   input = getData()
   print(input.info())
   print(input.describe())

   array = input.values
   X = array[:,0:13]
   y = array[:,13]

   dataframe = pd.DataFrame.from_records(X)
   dataframe[[1, 2, 5, 6, 8]] = dataframe[[1, 2, 5, 6, 8]].astype(str)

   # print(dataframe.isnull())

   numerical = dataframe.select_dtypes(include=['int64', 'float64']).columns
   categorical = dataframe.select_dtypes(include=['object', 'bool']).columns

   print(numerical)
   print(categorical)

   t = [('cat0', SimpleImputer(strategy='most_frequent'), [1, 2, 5, 6, 8]), ('cat1',     OneHotEncoder(), categorical), ('num0', SimpleImputer(strategy='median'), numerical), ('num1', MinMaxScaler(), numerical)]
   column_transforms = ColumnTransformer(transformers=t)

   pipeline = Pipeline(steps=[('t', column_transforms)])
   result = pipeline.fit_transform(dataframe)

   print(type(pd.DataFrame.from_records(result)))
   print(pd.DataFrame.from_records(result).to_string())
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.