I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...) I have used pickle to Serialize my model and save it as .sav file. I would like to use it in REST API created with Flask. I am able to load it and all, but the problem is that I have used some feature engineering, feature scaling, and feature encoding using pd.get_dummies during the pre-processing phase before training my model. I assume the data that get's passed in the POST request of my REST API should also be pre-processed to be similar to the data that has been used to train the model (otherwise I would get the error : ValueError: Number of features of the model must match the input. ), so I created a function to do that. The problem lies in the encoding of the features. The new features created by encoding on the training dataset (pandas dataframe with multiple rows of data) and the input data (only one row) are different. How should I approach this in your opinion ? Here are a two images that illustrate my problem, This is an input example that has been pre-processed to get the predicted output by the model :

An input example that has been encoded

And here's an example from the training dataset that has been pre-processed :

enter image description here

Here's the code that I plan on using to transform the data that gets sent to my API :

def input_data_transform(inp):
    sc_X = StandardScaler()
    numericals =  pd.DataFrame(sc_X.fit_transform(inp[['carat','depth','x','y','z','table']]),columns=['carat','depth','x','y','z','table'],index=inp.index)
    inp[['carat','depth','x','y','z','table']] = numericals[['carat','depth','x','y','z','table']]
    inp =  pd.get_dummies(inp)
    return inp

In your opinion how should this be approached ? Is there a way to resolve this problem for most cases ? A simple and optimized solution ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.