I have a metric (RevenueSoFar) that is a great predictor of my target FinalRevenue as you'd expect - it is a metric where we tend to get 90-95% of revenue so far on day 1 and then it can increase over the next 6 days. Therefore i'm also using DaysData (1,...,6) as a feature in the training set to attempt to get the model to learn about how confidence and revenue grows as we get closer to the final day.
In my pre-build analysis I have noticed correlations with several other metrics such as visits to our website, DOW of day1 etc. although not as strong as the correlation with RevenueSoFar. I've therefore ran Lasso Regression for feature importance on the features however my model has reduced every feature's coefficient to zero apart from RevenueSoFar (even the DaysData feature).
- Does this mean i should remove all of these from my model and only use the RevenueSoFar as the predictor?
- In the training data set there are a large proportion of products that have zero revenue and zero revenue so far (however the vast majority of these are for countries with low demand which is separated out as a feature in the training set). Could the zeros be skewing the model to think that RevenueSoFar and FinalRevenue mostly matches perfectly and therefore there is no need for other features?
Below is my code incase anybody is interested.
import numpy as np import pandas as pd from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split from sklearn.linear_model import Lasso from sklearn.feature_selection import SelectFromModel #import data DF = pd.read_csv("LearningDataSet.csv") DF_categorical = DF[["DOW","Market",”Country”]] DF_numerical = DF[["DaysData","D0Visits","D0MobVisits","D0DesVisits","D0Carts","D0MobCarts", "D0DesCarts", "RevenueSoFar"]] DF_target = DF[["RevenueFinal"]] #Scale numerical scaler = StandardScaler() DF_numerical_scaled = pd.DataFrame(scaler.fit_transform(DF_numerical)) #Dummy Categorical DF_categorical_dummy = pd.get_dummies(data=DF_categorical, drop_first=True) #Merge categorical and numerical DF_final = DF_categorical_dummy.merge(right=DF_numerical_scaled, how='left', left_index=True, right_index=True) #Split into training and test X_train, X_test, y_train, y_test = train_test_split(DF_final, DF_target, test_size=0.25, random_state=0) selection = SelectFromModel(estimator=Lasso()).fit(X_train,y_train) print(selection.get_support()) selected_feat = X_train.columns[(selection.get_support())] print(selected_feat) ```