How to find the right regression model for my project

Question

I tried several things to find the best regression model with the best parameters but i can't go higher than 40% right predictions.

So i have 67741 rows in an excel file. the data looks like this after cleaning (4 columns only , is it enough ?) :

and the target rows like this :

I'll try to explain my process .

I went to this website https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html. From this graph the models that should suit my data are Lasso and ElasticNet, i got very bad score with this code :

classifiers = [
ElasticNetCV(cv=5, random_state=0,max_iter=40000), # i added the max_iter cause i got a warning saying that i should increase it
linear_model.Lasso(alpha=0.1,max_iter=40000)] # i added the max_iter cause i got a warning saying that i should increase it

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.33, random_state=42)

for clf, param in zip(classifiers, param_grid):
    name = clf.__class__.__name_
    clf.fit(X_train, y_train)
    print("=" * len(name))
    print("{}".format(name))

    print(clf.score(X_test, y_test))

The scores:

============
ElasticNetCV
0.002404878871672067
=====
Lasso
0.0066801704903023396

Then i tried several other models and i finally got something with:

BaggingRegressor(base_estimator=DecisionTreeRegressor(),
              max_features=0.5, max_samples=0.5)

Score :

================
BaggingRegressor
0.3460147571634854

So i used GridSearch and then cross validation score to get the best parameters but every time i launch it i got a different result something like that :

BaggingRegressor(base_estimator=DecisionTreeRegressor(), bootstrap=True, 
bootstrap_features=False,
              max_features=0.2, max_samples=0.7, n_estimators=10,
              n_jobs=None, oob_score=False, random_state=None, verbose=0,
              warm_start=False)

i used this that way :

param_grid = [{'max_samples': [0.1, 0.2, 0.5, 0.7, 1],
           'max_features': [0.1, 0.2, 0.5, 0.7, 1],
           'n_estimators': [5,10,15,20,25]}

def grid_search(clf, param_grid):
    grid_search = GridSearchCV(clf, param_grid, cv=5)
    grid_search.fit(X, y)
    print(grid_search.best_estimator_)
    print("=" * len(name))
    print(grid_search.best_params_)
    print("=" * len(name))
    print(grid_search.best_score_)

and cross validation like that : scores = cross_val_score(clf, X, y, cv=10)

Sorry if it's a bit long, with 67000 rows i can't get something more than 30% correct predictions ? what's the problem ?

Thank you

Answer 1

This is less a question of finding the best regression model and parameters than of feature engineering. You have 3 id columns and 1 date column with which you try to do regression. The only column which is somewhat metric is the date column and I don't expect it to contain enough information for a solid regression. Also, I assume you have many trucks which drive to many cities, which means after dummy-encoding, you will have a large amount of dummy-variables.

Feature Engineering

Feature engineering is a very broad topic which cannot be covered in one SE-post. However, here are two simple rules of thumb:

• Aggregate categorical columns which have many categories and use different ways to aggregate them (e.g. aggregate cities by region, but also aggregate them by size, and so on).
• Don't look at id-columns as features, but more as ids. Use them to join additional datasets. For example, from the truck-id, you can probably get more information about the truck (model, type, ...) and about the driver (experience, ...). Also, from the city-id, you can get a whole lot of information (distance between cities, size/population of the city, weather, ...).

Here are some naive suggestions for additional features:

• The month and the weekday (maybe the season as well)
• Time of the day in a few categories (afternoon, night, ...)
• If there are many cities, remove the city-id column and aggregate it by region, size, or whatever makes sense to you.
• Also, if there are many trucks, remove the truck-id column and aggregate it as well (e.g. by truck-type, delivery-type, drivers years of employment, or whatever data you can get).
• The estimated road distance between the current city and the city in which the truck was before
• The size of the road network of the destination-city / city of origin
• The weather at the respective time and place (rain, snow, sunny, ...)
• The delay which the truck has had on its last delivery

These are just some suggestions. Use whatever features make sense to you and use whatever additional data you can find. Maybe, run a decision tree on your new dataset and inspect the feature importances. That will give you a glimpse of what features are the most promising for further feature engineering.

Finding the best model

When you're done with feature engineering, then you can think about finding the best model. For this data size, I would suggest ElasticNet and Lasso, but I would definitely give a decision tree, random forest and xgboost a try. Also, think about dividing the target-variable into several categories and to do classification instead of regression. That could also be suitable for your use-case.

Answer 2

This answer is based on additional information regarding the data provided in a comment to the question:

[...] So it's about delivery The first and second are Id corresponding to cities, the 3rd is a date, and the last is an id corresponding to the truck that is changing (i sorted it that's why its the same in the first 3 columns). And the target Y is delay.

If you just add the time column in a unformated way to the regression, the method(s) will not be able to digest the information in a good way. Think about your problem. Delivery may be delayed because of trafic conditions (time, or the day of the week). I suggest that you encode time e.g. in hour of day, day of week, week of year (or so) and add all these (possibly one-hot encoded) variables/features to your model (together with the remaining features). You can use regularization (e.g. lasso) to "shrink" unnecessary features. This should give you an okay fit.

Essentially you need to really make sure you get as much information as possible out of the "date" column. In Python you can encode dates by predefined functions, e.g.

import datetime
Todays date datetime.datetime.today()
Day of week datetime.datetime.today().weekday()
Week of year datetime.datetime.today().isocalendar()[1]

Find an example of how to apply lasso here.