I am using onehotencoding and RandomForestRegressor to predict scores of a set of soccer games. How can I use it into predict? I am sure I am doing it wrong at the moment as I am getting all predict values to be 1 (Probably because I am filling all NaN values as 1 for splitting and fitting)

What dataset should I pass as I am encoding a few columns and then transforming it?

My code is as below

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeRegressor
# Pandas needs you to define the column as date before its imported and then call the column and define as a date
# hence this step.
date_col = ['Date']
df = pd.read_csv(
    r'C:\Users\harsh\Documents\My Dream\Desktop\Machine Learning\Attempt1\Historical Data\Concat_Cleaned.csv'
    , parse_dates=date_col, skiprows=0, low_memory=False)

# Clean dataset by dropping null rows
data = df.dropna(axis=0)

# Column that you want to predict = y
y = data.Full_Time_Home_Goals

# Columns that are inputted into the model to make predictions (dependants), Cannot be column y
features = ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result']
# Create X
X = data[features]

# Split into validation and training data
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)
# Define and train OneHotEncoder to transform numerical data to a numeric array
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(train_X, train_y)

transformed_train_X = enc.transform(train_X)
transformed_val_X = enc.transform(val_X)

# Build a Random Forest model and train it on all of X and y.
# To improve accuracy, create a new Random Forest model which you will train on all training data
rf_model_on_full_data = RandomForestRegressor()

# Define columns we want to use for prediction
columns = ['Home_Team', 'Away_Team']
test_data = test_data[columns]
# Renaming Column Names to match with training dataset
test_data = test_data.rename({'Home_Team': 'HomeTeam', 'Away_Team': 'AwayTeam'}, axis=1)
# Adding NaN columns to dataset to match the training dataset
test_data['Full_Time_Result'] = np.nan
test_data['Full_Time_Away_Goals'] = np.nan
test_data['Full_Time_Home_Goals'] = np.nan
# Aligning dataframe to model defined
test_data_features = test_data[features]
# Filling all NA values as Encoder cannot handle nan values
df = test_data.fillna(1)

# Define Y for Fitting
Y = df

# We need nY as that would be the column used for splitting
ny = df.Full_Time_Home_Goals

# We need to encode and transform dataset so we have converted all nan to 1 and we are defining a new model as the
# val_x values are confusing, we will use n_
train_n_X, val_n_X, train_n_y, val_n_y = train_test_split(Y, ny, random_state=1)

# Since we have text again, we will need fitting and transforming the data
enc.fit(train_n_X, train_n_y)
transformed_train_n_X = enc.transform(train_n_X)
transformed_val_n_X = enc.transform(val_n_X)

# Fitting and then we will be using predict
rf_model_on_full_data.fit(transformed_train_n_X, train_n_y)

# Predicting. This step needs correction as predict should be on the new dataset and not just on on column.
test_preds = rf_model_on_full_data.predict(transformed_val_n_X)


What should go into predict() to get the outcome that I want?

Files used here

  • $\begingroup$ Hi, and welcome to datascience.stackexchange.com. Your code is very long, compared to a question (read some questions here for comparison), and most of it isn't important to the understanding of the problem... of course, sometimes you don't know in advance which part is important, and that's okay, but when your question is answered and your problem is solved, please revisit your question and edit it to make it shorter and more understandable for future readers. Good luck! $\endgroup$ – Itamar Mushkin Sep 21 '20 at 7:09

If I understand correctly, you're trying to predict the target variable 'Full_Time_Home_Goals' based on the four features ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result'] (the first two being being categorical and one-hot-encoded).

In your test set, you only receive (or only use) the first two, and fill the last two (which are, naturally, the more important ones) with a value of 1.

Now, let's imagine that your model decides that the teams aren't important, and makes the prediction based solely on the two latter variables... but in your test set, those all have the same value - so they'll get the same prediction.

The lesson here is that when you're training a model, do not use features that are (mostly or completely) missing from the test set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.