0
$\begingroup$

Using the two Python libraries GPyTorch and scikit-learn to perform Gaussian Process Regression (GPR) for a machine learning task, I have encountered a problem I failed to solve during the last days. I am using the same dataset, the same train-test split and the same kernel function in both cases (in GPyTorch it's the LinearKernel() and in sklearn it should be the DotProduct() if I got this right).
The problem is the following: Computing the mean absolute error in both versions of the code produces a much smaller error (a suspiciously small error) for the GPyTorch version than for the sklearn version. I initially thought that it depended on the way the dataset is splitted but varying the seed for the split for both cases made clear that it has nothing to do with the splitting, because the MAE for the sklearn version remains way higher on average than the MAE in GPyTorch. Below are the minimal working examples of both versions:

1. GPyTorch

# GPyTorch

X_data = []

for i in range(1, 123):
    X_array = np.loadtxt('/home/mp/ML/parameters/a1_l1_t2/' + str(i) + '.txt')
    X_data.append(X_array)

X_data = torch.tensor(X_data)

# Read target values

dataset = pd.read_csv('/home/mp/ML/parameters/targets/target_values.csv', skiprows=1,
                      names=('index', 'source', 'temperature', 'location'))

temperatures = dataset[['temperature']]

# Split dataset and convert temperatures to torch tensors

randomSeed = 42
train_X, test_X, train_temp, test_temp \
    = train_test_split(X_data, shifts, random_state=randomSeed, test_size=0.25, shuffle=True)

train_temp_tensor = torch.tensor(train_temp.values.astype(float).flatten())
test_temp_tensor = torch.tensor(test_temp.values.astype(float).flatten())

# Set up GPR

class ExactGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_X_tensor, train_shifts_tensor, likelihood):
        super(ExactGPModel, self).__init__(train_X_tensor, train_temp_tensor, likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.LinearKernel()

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)

        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

    def kernel_definition(self):
        return self.covar_module

# initialize likelihood and model
likelihood = gpytorch.likelihoods.GaussianLikelihood()
#likelihood.noise = 200.0 # set observation noise or standard deviation of the gaussian likelihood
model = ExactGPModel(train_X, train_temp_tensor, likelihood)
#model.covar_module.base_kernel.lengthscale = 1.0



model.eval()
likelihood.eval()

with torch.no_grad():
    observed_pred = likelihood(model(test_X))
    predictive_covariance = observed_pred.covariance_matrix


final_mae = mean_absolute_error(observed_pred, test_temp_tensor)
final_rmse = np.sqrt(mean_squared_error(observed_pred, test_temp_tensor))
#print(observed_pred.mean, test_shifts_tensor, observed_pred.covariance_matrix)

print(final_mae, final_rmse)

2. sklearn


X_data = []

for i in range(1, 123):
    X_array = np.loadtxt('/home/mp/ML/parameters/a1_l1_t2/' + str(i) + '.txt')
    X_data.append(X_array)

X_data = torch.tensor(X_data)

# Read target values

dataset = pd.read_csv('/home/mp/ML/parameters/targets/target_values.csv', skiprows=1,
                      names=('index', 'source', 'temperature', 'location'))

temperatures = dataset[['temperature']]

# Split dataset

randomSeed = 42
train_X, test_X, train_temp, test_temp \
    = train_test_split(X_data, temperatures, random_state=randomSeed, test_size=0.25, shuffle=True)
kernel = DotProduct()
gpr = GaussianProcessRegressor(kernel=kernel, random_state=randomSeed)
gpr.fit(train_X, train_temp)

# Make predictions
observed_pred, predictive_covariance = gpr.predict(test_X, return_cov=True)

# Calculate metrics
final_mae = mean_absolute_error(observed_pred, test_temp)
final_rmse = np.sqrt(mean_squared_error(observed_pred, test_temp))

# print(observed_pred, test_temp, predictive_covariance)

print(final_mae, final_rmse)

This is the first time I have encountered such a shift in performance when changing the library but using the very same ML method. I have tried to figure if there is something wrong with the way I am reading the data or if there are other bugs but I was not able to find some yet. I would appreciate any suggestions.

$\endgroup$

1 Answer 1

1
$\begingroup$

The solution to this problem is rather simple: Using different settings for the likelihood noise results in significant changes in the MAE values. Using the same likelihood noise in GPyTorch and sklearn results in very similar MAE values. I guess the default settings for this parameter are different for the two libraries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.