I posted this question earlier, but I did not post the correct pictures and I was unable to edit it
I am currently doing a class project to use a machine learning algorithm (SVM or Regression) to deduce whether two sentences are paraphrases of one another. We were given training, development, and test datasets, and when training my model I am given an accuracy that appears to be constant no matter which features are added/removed.

I believe it is possibly due to the model not properly attaining the features, but my primary concern is that depending on what I use, it produces a different constant accuracy.

Vscode: 0.5021459227467812
JupyterLab (Kaggle): 0.7421652421652422

Training DataFrame:
enter image description here Development DataFrame:
enter image description here


import os
import pandas as pd
import numpy as np

#read the training data file and drop NaN values
columns = ['id', 'sentence 1', 'sentence 2', 'gold label']
training_data = '../input/mlmidterm/testwithlabel.txt'
df = pd.read_csv(training_data, sep = '\t', names = columns)
df['gold label'] = pd.to_numeric(df['gold label'], errors='coerce')
df = df.dropna()
df['gold label'] = df['gold label'].astype(int)

training_data_dev = '../input/mlmidterm/devwithlabel.txt'
df_dev = pd.read_csv(training_data_dev, sep = '\t', names = columns)
df_dev['gold label'] = pd.to_numeric(df['gold label'], errors='coerce')
df_dev = df_dev.dropna()
df_dev['gold label'] = df_dev['gold label'].astype(int)

Code using training and development sets:

X_train = df_train.iloc[:,6:]
y_train = df_train['gold label'].values
X_dev = df_dev.iloc[:,6:]
y_dev = df_dev['gold label'].values

classifier = svm.SVC()
classifier.fit(X_train, y_train)

Y_pred = classifier.predict(X_dev)

print(classifier.score(X_dev, y_dev))


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.