I posted this question earlier, but I did not post the correct pictures and I was unable to edit it
I am currently doing a class project to use a machine learning algorithm (SVM or Regression) to deduce whether two sentences are paraphrases of one another. We were given training, development, and test datasets, and when training my model I am given an accuracy that appears to be constant no matter which features are added/removed.
I believe it is possibly due to the model not properly attaining the features, but my primary concern is that depending on what I use, it produces a different constant accuracy.
Vscode: 0.5021459227467812
JupyterLab (Kaggle): 0.7421652421652422
Training DataFrame:
Development DataFrame:
Preprocessing:
import os
import pandas as pd
import numpy as np
#read the training data file and drop NaN values
columns = ['id', 'sentence 1', 'sentence 2', 'gold label']
training_data = '../input/mlmidterm/testwithlabel.txt'
df = pd.read_csv(training_data, sep = '\t', names = columns)
df['gold label'] = pd.to_numeric(df['gold label'], errors='coerce')
df = df.dropna()
df['gold label'] = df['gold label'].astype(int)
training_data_dev = '../input/mlmidterm/devwithlabel.txt'
df_dev = pd.read_csv(training_data_dev, sep = '\t', names = columns)
df_dev['gold label'] = pd.to_numeric(df['gold label'], errors='coerce')
df_dev = df_dev.dropna()
df_dev['gold label'] = df_dev['gold label'].astype(int)
Code using training and development sets:
#development
X_train = df_train.iloc[:,6:]
y_train = df_train['gold label'].values
X_dev = df_dev.iloc[:,6:]
y_dev = df_dev['gold label'].values
classifier = svm.SVC()
classifier.fit(X_train, y_train)
Y_pred = classifier.predict(X_dev)
print(classifier.score(X_dev, y_dev))
