I posted this question earlier, but I did not post the correct pictures and I was unable to edit it
I am currently doing a class project to use a machine learning algorithm (SVM or Regression) to deduce whether two sentences are paraphrases of one another. We were given training, development, and test datasets, and when training my model I am given an accuracy that appears to be constant no matter which features are added/removed.
I believe it is possibly due to the model not properly attaining the features, but my primary concern is that depending on what I use, it produces a different constant accuracy.
JupyterLab (Kaggle): 0.7421652421652422
import os import pandas as pd import numpy as np #read the training data file and drop NaN values columns = ['id', 'sentence 1', 'sentence 2', 'gold label'] training_data = '../input/mlmidterm/testwithlabel.txt' df = pd.read_csv(training_data, sep = '\t', names = columns) df['gold label'] = pd.to_numeric(df['gold label'], errors='coerce') df = df.dropna() df['gold label'] = df['gold label'].astype(int) training_data_dev = '../input/mlmidterm/devwithlabel.txt' df_dev = pd.read_csv(training_data_dev, sep = '\t', names = columns) df_dev['gold label'] = pd.to_numeric(df['gold label'], errors='coerce') df_dev = df_dev.dropna() df_dev['gold label'] = df_dev['gold label'].astype(int)
Code using training and development sets:
#development X_train = df_train.iloc[:,6:] y_train = df_train['gold label'].values X_dev = df_dev.iloc[:,6:] y_dev = df_dev['gold label'].values classifier = svm.SVC() classifier.fit(X_train, y_train) Y_pred = classifier.predict(X_dev) print(classifier.score(X_dev, y_dev))