Can you use two different datasets as train and test sets with countVectorizer and test_train_split?

Question

So I managed to run my code on a combination of train data and validation data, but now I need to create a text file that contains the predictions for the test data and I just don't understand how. Is there any way to make the X_train work with train_data and X_test with test_data? I think that would solve my problem but I can't find how or if it is even possible.

train_data = np.genfromtxt('train_samples.txt', delimiter = '\t', dtype = None, encoding = 'utf-8', names = ('id', 'text'),
                               comments = None)

    train_labels = np.genfromtxt('train_labels.txt', delimiter='\t', dtype = None, names = ('id', 'label'))

    test_data = np.genfromtxt('test_samples.txt', delimiter = '\t', dtype = None, encoding = 'utf-8', names = ('id', 'text'),
                              comments = None)

    validation_data = np.genfromtxt('validation_samples.txt', delimiter='\t', dtype = None, encoding='utf-8',
                                    names = ('id', 'text'), comments = None)
    validation_labels = np.genfromtxt('validation_labels.txt', delimiter = '\t', dtype = None, names = ('id', 'label'))

    for x in range(len(train_data)):
        train_data[x][0] = train_labels[x][1]

    for x in range(len(validation_data)):
        validation_data[x][0] = validation_labels[x][1]

    train_data_text = np.append(train_data['text'], validation_data['text'])
    train_data_labels = np.append(train_data['id'], validation_data['id'])

    # show shape of training data
    cv = CountVectorizer()
    word_count_vector = cv.fit_transform(train_data_text)
    print(word_count_vector.shape)

    # train_data = np.concatenate((train_data, validation_data))
    X = cv.fit_transform(train_data_text).toarray()
    y = pd.get_dummies(train_data_labels)
    y = y.iloc[:, 1].values

    # Train Test Split
    from sklearn.model_selection import train_test_split

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

    # Training model using Naive bayes classifier
    from sklearn.naive_bayes import MultinomialNB

    results = MultinomialNB().fit(X_train, y_train)

    y_pred = results.predict(X_test)
    print(y_pred)

    from sklearn.metrics import accuracy_score

    # Evaluate accuracy
    print(accuracy_score(y_test, y_pred))

    classifier.fit(X_train, y_train)
    y_pred = classifier.predict(X_test)

Answer 1

You're doing a big mistake in your code, which is applying the vectoriser before the train/test splitting. The vectoriser should be fit only on the training dataset, then the learned counts should be applied to the test set. Instead you applied the vectoriser to the whole data which you then splitter into train and test.

# THIS IS OK
# train_data = np.concatenate((train_data, validation_data))
X = cv.fit_transform(train_data_text).toarray()
y = pd.get_dummies(train_data_labels)
y = y.iloc[:, 1].values

# NOT OK 
# Train Test Split
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

# CORRECT WAY
X_train = X  # already good as it is
y_train = y  # also good

# I suggest to use pandas to read the txt files
X_test = cv.transform(test_data['text']) # <-- APPLY VECTORIZER TO TEST DATA USING TRANSFORM ONLY

Then you can procede as you did in the rest of your code.