0
$\begingroup$

I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each label which is why I need some guidance trying to implement the code. How do I approach this issue?

Updating with code:

data = pd.read_csv("data/iris.data", sep=",", header=None)
data.columns = ["Sepal_length", "Sepal_width", "Petal_length", "Petal_width", "Species"]
iris_setosa = data.loc[data["Species"] == "Iris-setosa"]
iris_virginica = data.loc[data["Species"] == "Iris-virginica"]
iris_versicolor = data.loc[data["Species"] == "Iris-versicolor"]

train_setosa1 = iris_setosa.iloc[40, :]
test_setosa1 = iris_setosa.iloc[-10, :]
        
train_setosa2 = iris_setosa.iloc[-40, :]
test_setosa2 = iris_setosa.iloc[10, :]
        
train_setosa3 = iris_setosa.iloc[5:45, :]
test_setosa3 = iris_setosa.iloc[6:-4, :]
        
train_virginica1 = iris_virginica.iloc[40, :]
test_virginica1 = iris_virginica.iloc[-10, :]
        
train_virginica2 = iris_virginica.iloc[-40, :]
test_virginica2 = iris_virginica.iloc[10, :]
        
train_virginica3 = iris_virginica.iloc[5:45, :]
test_virginica3 = iris_virginica.iloc[6:-4, :]
        
train_versicolor1 = iris_versicolor.iloc[40, :]
test_versicolor1 = iris_versicolor.iloc[-10, :]
        
train_versicolor2 = iris_versicolor.iloc[-40, :]
test_versicolor2 = iris_versicolor.iloc[10, :]
        
train_versicolor3 = iris_versicolor.iloc[5:45, :]
test_versicolor3 = iris_versicolor.iloc[6:-4, :]
$\endgroup$
2
  • 3
    $\begingroup$ The best way for implement cross validation is with scikit-learn. maybe you need to add more information to your question. what have you tried? paste some code. $\endgroup$ Sep 23 at 23:53
  • $\begingroup$ I agree that scikit learn is the best option for implementing cross validation. However, I want build my own cross validation function. Currently, I have split my data in 3 species in iris dataset and then I create new data frames containing 40 records of each species for training my model and 10 records for testing on it @rubengavidia0x $\endgroup$
    – AGX301
    Sep 24 at 5:37
2
$\begingroup$

Which is preferable pandas or numpy?

It's totally up to you, if you really need to increase speed, go for numpy arrays, although at the same time the code will tend to get unwieldy and prone to errors, because you won't be able to keep track of feature columns easily. On the contrary, pandas will make your life easy when coding, so this choice is up to you.

What is the best way of creating folds for cross-validation?

Again there is no best algorithm or approach in machine learning that can be used in every case, it all depends on your preferences and needs. So I'll give you several options. I assume you are dealing with classification problem so I'll advise only on that,

  1. You can use sklearn.model_selection.StratifiedKFold function, as in this example,
kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=43)
for train_idx, test_idx in kf.split(X, y):
    X_train, X_valid = X.iloc[train_idx], X.iloc[test_idx]
    y_train, y_valid = y.iloc[train_idx], y.iloc[test_idx]

this option gives you more control over the code while debugging, and what's more important it stratifies the result according to y labels. Stratification here makes sure that distribution of values in y_train and y_valid will repeat the distribution in y proportionally, or simply put if one half of values in y were 1s and another half 0s, y_train and y_valid will have the same distribution of half being 1s and another half being 0s. With this function, you can use whichever scoring function you want.

  1. Predefine cross-validation folds beforehand,
kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=43)
for fold, (t_, v_) in enumerate(kf.split(X, y)):
    
    # where X = df.drop('target'), y = df.target
    df.loc[v_, 'fold'] = fold

these folds can be used as following:

folds = set(df['fold'].astype(int).unique())
for fold in folds:
    
    df_train, df_valid = df[~(df.fold==fold)], df[(df.fold==fold)]
    X_train, y_train = df_train.drop(['target', 'fold'], axis=1), df_train.target
    X_valid, y_valid = df_valid.drop(['target', 'fold'], axis=1), df_valid.target

this approach apparently may increase the speed of computations, as there is no need to compute folds at every step using complex algorithm, instead all we do is select rows with the current fold. Additionally, this approach makes reproducibility of the result possible, especially if you save the dataset with predefined folds to csv, and so that other people will get the same cross-validation folds.
Another important thing, this approach makes it possible to compute different folds in parallel. This may come in handy, with neural nets used in deep learning.
Here, you can use whichever scoring function you want as well.

  1. Use sklearn.model_selection.cross_val_score function as it was mentioned in the previous answer.

This function is not exactly what you want, because it evaluates a score by cross-validation, while you were asking about ways of cross validation itself. Although I could hardly imagine what you can use cross-validation for apart from evaluating a score. Anyway, let's address pros and cons of this function.
Pros: simple to use, stratification is possible.
Cons: apparently, this is a black-box function, you can't debug it, unless on local machine, even there it is hard to do that. It is slower than second approach for mentioned reasons. And it is restricted to scoring methods listed here, in case you want use different scoring.

UPDATE:

In your newly added code, you're slicing the datasets wrong,

train_setosa1 = iris_setosa.iloc[40, :]
test_setosa1 = iris_setosa.iloc[-10, :]

you are selecting only one row for each dataset, I suspect you were trying to slice it so that first 40 rows were included in train_setosa1 and the rest in test_setosa1, here's how you do that,

train_setosa1 = iris_setosa.iloc[:40]
test_setosa1 = iris_setosa.iloc[40:]

But even in this case it is not proper cross-validation, because the whole point of that is using randomness of selection, and usually machines are better at this than people are. Here's equivalent implementation of your code,

iris_species = ["Iris-setosa", "Iris-virginica", "Iris-versicolor"]

kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=43)

for species in iris_species:
    species_df = data[data["Species"] == species]
    X = species_df.drop('Species', axis=1)
    y = species_df.Species
    for train_idx, valid_idx in kf.split(X, y):
        
        X_train, X_valid = X.iloc[train_idx], X.iloc[valid_idx]
        y_train, y_valid = y.iloc[train_idx], y.iloc[valid_idx]
        
        # do whatever it is you want with training and validation data here
        pass

Cheers,

$\endgroup$
1
  • 1
    $\begingroup$ Thanks! This answer gave a lot of information I can use. Appreciate the time taken for this! $\endgroup$
    – AGX301
    Oct 4 at 4:30
3
$\begingroup$

What is the best way to implement this?

As @rubengavidia0x said, the best way to implement Cross-Validation is with Scikit-Learn. Scikit-Learn handles all of the complexities for you and all you need to do is pass your data and model to the cross_val_score() function.

Following is how to do this -

from sklearn import datasets
from sklearn import svm
from sklearn.model_selection import cross_val_score

# Iris dataset
X, y = datasets.load_iris(return_X_y=True)
print(X.shape, y.shape)

# SVM Classifier
clf = svm.SVC(kernel='linear', C=1, random_state=42)

# The following cross_val_score will carry out 5 fold cross validation and return scores for all 5 experiments
scores = cross_val_score(clf, X, y, cv=5)

print(scores)
print("%0.2f accuracy with a standard deviation of %0.2f" % (scores.mean(), scores.std()))

Following is the output -

(150, 4) (150,)
[0.96666667 1.         0.96666667 0.96666667 1.        ]
0.98 accuracy with a standard deviation of 0.02

References - https://scikit-learn.org/stable/modules/cross_validation.html

$\endgroup$
1
  • $\begingroup$ I agree with the implementation for cross validation, however I would like to build the a simpler cross validation from scratch. Check my code for reference. $\endgroup$
    – AGX301
    Sep 24 at 5:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.