Let's build some artificial data. There are many ways to do this. I usually always prefer to write my own little script that way I can better tailor the data according to my needs.
Let us first go through some basics about data. A lot of the time in nature you will find Gaussian distributions especially when discussing characteristics such as height, skin tone, weight, etc. Let us take advantage of this fact.
According to this article I found some 'optimum' ranges for cucumbers which we will use for this example dataset.
- Temperature: normally distributed, mean 14 and variance 3. If a value falls outside the range $[10,18]$ then it is said to be not edible.
- Color: we will set the color to be 80% of the time green (edible). 10% of the time yellow and 10% of the time purple (not edible).
- Moisture: normally distributed, mean 96, variance 2. If the moisture is outside the range $[94, 98]$ then the cucumber is not edible.
Building the dataset
We will build the dataset in a few different ways so you can see how the code can be simplified.
A very verbose example
This example will create the desired dataset but the code is very verbose.
import numpy as np
# Number of samples
n = 100
data = []
for i in range(n):
temp = {}
# Get a random normally distributed temperature mean=14 and variance=3
temp.update({'temperature': np.random.normal(14, 3)})
# Get a color with 80% probability green, 10% probability yellow
# and 10% probability purple
color = 'green'
color_random_value = np.random.randint(0,10)
if color_random_value == 8:
color = 'yellow'
elif color_random_value == 9:
color = 'purple'
temp.update({'color': color})
# Get a random normally distributed moisture mean=96 and variance=2
temp.update({'moisture': np.random.normal(96, 2)})
# Verify if the instance is edible (label=0) or not (label=1)
label = 0
if temp['temperature'] < 10 or temp['temperature'] > 18:
label = 1
elif temp['color'] != 'green':
label = 1
elif temp['moisture'] < 94 or temp['moisture'] > 98:
label = 1
temp.update({'label': label})
data.append(temp)
Then we can put this data into a pandas DataFrame as
df = pd.DataFrame(data=data)
df.head()

A cleaner example
import numpy as np
n = 100
data = {'temperature': np.random.normal(14, 3, n),
'moisture': np.random.normal(96, 2, n),
'color': np.random.choice(['green', 'yellow', 'purple'],
size=100,
p=[0.8, 0.1, 0.1])}
df = pd.DataFrame(data=data)
Then we will get the labels from our DataFrame
def get_label(color, moisture, temperature):
if temperature < 10 or temperature > 18:
return 1
elif color != 'green':
return 1
elif moisture < 94 or moisture > 98:
return 1
return 0
df['label'] = df.apply(lambda row: get_label(row['color'],
row['moisture'],
row['temperature']), axis=1)
Getting the data ready for applying a classifier
One of our columns is a categorical value, this needs to be converted to a numerical value to be of use by us.
This can be achieved using
df['color_codes'] =df['color'].astype('category').cat.codes
Now we are ready to try some algorithms out and see what we get.
Visualizing the data
The first important step is to get a feel for your data such that we can try and decide what is the best algorithm based on its structure.
I prefer to work with numpy arrays personally so I will convert them
X = np.asarray(df[['color_codes', 'moisture', 'temperature']])
y = np.asarray(df['label'])
Let's plot the data in 3D
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X[:,0], X[:,1], X[:,2], c=y)
ax.set_xlabel('Color')
ax.set_ylabel('Moisture')
ax.set_zlabel('Temperature')
plt.show()

The blue dots are the edible cucumber and the yellow dots are not edible. We can see that this data is not linearly separable so we should expect any linear classifier to be quite poor here. I would presume that random forests would be the best for this data source.
Let's split the data into a training and testing set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
Let's see the distribution of the two different classes in both the training set and testing set
import matplotlib.pyplot as plt
%matplotlib inline
n_classes = 2
training_counts = [None] * n_classes
testing_counts = [None] * n_classes
for i in range(n_classes):
training_counts[i] = len(y_train[y_train == i])/len(y_train)
testing_counts[i] = len(y_test[y_test == i])/len(y_test)
# the histogram of the data
train_bar = plt.bar(np.arange(n_classes)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(n_classes)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')
plt.xlabel('Labels')
plt.xticks((0,1))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

Linear classifier
from sklearn import linear_model
clf = linear_model.SGDClassifier(max_iter=1000)
clf.fit(X_train, y_train)
clf.score(X_test, y_test)
0.54545454545454541
Support Vector Classifier
from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train, y_train)
clf.score(X_test, y_test)
0.72727272727272729
K-Nearest Neighbors
from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X_train, y_train)
neigh.score(X_test, y_test)
0.66666666666666663
Random Forests
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 100)
forest.fit(X_train, y_train)
print('Score: ', forest.score(X_test, y_test))
predictions = forest.predict(X_test)
Score: 1.0
Well we got a perfect score. As expected this data structure is really best suited for the Random Forests classifier.
make_classification
fromsklearn.datasets
. $\endgroup$ – ignoring_gravity Oct 2 '18 at 16:49