# Bag of Visual Words

What I am trying to do:

I am trying to classify some images using local and global features.

What I have done so far:

I have extracted sift descriptors for each image and I am using this as my input for k-means to create my vocabulary from all of the features of every image. From here I create a histogram from my clusters for each image by passing the image sift features to the predict method in k-means giving me the labels of the clusters. From here I create the histogram by counting the number of labels for each bin. Now I have a nxm matrix where n is the number of images, and m being the number of clusters (features/words) of each image.

I will feed this matrix to a classifier to get my image classifications.

Steps in a nutshell:

1. Extract sift feature descriptors giving us a nx128 matrix for each image

2. Stack all feature descriptors into one large list

3. Fit all of these features into the kmeans algorithm setting k=100

4. For every image use its sift features to predict the labels of the clusters using the same trained kmeans model

5. Create a histogram from the clusters using k as the number of bins, adding 1 to the bin for each label in the model. (if an image has 10 features from sift, it will give us 10 labels, these 10 labels will be in the range of k, so for each label we will add it to the corresponding bin for our histogram).

6. We now have a nxk matrix, where n is the number of images, and k being the number of clusters.

7. We now feed in the histograms to a classifier and ask it to predict on the testing data.

The problem:

Am I correctly performing bag of visual words?

Here is my code:

def extract_features(df):
IF = imageFeatures()
global_features = []
sift_features = []
labels = []
for i, (index, sample) in enumerate(df.iterrows()):
image = cv2.resize(image, shape)
hist = IF.fd_histogram(image)
haralick = IF.fd_haralick(image)
hu = IF.fd_hu_moments(image)
lbp = IF.LocalBinaryPatterns(image, 24, 8)
kp, des = IF.SIFT(image)
if len(kp) == 0:
#print (i)
#print (index)
#print (sample)
#return 0
des = np.zeros(128)
sift_features.append(des)
global_feature = np.hstack([hist, haralick, hu, lbp])
global_features.append(global_feature)
labels.append(sample["class_id"])
scaler = MinMaxScaler(feature_range=(0, 1))
rescaled = scaler.fit_transform(global_features)
return sift_features, rescaled, labels

def BOVW(feature_descriptors, n_clusters = 100):
print("Bag of visual words with {} clusters".format(n_clusters))
#take all features and put it into a giant list
combined_features = np.vstack(np.array(feature_descriptors))
#train kmeans on giant list
print("Starting K-means training")
kmeans = MiniBatchKMeans(n_clusters=n_clusters, random_state=0).fit(combined_features)
print("Finished K-means training, moving on to prediction")
bovw_vector = np.zeros([len(feature_descriptors), n_clusters])#number of images x number of clusters. initiate matrix of histograms
for index, features in enumerate(feature_descriptors):#sift descriptors in each image
try:
for i in kmeans.predict(features):#get label for each centroid
bovw_vector[index, i] += 1#create individual histogram vector
except:
pass
return bovw_vector#this should be our histogram

if __name__ == '__main__':
n_clusters = 100
#set model
model = GaussianNB()
image_list_subset = image_list.groupby('class_id').head(80)#image_list.loc[(image_list["class_id"] == 0) | (image_list["class_id"] == 19)]
shape = (330,230)
train, test = train_test_split(image_list_subset, test_size=0.1, random_state=42)

train_sift_features, train_global_features, y_train = extract_features(train)
train_histogram = BOVW(train_sift_features, n_clusters)
import matplotlib.pyplot as plt
plt.plot(train_histogram[100], 'o')
plt.ylabel('frequency');
plt.xlabel('features');

test_sift_features, test_global_features, y_test = extract_features(test)
test_histogram = BOVW(test_sift_features, n_clusters)

'''Naive Bays'''
y_hat = model.fit(train_histogram, y_train).predict(test_histogram)
print("Number of correctly labeled points out of a total {} points : {}. An accuracy of {}"
.format(len(y_hat), sum(np.equal(y_hat,np.array(y_test))),
sum(np.equal(y_hat,np.array(y_test)))/len(y_hat)))

• If you downvote, please explain why. – Kevin May 2 '18 at 22:51
• I did not downvote, but it would be very useful to know why you think that there is something wrong with the code. Are you asking for a generic code review (that might hit the off-topic borderline)? Can you provide some sample inputs to the BOVW function, for testing purposes? – E_net4 the curator May 3 '18 at 17:35
• @E_net4 I am trying to make sure that I have the concept correct. The reason is that BOVW seems to not really improve results much. There may be many reasons for this, maybe the data is bad, or I dont have enough clusters, or maybe my features are not good. I just want to ensure that my approach is correct. I can provide a more succinct example, but I also have to provide data. Is there a way for me to do this? Maybe I can generate some data using numpy? – Kevin May 3 '18 at 18:04
• Could you tell us what are comparing the performance to? And to what data you are using? – Tony Knapp Nov 21 '18 at 22:09

The best way to answer your question is to go to the original paper that introduced the method:

The article is not long and written in an easy to understand manner. For your question, you can read just the first 6 pages.

Taken from the article "Visual Categorization with Bags of Keypoints":

The main steps of our method are:

• Detection and description of image patches

• Assigning patch descriptors to a set of predetermined clusters (a vocabulary) with a vector quantization algorithm

• Constructing a bag of keypoints, which counts the number of patches assigned to each cluster

• Applying a multi-class classifier, treating the bag of keypoints as the feature vector, and thus determine which category or categories to assign to the image.