What I am trying to do:

I am trying to classify some images using local and global features.

What I have done so far:

I have extracted sift descriptors for each image and I am using this as my input for k-means to create my vocabulary from all of the features of every image. From here I create a histogram from my clusters for each image by passing the image sift features to the predict method in k-means giving me the labels of the clusters. From here I create the histogram by counting the number of labels for each bin. Now I have a nxm matrix where n is the number of images, and m being the number of clusters (features/words) of each image.

I will feed this matrix to a classifier to get my image classifications.

Steps in a nutshell:

  1. Extract sift feature descriptors giving us a nx128 matrix for each image

  2. Stack all feature descriptors into one large list

  3. Fit all of these features into the kmeans algorithm setting k=100

  4. For every image use its sift features to predict the labels of the clusters using the same trained kmeans model

  5. Create a histogram from the clusters using k as the number of bins, adding 1 to the bin for each label in the model. (if an image has 10 features from sift, it will give us 10 labels, these 10 labels will be in the range of k, so for each label we will add it to the corresponding bin for our histogram).

  6. We now have a nxk matrix, where n is the number of images, and k being the number of clusters.

  7. We now feed in the histograms to a classifier and ask it to predict on the testing data.

The problem:

Am I correctly performing bag of visual words?

Here is my code:

def extract_features(df):
    IF = imageFeatures()
    global_features = []
    sift_features = []
    labels = []
    for i, (index, sample) in enumerate(df.iterrows()):
        image = cv2.imread(sample["location"])
        image = cv2.resize(image, shape)
        hist = IF.fd_histogram(image)
        haralick = IF.fd_haralick(image)
        hu = IF.fd_hu_moments(image)
        lbp = IF.LocalBinaryPatterns(image, 24, 8)
        kp, des = IF.SIFT(image)
        if len(kp) == 0:
            #print (i)
            #print (index)
            #print (sample)
            #return 0
            des = np.zeros(128)
        global_feature = np.hstack([hist, haralick, hu, lbp])
    scaler = MinMaxScaler(feature_range=(0, 1))
    rescaled = scaler.fit_transform(global_features)
    return sift_features, rescaled, labels

def BOVW(feature_descriptors, n_clusters = 100):
    print("Bag of visual words with {} clusters".format(n_clusters))
    #take all features and put it into a giant list
    combined_features = np.vstack(np.array(feature_descriptors))
    #train kmeans on giant list
    print("Starting K-means training")
    kmeans = MiniBatchKMeans(n_clusters=n_clusters, random_state=0).fit(combined_features)
    print("Finished K-means training, moving on to prediction")
    bovw_vector = np.zeros([len(feature_descriptors), n_clusters])#number of images x number of clusters. initiate matrix of histograms
    for index, features in enumerate(feature_descriptors):#sift descriptors in each image
            for i in kmeans.predict(features):#get label for each centroid
                bovw_vector[index, i] += 1#create individual histogram vector
    return bovw_vector#this should be our histogram

if __name__ == '__main__':
    n_clusters = 100
    #set model
    model = GaussianNB()
    image_list = pd.read_csv("image_list.csv")
    image_list_subset = image_list.groupby('class_id').head(80)#image_list.loc[(image_list["class_id"] == 0) | (image_list["class_id"] == 19)]
    shape = (330,230)
    train, test = train_test_split(image_list_subset, test_size=0.1, random_state=42)

    train_sift_features, train_global_features, y_train = extract_features(train)
    train_histogram = BOVW(train_sift_features, n_clusters)
    import matplotlib.pyplot as plt
    plt.plot(train_histogram[100], 'o')

    test_sift_features, test_global_features, y_test = extract_features(test)
    test_histogram = BOVW(test_sift_features, n_clusters)

    '''Naive Bays'''
    y_hat = model.fit(train_histogram, y_train).predict(test_histogram)
    print("Number of correctly labeled points out of a total {} points : {}. An accuracy of {}"
          .format(len(y_hat), sum(np.equal(y_hat,np.array(y_test))), 
  • 4
    $\begingroup$ If you downvote, please explain why. $\endgroup$
    – Kevin
    May 2, 2018 at 22:51
  • $\begingroup$ I did not downvote, but it would be very useful to know why you think that there is something wrong with the code. Are you asking for a generic code review (that might hit the off-topic borderline)? Can you provide some sample inputs to the BOVW function, for testing purposes? $\endgroup$
    – E_net4
    May 3, 2018 at 17:35
  • $\begingroup$ @E_net4 I am trying to make sure that I have the concept correct. The reason is that BOVW seems to not really improve results much. There may be many reasons for this, maybe the data is bad, or I dont have enough clusters, or maybe my features are not good. I just want to ensure that my approach is correct. I can provide a more succinct example, but I also have to provide data. Is there a way for me to do this? Maybe I can generate some data using numpy? $\endgroup$
    – Kevin
    May 3, 2018 at 18:04
  • $\begingroup$ Could you tell us what are comparing the performance to? And to what data you are using? $\endgroup$
    – Tony Knapp
    Nov 21, 2018 at 22:09

1 Answer 1


The best way to answer your question is to go to the original paper that introduced the method:

"Visual Categorization with Bags of Keypoints" (2004)

The article is not long and written in an easy to understand manner. For your question, you can read just the first 6 pages.

Taken from the article "Visual Categorization with Bags of Keypoints":

The main steps of our method are:

• Detection and description of image patches

• Assigning patch descriptors to a set of predetermined clusters (a vocabulary) with a vector quantization algorithm

• Constructing a bag of keypoints, which counts the number of patches assigned to each cluster

• Applying a multi-class classifier, treating the bag of keypoints as the feature vector, and thus determine which category or categories to assign to the image.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.