Bag of Visual Words

Question

What I am trying to do:

I am trying to classify some images using local and global features.

What I have done so far:

I have extracted sift descriptors for each image and I am using this as my input for k-means to create my vocabulary from all of the features of every image. From here I create a histogram from my clusters for each image by passing the image sift features to the predict method in k-means giving me the labels of the clusters. From here I create the histogram by counting the number of labels for each bin. Now I have a nxm matrix where n is the number of images, and m being the number of clusters (features/words) of each image.

I will feed this matrix to a classifier to get my image classifications.

Steps in a nutshell:

1. Extract sift feature descriptors giving us a nx128 matrix for each image

2. Stack all feature descriptors into one large list

3. Fit all of these features into the kmeans algorithm setting k=100

4. For every image use its sift features to predict the labels of the clusters using the same trained kmeans model

5. Create a histogram from the clusters using k as the number of bins, adding 1 to the bin for each label in the model. (if an image has 10 features from sift, it will give us 10 labels, these 10 labels will be in the range of k, so for each label we will add it to the corresponding bin for our histogram).

6. We now have a nxk matrix, where n is the number of images, and k being the number of clusters.

7. We now feed in the histograms to a classifier and ask it to predict on the testing data.

The problem:

Am I correctly performing bag of visual words?

Here is my code:

def extract_features(df):
    IF = imageFeatures()
    global_features = []
    sift_features = []
    labels = []
    for i, (index, sample) in enumerate(df.iterrows()):
        image = cv2.imread(sample["location"])
        image = cv2.resize(image, shape)
        hist = IF.fd_histogram(image)
        haralick = IF.fd_haralick(image)
        hu = IF.fd_hu_moments(image)
        lbp = IF.LocalBinaryPatterns(image, 24, 8)
        kp, des = IF.SIFT(image)
        if len(kp) == 0:
            #print (i)
            #print (index)
            #print (sample)
            #return 0
            des = np.zeros(128)
        sift_features.append(des)
        global_feature = np.hstack([hist, haralick, hu, lbp])
        global_features.append(global_feature)
        labels.append(sample["class_id"])
    scaler = MinMaxScaler(feature_range=(0, 1))
    rescaled = scaler.fit_transform(global_features)
    return sift_features, rescaled, labels

def BOVW(feature_descriptors, n_clusters = 100):
    print("Bag of visual words with {} clusters".format(n_clusters))
    #take all features and put it into a giant list
    combined_features = np.vstack(np.array(feature_descriptors))
    #train kmeans on giant list
    print("Starting K-means training")
    kmeans = MiniBatchKMeans(n_clusters=n_clusters, random_state=0).fit(combined_features)
    print("Finished K-means training, moving on to prediction")
    bovw_vector = np.zeros([len(feature_descriptors), n_clusters])#number of images x number of clusters. initiate matrix of histograms
    for index, features in enumerate(feature_descriptors):#sift descriptors in each image
        try:
            for i in kmeans.predict(features):#get label for each centroid
                bovw_vector[index, i] += 1#create individual histogram vector
        except:
            pass
    return bovw_vector#this should be our histogram

if __name__ == '__main__':
    n_clusters = 100
    #set model
    model = GaussianNB()
    image_list = pd.read_csv("image_list.csv")
    image_list_subset = image_list.groupby('class_id').head(80)#image_list.loc[(image_list["class_id"] == 0) | (image_list["class_id"] == 19)]
    shape = (330,230)
    train, test = train_test_split(image_list_subset, test_size=0.1, random_state=42)

    train_sift_features, train_global_features, y_train = extract_features(train)
    train_histogram = BOVW(train_sift_features, n_clusters)
    import matplotlib.pyplot as plt
    plt.plot(train_histogram[100], 'o')
    plt.ylabel('frequency');
    plt.xlabel('features');

    test_sift_features, test_global_features, y_test = extract_features(test)
    test_histogram = BOVW(test_sift_features, n_clusters)

    '''Naive Bays'''
    y_hat = model.fit(train_histogram, y_train).predict(test_histogram)
    print("Number of correctly labeled points out of a total {} points : {}. An accuracy of {}"
          .format(len(y_hat), sum(np.equal(y_hat,np.array(y_test))), 
                  sum(np.equal(y_hat,np.array(y_test)))/len(y_hat)))

Answer 1

The best way to answer your question is to go to the original paper that introduced the method:

"Visual Categorization with Bags of Keypoints" (2004)

The article is not long and written in an easy to understand manner. For your question, you can read just the first 6 pages.

Taken from the article "Visual Categorization with Bags of Keypoints":

The main steps of our method are:

• Detection and description of image patches

• Assigning patch descriptors to a set of predetermined clusters (a vocabulary) with a vector quantization algorithm

• Constructing a bag of keypoints, which counts the number of patches assigned to each cluster

• Applying a multi-class classifier, treating the bag of keypoints as the feature vector, and thus determine which category or categories to assign to the image.