BOVW - Combine vocabularies

Question

What I have so far

I have a set of images that I am trying to classify. I can also extract different feature descriptors from the images using algorithms such as hu moments, color histogram, and SIFT. I can the build a vocabularies from each of these algorithms.

What I do not understand is how can I combine these vocabularies together. From my understanding these feature descriptors can be of different size depending on what the features are. What would be the proper way to build this vocabulary from these different algorithms.

What is confusing me is how would we be combining feature descriptors from different algorithms which have different sizes? SIFT will give a m x 128 array where m is the number of feature descriptors of set size 128. A color histogram may give a count of each color in the range of 0-255. How would I combine these two in a meaningful way?

Here is the paper I was reading which gave me the idea of combining the vocabularies.

A Visual Vocabulary for Flower Classification

Here is what I was reading before:

Implementing Bag of Visual words for Object Recognition

If anyone can point me in the right direction I would greatly appreciate it.

Answer 1

I also found an interesting paper (with code) where they merged local and global descriptors.

Scene Recognition by Combining Local and Global Image Descriptors

According to the paper: They take the daisy features (local features), cluster the features into k clusters, then create a histogram with k bins. They normalize the histogram, once they do this they concatenate this vector (the histogram) with the normalized HOG vector (global features). So in the end we just cluster the local features and create histogram, then just concat with global vector and there we have a training sample.

Also there may be an alternate method where you just use local features to cluster: https://www.quora.com/How-do-I-merge-features-from-different-feature-extractors-i-e-color-histogram-and-SIFT-for-bag-of-visual-words#