1
$\begingroup$

I use openface to extract feature vectors in 128D. I need to find a suitable database engine to store these vectors for future comparisons (calculating the euclidean distance between newly extracted features and those stored in the database).

Here is an example of what I'm trying to do:

I use OpenFace to extract the face representation, this is a 128D feature vector. I then calculate the euclidean distance between this vector and those stored in the database, returning the vector that has the lowest distance (less than 0.9) as a match. If no vector stored in the database matches this criteria, I store the newly extracted features as a new entry in my dataset.

What is a good database engine to achieve this?

Improve this question
$\endgroup$
3
  • $\begingroup$ The Euclidean distance between 128D vectors is not meaningful due to the concentration of distance (around the mean). $\endgroup$
    – Emre
    Commented Nov 15, 2016 at 18:14
  • $\begingroup$ hey Zaid what was the solution you end up using ? I am having a similar situation where i need to store feature embeddings for certain objects detected in video, which means i have to store probably 2-3M feature vectors at any given time and new being added every second, do you have any solution that you would recommend ? $\endgroup$
    – Pawan
    Commented Jul 23, 2020 at 10:49
  • $\begingroup$ @Pawan my use case is different than yours as I do not need to store that many vectors. I ended up using elasticsearch as it gave me the fastest distance query time. Had to save the vector as string though to maintain the order of elements $\endgroup$
    – Zaid Amir
    Commented Jul 26, 2020 at 10:25

1 Answer 1

Reset to default
2
$\begingroup$

If you really need to do that (I will argue it is not a good idea), with Postgres you can store an array type and write a stored procedure for new item insertion. This stored procedure can do whatever distance checking you wish, such as checking the distance of the new vector against all others in the database, before storage.

I would argue against this design because I suspect the criteria for uniqueness* could easily change over time. I think it'd be a better idea to store all of the vectors except exact matches. Then, create another table which uses the definition of uniqueness. Creating this table would be handled on the application side. If your definition of uniqueness changes, no problem, just make a new table. You could even compare several different definitions to see how your results are sensitive to it. If you do it this way, Cassandra is a great database choice. It's specifically designed for denormalized data storage (you have the same data stored in different forms or variations, so that your application gets exactly what it needs without further computation).

*In your post you stated that a similarity of < 0.9 would result in storing a new vector. That's what I mean by criteria for uniqueness.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The vectors are unique, there are no close matches in the original set. And matches will not be stored in the database. Not sure if cassandra is suitable for calculations of this magnitude, as I project in the first year alone there might be close to 1,000,000 unique vectors. $\endgroup$
    – Zaid Amir
    Commented Nov 17, 2016 at 15:29
  • $\begingroup$ If you take the Cassandra route you wouldn't be calculating anything in-database. The work would be offloaded. $\endgroup$
    – Pete
    Commented Nov 17, 2016 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.