4
$\begingroup$

I have a time-series dataset of incoming face data. Each data point is a facial-feature-vector of length 256 that represents the facial features of a person (it is generated by a modified RESNET). Features that are close together are deemed to belong to the same person.

I am (successfully) clustering the incoming face features by DBSCANing. I've recently switched to HDBSCAN also with good results.

My problem is this: DBSCAN and HDBSCAN require I have all the data together at one time. I often have >200,000 features which can be a very large download.

I would much prefer to be able to take every incoming f and assign it to a person without having to collect all the information at one time.

Is there an alternative to this (preferable with a Python implementation)?

Improve this question
$\endgroup$
2
+100
$\begingroup$

It sounds to me like what you should really do is train a (multi-class) classifier on the dataset, and then use it to 'predict' each new incoming face.
If you don't have another source of labels, you can use your DBScan result as a label (i.e. use the cluster as a class label).

That being said, you technically can check a new data sample by comparing in to the previous samples, but it's a heavier computation than inference on a classifier, because you'll need to load your data (or more accurately - the core points in the DBScan model; that's not a whole lot better, and not fitting for a stream) instead of loading a thinner, leaner classifier model.

There's also some useful discussion in this question in Stack Overflow, and an example code snippet to compare a new point to the existing points in a DBScan model; if you want to go by that route, you can try it. But really, if DBScan solves your problem to your satisfaction, I'm assuming you can also train a classifier with the DBScan clusters as labels. Good luck!

Improve this answer
$\endgroup$
3
  • $\begingroup$ This was the first idea that came to mind, the problem here is that there are often new faces appearing in the dataset. I guess there is no real way to distinguish a new face from noise, as the only difference is that a new face would satisfy the "min_cluster_size" metric (in the sklearn DBSCAN implementation anyway) $\endgroup$ – A_toaster Jul 19 '20 at 23:13
  • 1
    $\begingroup$ I ended up using HDBSCAN as it comes with an approximate_predict function which does what I need! The bigger problem with it is that I need to recluster my data every now and then in order to detect new groups, but I guess that's the only way to really know if incoming data is noise or a new group $\endgroup$ – A_toaster Jul 19 '20 at 23:19
  • $\begingroup$ 'Reclustering every now and then' sounds like common practice; it's not that different than re-training your model every now and then. Can you cluster it 'incrementally' or from scratch each time? Anyway, consider answering the question and accepting your own answer so others can learn from your solution and know that the question is closed. $\endgroup$ – Itamar Mushkin Jul 20 '20 at 5:40
4
$\begingroup$

You are describing incremental learning, input data is continuously used to extend the existing model's knowledge.

There is a Python implementation of incremental DBSCAN.

There is no current Python implementation of incremental HDBSCAN.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Do HDBSCAN and DBSCAN count as incremental learning? $\endgroup$ – Itamar Mushkin Jul 19 '20 at 5:43
2
$\begingroup$

I suggest you to use an Autoencoder for dimensionality reduction. An Autoencoder is a Neural Network with a hourglass shape, that is meant to learn a compressed representation of your data. You can train it first on the data you already have, and then use it to extract a compressed representation at a time. In your case, what you need is an Autoencoder with Convolutional layers, since you need to process pixel data.

Once you trained it well enough you can generate a representation of each image from your stream one by one. You could load one at a time without memory issues. Each time you generate a compressed representation, you can compare it with previous ones. Two pictures from the same person would end up being very similar to each other in the latent compressed space. Or you could even train a simple classifier that does the matching for you (that could be especially useful if you want to match two pictures of the same face, but turned in opposite directions for example).

The Internet and GitHub are plenty of Autoencoder works. Here I wrote a simple tutorial for a feedforward one in Python + TensorFlow 2. In your case, you need a Convolutional version of it. Your Encoder part will require 2D Conv layers, while the Decoder will require an inverse operation that can be done either with Upsampling layers or Transpose Conv layers (I've seen both implementations, this is an explanation of the two and how to use them).

An Autoencoder model in tensorflow.keras would look like something like this:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D

autoencoder = Sequential([
    # Encoder
    Conv2D(16, (3, 3), input_shape=(28,28,1) activation='relu', padding='same'),
    MaxPooling2D((2, 2), padding='same'),
    Conv2D(8, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2), padding='same'),
    Conv2D(8, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2), padding='same'),  # This is the compressed representation

    # Decoder
    Conv2D(8, (3, 3), activation='relu', padding='same'),
    UpSampling2D((2, 2)),
    Conv2D(8, (3, 3), activation='relu', padding='same'),
    UpSampling2D((2, 2)),
    Conv2D(16, (3, 3), activation='relu'),
    UpSampling2D((2, 2)),
    Conv2D(1, (3, 3), activation='sigmoid', padding='same')
])

This is a slightly modified version with Upsampling layers that I took from this Keras blog post. Alternatively, a version with Conv Traspose layers can be found here.

PS: Take a look at this article about finding alignments in hand written digits. Not exactly your problem, but rich of analogies IMHO.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for this writeup! I am currently using a Resnet based CNN for feature extraction, so doesn't it make more sense to continue using that rather than to pass the images through another neural network? $\endgroup$ – A_toaster Jul 13 '20 at 23:41
  • $\begingroup$ How are you using Resnet for feature extraction? How are your output nodes? ResNets are usually used for classification-related tasks, how do you use it for feature extraction? (Not a critique, I'm just curious!) $\endgroup$ – Leevo Jul 14 '20 at 7:36
  • $\begingroup$ Well basically the output of the Resnet is the 256d feature vector! I didn't create it, but I'm guessing that the final classification layers were removed and the last FC layers were kept $\endgroup$ – A_toaster Jul 19 '20 at 23:11
  • $\begingroup$ You are meant to add the final layers to the downloadable Resnet architecture. It's designed to make the model available for any specific task, simply by adding the last classification layers. Usually people donwload its pretrained weight, they freeze them (i.e. they stop their training), then they add few last layers they need for their specific task and then train only those last additional ones. $\endgroup$ – Leevo Jul 20 '20 at 7:21
0
$\begingroup$

Sure. As i got from your question, you need to have some online or on the fly method in order to classify incoming in to existing categories or may be new. So Here clustering technique like Kmeans and DBSCAN will be failed because as u said it need data at a time and whatever hypothesis learned from data its scope is to specific that batch of data only. you might have used that clustering model if your data would have been consistent categories. Here some of the solution that i would like to point below:

  • Ensemmble Modeling
    • How it is possible? I assume that your data come in iterative batch manner than u can train model for each batch and keep in mind that batch size should be large such that it can fit in your memory. At the end u will end up will so many model in your hand. That's fine as it is stored on disk.
    • How to use it for prediction? Now tricky part starts here. It will be good is your query(image vector representation) is passed to each and every model and see the result of each and every model. Again question araise that what could be evaluation matric for that? right. So u will have design evaluation matric your own which really represent the degree of particular sample belong to specific class. I mean to say that kind of confidence measure which show how much sample or data point belong to one class. U may use accuracy, entropy, etc. so u will get score with respect to all the class or cluster.As same each model return same. Now you will have decide that which model is confidently saying that sample belong specific class or cluster so u can pick up that and process your output accordingly. if u feel like no any model is showing confidence than u just put your data point in churn(garbage) and train model on garbage data base so it will come up will new insight.
    • Advantage : With each batch u will have a different models so u can take advantage of different hypotheses. The main advantage is your output will be consistent up to end. so u may no be face the issue of degradation of accuracy.

Reference : https://machinelearningmastery.com/ensemble-methods-for-deep-learning-neural-networks/

Please review my solution and give feedback. happy to hear your point of view. Good luck!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.