For a computer vision project I am working with images that the company only allows me to have on my computer for a maximum of 24 hours due to regulations. Every day a few hundred images come in via the system. I am trying to solve a object detection problem to prioritize workflow. For this I need more than a few hundred images to train RCNN's.

Q: I am wondering if there is a way to save images in such a way that they are NOT re-traceable anymore to the original image, while at the same time it IS possible to train some kind of RCNN on them?

I was thinking about saving saving convolutional features and labeled bounding boxes instead of the images. But that way it might be re-traceable and I want to be compliant with the regulations. Also, this way I can't train the convolutional layers of a network anymore, since I will only be working with it's output.


1 Answer 1


Wow. It is common for medical images to be heavily regulated (have to be kept on a separate server, cannot be copied, monitored access, etc...), but your situation is even tougher! Anyway, here's a few thoughts:

  • I don't think you can extract conv features, because a great deal of information (sometimes nearly everything) for reconstruction can be stored in those features (e.g., see Understanding Deep Image Representations by Inverting Them). Presumably this means you cannot simply train some kind of autoencoder and keep the latent representations either.

  • You could try to run the training in an "online" manner, meaning train on the new set of images each time they come. However, we must fight to avoid overfitting, so some heavy regularization (dropout, weight decay, early stopping, maybe even using a Bayesian neural network) and data augmentation probably would be needed. One methodology very relevant to this is work on catastrophic forgetting and continual learning. For instance, see Overcoming catastrophic forgetting in neural networks. You may be able to adapt these techniques to your case by treating each new set of data as a new task. One approach is to identify the important parameters each time you've trained on a new data subset, and prevent them from being changed to much when training on the next set.

  • A related approach is to use ensembles (similar to boosting). I imagine something like this (I'm sure you can think of something better): let $B_t$ be the $t$th data subset you receive. Suppose you already have a set of models $\mathcal{M}_{t-1} = M_1,\ldots,M_{t-1}$ with fixed weights from the previous time steps (denote $\Theta_i$ as the parameters of $M_i$). Let ${f}$ be a model that combines the outputs of $\mathcal{M}_{t-1}$ (i.e., if $M_i:\mathcal{I}\rightarrow{\mathcal{D}}$, then $f:2^\mathcal{D}\rightarrow\mathcal{S}$). Then you train $f$ and $M_t$ on $B_t$, but your loss focuses on data examples where the outputs of $\mathcal{M}_{t-1}$ are poor. The idea is to avoid overfitting by training $M_t$ to only care about learning things that the other older models don't already know. This is a little tough in the detection case (I haven't thought about how to do it honestly), since the output is complicated (maybe an architecture like Deep Sets could help). If it was simple regression ($\mathcal{S}=\mathbb{R}^n$), you could have $f(I)=\sigma_a(\sum_{\ell=1}^t c_\ell M_\ell(I))$, with a regularized loss function that enforces $c_u,\,u<t$ don't change too much, and $c_t$ isn't too large (note only all the $c_k$'s and $\Theta_t$ are trainable for $B_t$).

  • One other idea might be to train generative models (like GANs), and use them to generate novel data from the same distribution as these inputs. How to avoid forgetting for the critics is an issue of course. Usually, you can generate a GAN sample $S$, find its nearest neighbour in the original set $N_S$, and (empirically) $S$ tends to be fairly far from $N_S$. But this may not be sufficient to pass the requirements in your case.

  • Finally, you may be able to use some work from the privacy and fairness machine learning literature. The idea is that you can destroy or ignore certain "undesirable" information in the model, data, and/or representations, in order to preserve privacy and/or enforce some notion of fairness in the model. In your case, you may be able to train some kind of "fair, minimal autoencoder" that can learn a representation of your data that does not contain problematic properties, yet is still useful for training. For instance, if the images are sensitive because they contain information about some particular property of a given person, you may be able to learn a representation that does not hold information about that property (and then save and keep those). It really depends on the specifics of your situation of course. I am not an expert in this area, unfortunately, but a place to start might be Learning Adversarially Fair and Transferable Representations.

  • $\begingroup$ Thank you for your extensive answer! I will read the articles and experiment with these approaches, starting with the online learning approach. Maybe I can ask if in the beginning I can keep the images for 1 month so I can train the convolutional layers, then freeze these layers and do online/incremental learning with only the classifier, using the convolutional features. I will let you know which approach was most successful. $\endgroup$
    – DS_GB
    Jul 23, 2019 at 7:49
  • $\begingroup$ @DS_GB Feel free to comment back here if you figure something out! PS: I think some people call that field lifelong learning. $\endgroup$ Jul 23, 2019 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.