# Image recognition of selfie images

I developed an Android app that lets anyone upload pictures of encyclopedic things (bridges, museums, dishes, landscapes, paintings, etc) to Wikimedia Commons.

Unfortunately, 5% of the users find it funny to upload their own selfie. So I want to programmatically check whether the picture is a selfie or not, and if it probably is, warn them that selfies are off-topic.

As a data set, I have:

• 1000 pictures that I consider as undesirable selfies. It is in part subjective, but usually such pictures show one or two human faces taken from an arm's distance and random backgrounds.
• 1000 pictures that are not selfies (bridges, museums, dishes, etc, anything really). Tricky: this also includes pictures of famous people, usually they are easy to distinguish from a selfie because the persons are at a further distance. If you see an extended arm then you can be sure it is a selfie.

All pictures are taken with smartphones (hundreds of different models), they are JPG files of 2MB to 5MB in various sizes and ratios, in portrait or landscape mode.

I must use only open source, and the resulting detection code must run in less than a second on low-end Android phones.

What approach and steps does this task call for?

• Wait, wait, wait - you want to run the algorithm process client side??? Why??? That sounds like a bad idea and is going to limit your growth over the long term. The better way is to just expose a REST service, send the image, run the algorithm on a server and send a reply back. Dec 12, 2018 at 21:14
• @UnknownCoder: My question is about performing that task on client-side, this is not debatable, but feel free to post a separate question about server-side if you want. We have very good reasons to perform the task client-side, too long to explain here, but that involves confidentiality and a requirement to work offline. Dec 13, 2018 at 2:57

I'd go transfer learning way. The idea is to take the net, that's already been trained on large data set and has developed a number of conv filters, which can be reused. There are a few of them available in tensorflow. You take net pre-trained on ImageNet, chop off the last layers responsible for classifying those filters, and substitute it with your own. That way you don't really need to have that much data to reach seminal scores.

You can also provide your own top layer to change the input shape.

base_model = MobileNetV2(
weights="imagenet",
include_top=False,
input_shape=(HEIGTH, WIDTH, DEPTH)
)


You can choose whether to retrain the net.

base_model.trainable = False


Now just instantiate a new model with the base model and add your final layers.

model = Sequential(base_model)



Then compile the model as usual.

model.compile(
loss='categorical_crossentropy',
learning_rate=0.0001
),
metrics=['accuracy']
)


Please note, that you have to choose the right number of final dense layers and their shape. You still need to fine-tune it: which activation function to use, should you do weight decay, at which rate, what optimizer, what learning rate, etc.

For this, you don't need to go through the pain of creating a model yourself. Use MTCNN to detect the face and get on with it.

For posterity, compare face size and position w.r.t. to the image amd take optimal decision.

Implementation of Image Recognition techniques There are lots of open source libraries available for the image recognition and classification. You can make use of the "TensorFlow" library for image recognition and can be integrated with your android application.

• I am indeed considering TensorFlow among other options, but there are dozens of ways to use TensorFlow. Would you mind giving more insight, for instance what pre-processing steps, how many hidden layers and why? etc Thanks :-) Sep 13, 2018 at 13:27

The best way is to use like

1. Pre-trained transfer learning model with U-Net(Mobile Net).
2. VGG-face with pre-defined-weights.

Mobile net with U-Net

def create_model(trainable=True):
model = MobileNet(input_shape=(IMAGE_HEIGHT, IMAGE_WIDTH, 3), include_top=False, alpha=ALPHA, weights="imagenet")

for layer in model.layers:
layer.trainable = trainable

block1 = model.get_layer("conv_pw_5_relu").output
block2 = model.get_layer("conv_pw_11_relu").output
block3 = model.get_layer("conv_pw_13_relu").output

x = Concatenate()([UpSampling2D()(block3), block2])
x = Concatenate()([UpSampling2D()(x), block1])

x = Conv2D(1, kernel_size=1, activation="sigmoid")(x)
x = Reshape((HEIGHT_CELLS, WIDTH_CELLS))(x)

return Model(inputs=model.input, outputs=x)


VGG-Face with pre-defined-weights

from keras_vggface.vggface import VGGFace
model=VGGFace(weights_path)
model.summary()