This is a captcha where you have to select the side that makes more sense. I am trying to use machine learning trying to break it. My approach is to use Google's vision AI to extract keywords from those images and then use a markov chain and machine learning to predict the correct order.

Is there a better approach in your opinion? What are the chances to get it working?


Hmm the first few ideas of other things you could look for to make your model a little more sharp...

  1. Examine id's of the images/metadata to see if a statistically significant pattern emerges for how to arrange the images. (i1.png,i2.png,i3.png,etc),(Largest image is always on top, etc)
  2. They may purposely scramble the images in a way were they never let a picture be purposely on the right location which can be used to your advantage. You only have to decide in between three locations the correct location for any image.
  3. You may want to see if you can incorporate word2vec and/or LSTM for getting image to->words to->story syntax/context correct.
  4. If they allow two chances before reseting then your model will be more accurate if you can incorporate that. If it outputs a probability each of the images are at a location then you need to create two guesses not one.

Chances are low but I would be interested to see your results I think if they didn't make a terrible captcha you will have about 16% < x < 30 accuracy if they have systemic flaws could be a lot better.

  • $\begingroup$ Thank you for the word2vec tip $\endgroup$
    – Jan Moritz
    Nov 11 '19 at 9:19
  • $\begingroup$ No problem @JanMoritz! If you need anymore help/direction just ask. $\endgroup$ Nov 11 '19 at 22:05
  • $\begingroup$ I am around 68% to 75% accuracy now ;-) $\endgroup$
    – Jan Moritz
    Nov 12 '19 at 12:53
  • $\begingroup$ Wow I’m really impressed, nice job! $\endgroup$ Nov 12 '19 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.