I want to test a few devices as pure black box tests. That would mean some kind of robotics to press buttons and the touch screen and detecting screen elements like buttons in varying light conditions with a video camera. I can handle the mechanics but I am not sure how to got about screen detection.

Would machine learning be suitable for such a task? I imagine training the system under different lighting conditions.

  • $\begingroup$ The cost of your robotics development is likely to be high, so I would guess you have time and money to burn on this? If you are testing software on the devices, and not the physical devices themselves, usually you would automate at a lower level. Emulator, or even a device rigged up to access screen more directly e.g. developer.samsung.com/rtlLanding.do $\endgroup$ – Neil Slater Jan 7 '17 at 9:40
  • $\begingroup$ Either way, yes ML-based vision systems could help solve your problem, but your question is too vague to write a longer, meaningful answer. Most importantly, it is not really possible to assess how much work it would be, or suggest anything practical. Can ML vision systems be trained to recognise and locate application elements on a screen, such as buttons and sliders? Yes they can. $\endgroup$ – Neil Slater Jan 7 '17 at 9:44
  • $\begingroup$ The robotics stuff should be pretty cheap with something like Lego and some 3D printing. I am not worried about that. Do you have any pointers for where to start with ML vision? I am totally new to that area. $\endgroup$ – maxxxx Jan 7 '17 at 9:54
  • $\begingroup$ You built a similar robot before? I think you are talking about a major project even before considering vision system. It could well be cheap in terms of components, but take months of your time. Maybe get your idea for a rig assessed at robotics.stackexchange.com (if you post there, please check their site rules, they don't like vague, broad questions such as "Can I build a test rig using Lego+3D printing?" - give details). $\endgroup$ – Neil Slater Jan 7 '17 at 10:00
  • $\begingroup$ The devices all have a flat computer screen and some buttons so it's pretty straightforward X-Y movements and straight up and down. It will take a little time but we probably can get an intern to do it. "Robot" sounds a little more complex than it is. $\endgroup$ – maxxxx Jan 7 '17 at 10:05

Object recognition is a subfield of Computer Vision which consists of methods/algorithms aimed at determining whether a given object is present on a 2D image. Many object recognition systems just return a 'yes/no' answer, but there are also many methods that return the location of the object on the image by means of a bounding box.

This subfield comprises a vast range of methods, from low-complexity ones (like methods based solely on applying simple filters on the image, like colour/edges/etc.) to more complex ones in which machine learning methods (like deep learning) play a very important role.

Depending on the complexity of the objects you are planning to detect, and on the overall complexity of the images, machine learning may be completely unnecessary.

This Wikipedia page summarises many of these methods. You can start exploring from there, or you can start by using any introductory computer vision text.


Yes. There has been work on using computer vision techniques to detect screen elements, to aid testing of GUIs and for automation. See, e.g., the following papers:

This led to the Siku software package. There's probably lots of other work out there; those are just two papers I happen to be familiar with.

The actual machine learning needed is not too sophisticated. This is a basic object detection task. If you put together a good training set and train a convolutional neural network, I would imagine that the result should be pretty effective.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.