29

TL;DR: diseases are rare, so the absolute number of false positives is a lot more than that of false negatives. Let's assume that our system has the same false positive and false negative rate of 1% (pretty good!), and that we're detecting the presence of new cancers this year: 439.2 / 100,000 people, or 0.5% of the population. [source] No cancer, no ...


19

Ok, so after a lot of experimentation I have managed to get some results/insights. In the first place, everything being equal, smaller batches in the training set help a lot in order to increase the general performance of the network, as a negative side, the training process is muuuuuch slower. Second point, data is important, nothing new here but as I ...


16

You know the story of the boy who cried wolf, right? It's the same idea. After some classifier gives false alarms (cries wolf) so many times, the medical staff will turn it off or ignore it. "Oh, this again! NOPE!" At least with the bioengineering group I've worked with, the emphasis is on reducing FPR specifically because the goal is to make a tool that ...


12

There is no requirement for specific pixel dimensions for convolutional neural networks to function normally. It is likely the values have been chosen for pragmatic reasons - such as a compromise between using image details vs number of parameters and training set size required. In addition, if source data has a range of different aspect ratios, some ...


12

In images, some frequently used techniques for feature extraction are binarizing and blurring Binarizing: converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc. This is how you do ...


11

This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that the padded area is not relevant for classification, and it does not have to learn that if you use interpolation, for instance.


11

You should validate only on the original images. The augmentation is there so that it can help your model generalize better, but to evaluate your model you need actual images, not transformed ones. To do this in keras you need to define two instances of the ImageDataGenerator, one for training and one for validating. To train the model you need to set both ...


8

This great tutorial covers the basics of convolutional neuraltworks, which are currently achieving state of the art performance in most vision tasks: http://deeplearning.net/tutorial/lenet.html There are a number of options for CNNs in python, including Theano and the libraries built on top of it (I found keras to be easy to use). If you prefer to avoid ...


8

The short answer is it depends on two things: what definition of information you use and if you really use that information. From information theory point of view, if your transformation is reversible than the information is there. This happens because you can apply the inverse transformation to recover the original. So nothing is lost. This is similar with ...


8

Upsampling layer is used to increase the resolution of the image. In segmentation, we first downsample the image to get the features and then upsample the image to generate the segments. For deconvolution operation we pad the image with zeroes and then do a convolution operation on that, hence it is upsampled. For eg: - If after downsampling the images ...


7

As far as I understand the issue is the following: In image recognition the inputs to your network could be the pixels (grayscale or only 1 and 0 for black and white). If you want to, e.g. recognize handwritten numbers it is very difficult to only work with such values as you never know where exactly the number (i.e. the black values) will be. Is pixel 140 ...


7

Welcome to DataScience.SE! I'd never heard of this problem so I looked it up. It is explained on the third slide of this presentation by Geoff Hinton: More things that make it hard to recognize objects • Changes in viewpoint cause changes in images that standard learning methods cannot cope with. – Information hops between input dimensions (i....


7

The problem you face is commonly called the class imbalance and has been the subject of quite a bit of research. Here's a literature review, if you're interested: He, H., & Garcia, E. A. (2008). Learning from imbalanced data In particular, you might encounter two forms of imbalance: Absolute imbalance/rarity occurs when, while you have plenty of data ...


7

Summary: the question probably* isn't whether one false negative is worse than one false positive, it's probably* more like whether 500 false positives are acceptable to get down to one false negative. * depends on the application Let me expand a bit on @Dragon's answer: Screening means that we're looking for disease among a seemingly healthy population. ...


7

The simplest pipeline would be to do the following: OCR Named Entity Extraction Entity Disambiguation OCR This is basically transforming your receipts into plain text. If you have scans (pictures) of the receipts, then you need a method that can deal with images. For example, you could use tesseract. Named Entity Extraction This is detecting the parts ...


6

I couldn't find an URL text file for the ILSVRC2012 training set, but for complete imagenet you can download the URLs only as a text file: http://image-net.org/download I wrote the following script to get a feeling for the data: #!/usr/bin/env python """Analyze the distribution of classes in ImageNet.""" classes = {} images = 0 with open("fall11_urls....


6

You have a few options: For Small Images: upsample through interpolation pad the image using zeros If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the excess pixels in the largest dimension. Of course this would result in losing data, but you can repeatedly shift the center of your crop. This would help your ...


6

Ideally, data augmentation is a step in your training pipeline, which comes after splitting your data into train/validation/test sets. Otherwise, you have the same data point in both training and testing, even if it a little rotated. So your training pipeline could be something like this: +-> training set ---> data augmentation --+ ...


5

The objective function they use to train the CNN minimizes the squared L2 distance (i.e. the squared Euclidean distance) between two similar (positive) images and simultaneously maximizes the distance between two different (negative) images. That means, the (squared) Euclidean distance between two representations is a measure of their similarity. Then, ...


5

A general solution to this does not exist, even if we add some assumptions about the distribution of e.g. colours and shapes in the images or temporal coupling such as consecutive frames being similar. Problem Let $F_1,\dots,F_i$ be the $n$ original frames, each with $m$ pixels. Let $P$ be the permutation that is applied to the pixels of each frame before ...


5

The class_indices attribute in Keras’ flow_from_directory(directory) creates a dictionary of the classes and their index in the output array: classes: optional list of class subdirectories (e.g. [‘dogs’, ‘cats’]). Default: None. If not provided, the list of classes will be automatically inferred from the subdirectory names/structure under directory, where ...


5

A very clear and in-depth explanation is provided by the slow R-CNN paper by Author(Girshick et. al) on page 12: C. Bounding-box regression and I simply paste here for quick reading: Moreover, the author took inspiration from an earlier paper and talked about the difference in the two techniques is below: After which in Fast-RCNN paper which you ...


4

From How few training examples is too few when training a neural network? on CV: It really depends on your dataset, and network architecture. One rule of thumb I have read (2) was a few thousand samples per class for the neural network to start to perform very well. In practice, people try and see. A good way to roughly assess to what extent it could be ...


4

You could use the Food 101 dataset or UEC Food 256, both contains real-world food images.


4

This is a fascinating combinatorial problem. I would featuring each pixel using its full temporal trajectory, then embed them in a grid using the k nearest neighbors. The real goal is to maximize the likelihood of the video being a sequence of natural (real life) images, which you can test with a classifier, but you might be able to get away with just a ...


4

The paper cited does not mention linear regression at all. What it does is using a neural network to predict continuous variables, and refers to that as regression. The regression that is defined (which is not linear at all), is just a CNN with convolutional layers, and fully connected layers, but in the last fully connected layer, it does not apply sigmoid ...


3

If I were you I would use deep learning. You can use an autoencoder format for this. Essentially you would feed the image in, then each layer yields a smaller output. Then you would feed the output in reverse, using transposed weight matrices, which would yield progressively larger outputs. The final output from the backward pass would be the filled in image....


3

This is indeed a simple problem if tried to be tackled using semantic segmentation. Semantic segmentation itself is a computer vision problem that could be understood as an extension of object detection and could be understood as follows: Using semantic segmentation done using a network called as UNET, the model could be trained for the required image and ...


3

I took a deeper look at this problem with some example data (that the OP provided) and some simplifications. I just predicted brick height param as opposed to the 17 example params in the question. I used the Keras library in Python to explore a few different architectures. Initially I replicated the problem, simple CNNs were predicting identical values for ...


3

In Image Processing, this task is known as localization. You basically want to localize each digit in the image and then use your digit recognizer over the digits. A cursory google search for digit localization in images gives me following papers which seem to be very helpful. Reading Digits in Natural Images with Unsupervised Feature Learning Character ...


Only top voted, non community-wiki answers of a minimum length are eligible