3
$\begingroup$

I am using a CNN to predict large changes in my target variable X. I am classifying several "set-up" states visually from my images.

I am only interested in big changes, or maybe no change. So, I can classify in 3 ways:

x is future period absolute change in X

  1. Label BigChange when x > A%. Label Flat when x < A%.
  2. Label BigChange when x > A%. Label Flat when x < B%, where B < A.
  3. Label BigChange when x > A%. Label Flat when x < B%. Label SmallChange when x > B && x < A.

Which of these approaches is likely to give me the best predictive accuracy for BigChange?

EDIT 1: Additional clarification

Input data is a set of images (60x60) with connection to Target variable (Float) X.

x is the delta between X values, and a big value might be 2-3% change. I am interested when the next x > some A (maybe 2%) i.e. there is a BigChange, but if x < B ( maybe 0.25% ) then I call that Flat. I want to classify my images so that I can predict a big change in x or when change is small. But I am NOT interested in predicting anything else (like changes > 0.25% but < 2%).

So, I am asking whether I should just classify and train on images that I have previously labelled BigChange or Flat. BUT this means when I run model on new data it will received lots of images that represent changes in x that are NOT Big or Flat. Is this problematic? Should I label things I am not interested in aswell for completeness?

$\endgroup$
  • $\begingroup$ Hi ManInMoon, A% and B% are typical statistical thresholds that you can set either via experiments or via knowledge of the data distribution. Without knowing either it would be foolish for me to suggest anything. For e.g. if you were looking at contour maps and want to decide whether the image depicts a "mostly flat" region or a "hilly region" or when it shifts from the former to the latter, you will need to set thresholds based on geo data from the region or some study. Once you set the thresholds and have a fair idea about the data dist, then its high school probability. Kindly elaborate $\endgroup$ – Vikram Murthy Feb 18 '19 at 14:06
  • $\begingroup$ Which labels do you have to train the model? $\endgroup$ – n1k31t4 Feb 18 '19 at 15:23
  • $\begingroup$ I am NOT looking for values for A or B. I am trying to find the correct approach between 1,2 or 3. $\endgroup$ – ManInMoon Feb 19 '19 at 9:16
  • $\begingroup$ You are training a CNN, so I assume you have some input data and some target output. Some dependent and independent variables. What are they exactly? Or do you want a fully unsupervised method? What is a change in an image? Raw pixel values or semantic content? $\endgroup$ – n1k31t4 Feb 19 '19 at 10:44
  • $\begingroup$ @n1k31t4 This is supervised. I want to link an image (raw pixel 60x60) with a change in my target variable (The variable I am trying to predict). My question is a little like identifying animals. If I want to identify horses. Do I just train it on pictures of horses? Or do I train it on an exhaustive list of different animals? Even though I only want to identify horses? Would having other classificationa make the model better or worse? $\endgroup$ – ManInMoon Feb 19 '19 at 15:01
1
$\begingroup$

I can only offer comments base on your statements. First of all,set your goal clear and straight. I think it is the maximization of BigChange and Flat Sensitivities .

I think Number 2 is problematic because of your concern of upcoming images. Classification is not Regression. There is a chance that model will never have a clue about the middle ranges. But you can set the classification threshold for flat and big change prediction for maximum sensitivities. The remaining unclassified images will be middle range changes. The problem here is that there could be a lot of mixed-ups from two different thresholds and unusual metric.

So I think 3 wins because 1 doesn't care if it is small or middle range changes. It is the most straightforward thing towards your goal. Note that even if you are classifying with 3 things the metric for optimization can just involve two things. But you should verify it. I think it is all right to compare the two remaining models. Cross validate and choose the winner base on combined sensitivity metric (including only flat and big changes). Just don't be fooled by randomness in selection.

You might just cross validate three models in the end just to be sure and always keep in mind the unusual metric for optimization. In addition, you should always remember classification thresholding (not target thresholding) for all models and there can always be a choice of no classification for some examples after thresholding.

I'll be happy if you share the result of your experimentation and verify my hunch if it is wrong or not.

$\endgroup$
1
$\begingroup$

Is calculating the change automatic, or you need to do it by hand?

If it is the former, can't you test simply training your CNN the $60\times 60$ img $\to x\%$ relationship? The continuous nature of $x\%$ might help the training better than a $2$ or $3$ value discrete one.

But of course this depends on your data, just test it, and you'll see if it helps or not. If $x\%$ can be high, and $A$ is relatively low, you might want to cut of $x$ over a certain value, but if you do that, do it way over $A\%$.

$\endgroup$
  • $\begingroup$ OK Understood. It is automatic, so I will try your suggestion - thank you $\endgroup$ – ManInMoon Feb 21 '19 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.