1
$\begingroup$

We have a huge dataset with us that looks like below

Factor -|- ... -|- Rank1 -|- Rank2 -|- Calls
A11-----|- ... -|-0.1234--|-3.2345--|- Cat A
A11-----|- ... -|-1.1234--|-0.2345--|- Cat B
A12-----|- ... -|-2.1234--|-3.2345--|- Cat C
A12-----|- ... -|-2.1234--|-3.2345--|- Cat C
...
A13-----|- ... -|-0.1234--|-3.2345--|- Cat A
A13-----|- ... -|-3.1234--|-0.1345--|- Cat B
A13-----|- ... -|-2.1234--|-2.2345--|- Cat C
A14-----|- ... -|-4.1234--|-4.2345--|- Cat C

and we have about 10 million of such data points. We also have a test set with about .2 million data points where we need to accurately call them out into different classes.

At this point, we are trying a mix of K-means & Random Forest approach coded in (python2.7-sklearn) which gives us about 90% accuracy at classification but we wanna reach for more.

I am interested to apply some kind of deep learning approach to this and that's why I'm learning about TensorFlow. But every link I've gone though in deep learning talks (tf documentation + youtube videos) about CNN and image recognition (MNIST .. etc) only and I'm not finding any idea how to start solving this.

I'm looking for your suggestions and guidance on how to approach this kind of problem / What neurons to stack or How to build models for this kind of data? I'm willing to burn my share of the midnight oil on any links and suggestions offered on this but I need to know if I'm even thinking right by trying to solve it with DL or if there's any other approach for this kind of data which may work better, or to solve this what should I study or learn?

Edit 1: So, I managed to go through the TensorFlow documentation for MNIST an perhaps I can see some correlation.

In the sense if I pass the rank1 & rank2 as input np.array for the 2 neuron input layer and each call as a one_hot np.array, where Cat A = [1,0,0],Cat B = [0,1,0],Cat C = [0,0,1] and put 3 neuron on the outermost layer each predicting one Cat(category) through a softmax function, will that work?

Even if so, what should I use as internal/hidden layers? Can I use the other factors as well in the input layers? Do I need to convert them into numeric(int/float)? Should I pass these rank 1 and rank 2 values to the input layer just as it is or should I engineer them anyway beforehand?

$\endgroup$
1
$\begingroup$

Yes, you can use DL network to solve this problem. It is a easy multi-classes classification task. And just a full connected network will do this work.

To construct the network I recommend using Keras which is easy to use. And before training the network, it is better to preprocess the data(standardizing the numeric column, and embedding the string column).

EDIT:

For the data normalisation you can reference this blog: Neural Network Data Normalization and Encoding, that show the basic methods and codes.

$\endgroup$
  • $\begingroup$ Many thanks for your response Yangguang. Can you please elaborate on "standardizing the numeric column, and embedding the string column". perhaps a small example. $\endgroup$ – S4nd33p Aug 25 '17 at 7:13
  • $\begingroup$ @S4nd33p I have updated the answer, you can reference the blog for data normalisation methods. $\endgroup$ – Yangguang Aug 25 '17 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.