5
$\begingroup$

I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through.

I'm using logistic regression with R's glmnet package and alpha = 0 for ridge regression.

I'm using ridge regression actually since lasso deleted all my variables and gave very low area under curve (0.52) but with ridge there isn't much of a difference (0.61).

My dependent variable/output is probability of click, based on if there is a click or not in historical data.

The independent variables are state, city, device, user age, user gender, IP carrier, keyword, mobile manufacturer, ad template, browser version, browser family, OS version and OS family.

Of these, for prediction I'm using state, device, user age, user gender, IP carrier, browser version, browser family, OS version and OS family; I am not using keyword or template since we want to reject a user request before deep diving in our system and selecting a keyword or template. I am not using city because they are too many or mobile manufacturer because they are too few.

Is that okay or should I be using the rejected variables?

To start, I create a sparse matrix from my variables which are mapped against the column of clicks that have yes or no values.

After training the model, I save the coefficients and intercept. These are used for new incoming requests using the formula for logistic regression:

1 / (1+e^-1*sum(a+k(ith)*x(ith)))

Where a is intercept, k is the ith coefficient and x is the ith variable value.

Is my approach correct so far?

Simple GLM in R (that is where there is no regularized regression, right?) gave me 0.56 AUC. With regularization I get 0.61 but there is no distinct threshold where we could say that above 0.xx its mostly ones and below it most zeros are covered; actually, the max probability that a click didn't happen is almost always greater than the max probability that a click happened.

So basically what should I do?

I have read how stochastic gradient descent is an effective technique in logit so how do I implement stochastic gradient descent in R? If it's not straightforward, is there a way to implement this system in Python? Is SGD implemented after generating a regularized logistic regression model or is it a different process altogether?

Also there is an algorithm called follow the regularized leader (FTRL) that is used in click-through rate prediction. Is there a sample code and use of FTRL that I could go through?

Improve this question
$\endgroup$
2
  • $\begingroup$ Instead of "saving the coefficients" you could save the whole model to a file, later load it again and use the predict() function. Should make the process a bit easier and less error prone. $\endgroup$
    – stmax
    Commented Aug 7, 2014 at 8:41
  • $\begingroup$ How much data did you use for training? Did you tune glmnet's lambda parameter and how? Cross validation? $\endgroup$
    – stmax
    Commented Aug 7, 2014 at 8:43

3 Answers 3

Reset to default
4
$\begingroup$

Stochastic gradient descent is a method of setting the parameters of the regressor; since the objective for logistic regression is convex (has only one maximum), this won't be an issue and SGD is generally only needed to improve convergence speed with masses of training data.

What your numbers suggest to me is that your features are not adequate to separate the classes. Consider adding extra features if you can think any any that are useful. You might also consider interactions and quadratic features in your original feature space.

Improve this answer
$\endgroup$
4
  • $\begingroup$ okay, can you elaborate what does interaction mean? I added classes like wifi enabled, gps enabled. now lasso performed slightly better than ridge, I also added more days for computation. However, the AUC is now in range of .51 to .55 only. Is there anything else i can do? Will try adding quadratic features, and what else? $\endgroup$
    – aasthetic
    Commented Jul 10, 2014 at 5:34
  • 1
    $\begingroup$ For every feature, you can create quadratic features (x_i ^ 2) and interaction features (x_i * x_j). Also be aware that there are hyper-parameters for both methods of regularization that should be tuned rather than left at their defaults $\endgroup$
    – Ben Allison
    Commented Jul 10, 2014 at 8:19
  • $\begingroup$ i took alpha in between 0 and 1 as well for trying regression, is that what you mean? $\endgroup$
    – aasthetic
    Commented Jul 10, 2014 at 14:47
  • 1
    $\begingroup$ I suspect by alpha you mean the step size? If so, this is not the parameter I'm referring to---there's another parameter that regulates how much the regularization term is multiplied by in the objective, which allows you to balance model fit (likelihood) with sparseness as measured by the regularizer. This needs tuning. $\endgroup$
    – Ben Allison
    Commented Jul 14, 2014 at 10:04
1
$\begingroup$

A lot of the features you mentioned are categorical, and with so many levels of each, the dimensions of your problem will expand. Rather than initially focusing on Lasso and Ridge regression, why don't you first look for clusters among the samples (records) to learn about the data set? Under you current approach, you're throwing everything into a model and expecting a high AUC(?). You may find several clusters where frequencies of the category levels predominate in one or more potential clusters. If you don't know what the cluster structure is of the samples (records), try k-means clustering based on centroids of feature values to see if there are unique clusters.

Once you get a handle on the cluster structure, then address regression issues. Your regression models may be breaking down, in part, because of large inhomogeneities in your data, along with the previously suggested issues.

Machine learning is all about performing unsupervised class discovery followed by class prediction (your output binary variable).

At this point, it's not clear that you studied the data to learn about its cluster structure, and rather threw it into a supervised model expecting to attain high AUC values.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Machine learning is not about performing unsupervised class discovery followed by class prediction. Some researchers do study this, but this is not a general fact of ML. Additionally, in a simple linear model such as a Logistic Regression, there is no unsupervised class discovery. It is fully supervised. More importantly, specifying clusters before piping your output into a supervised algorithm in the best case will do you no better than just throwing in the original features. In the worst case, you can select clusters that are meaningless for predicting your outcome. $\endgroup$
    – franciscojavierarceo
    Commented Nov 21, 2016 at 11:48
  • $\begingroup$ Yes, everyone knows logistic regression has nothing to with unsupervised, so trying to understand why you stated it has nothing to do with unsupervised. If the data are novel (new), the intent is to use ML to learn about the cluster structure of the data first to gain a full understanding of the data before becoming victim of the popularized method to unknowingly throw the data into a "sausage machine" and expect a classifier to do the dirty work. Directly Inputting data into a classifier assumes you performed microsurgery on the data - and know full well about its patterns. $\endgroup$
    – user9086
    Commented Nov 23, 2016 at 1:40
  • $\begingroup$ Users who straightforwardly go to class prediction before class discovery likely already know the number of classes via a gold standard. However, data mining with CV should be applied from the perspective of a customer saying: "the data are totally new, and we don't know anything about it." However, far too often, public domain data are used, for which a priori an e.g. "2-class" or "3-class" data set needs to be classified. This approach is of no help for new data. Always analyze data as if it's novel, since you will never get a second chance when a lab gives you the good stuff. $\endgroup$
    – user9086
    Commented Nov 23, 2016 at 1:57
0
$\begingroup$

SGD has nothing to do with regularization, and so does FTRL. They are learning methods approximating the optimal solution in classification or regression problem.

If you want to see how FTRL works, you can check my code which was applied in my industrial project.

Here is another learning method called TDAP based on FTRL, you can check the code.

Hope this will help you, and good luck!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.