Best method to deal with too many zeroes in regression problem?

Question

I'm trying to solve a simple regression problem using TensorFlow and Pandas to see what's the expected conversion rate for a given product given how much I'm willing to pay for each visit (cost per click or cpc).

Each data point represents a single day and its given acquisition cost and conversion rate. I expect conversion rate to decrease exponentially as visits increase because my audience gets wider and not as focused anymore.

The data seems to agree with this:

The problem is that for the majority of data points, the conversion rate is zero (i.e. no products were sold) so any model is likely to perform well by merely guessing zero sales. For this particular subset (one of the best selling products) the ratio of zero-to-non-zero is roughly 4:1 but in the general data it's about 300:1.

The two best ways that I can think of dealing with this are

a) summarizing data by time periods large enough to contain at least one conversion in them (problem with this is that I lose data by having to settle for the average cost instead of the more detailed daily data)

b) splitting the problem into a classification problem (will I sell at least one unit) and a regression problem (given that I'm going to sell at least one unit, how many units can I expect to sell)

Do either of those sound like a smart idea? Are there any tried-and-true methods to solve this kind of problem?

Answer 1

Option (b) is a good approach. If you use a classifier that outputs probabilities, you can even multiply the two model outputs together to calculate a risk expectation that's interpretable as the joint likelihood.

If cpc is discrete valued and non negative, you could interpret that as a count and use a zero-inflated Poisson model. If it's not, you could convert it by rounding to the nearest cent or something like that.

Answer 2

You asked about „tried-and-true“ method, thus I‘m suggesting the logistic regression.

I‘m not familiar with TF yet, so not sure whether this approach is easily doable there.

Below is the link to the answer explaining how to do it with R‘s glm: to put it shortly, besides specifying the response column (rates between 0 and 1) you have to provide the weight column representing the total number of trials.

how to do logistic regression for fractional outcome

Answer 3

One possibility would be to go for hurdle-ish model with separate heads for classification and regression. The classification should be understood as whether products were sold (non-zero regression value) or not (regression value zero). The benefit is that it is possible to manually zero out the regression values, in case the classifier votes for the negative class. As an example, here's a two-head tf.keras model that could be suitable for image regression.

from tensorflow.keras.layers import *
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.applications import ResNet50

...

input_shape = (256,256,3)
inputs = Input(input_shape)

# feature extration
base_model = ResNet50(
    input_shape=input_shape,
    weights='imagenet',
    include_top=False,
)
base_model = base_model(inputs)

# bottleneck
bottleneck = GlobalAveragePooling2D()(base_model)
bottleneck = Dense(256, activation='relu')(bottleneck)
bottleneck = Dropout(0.25)(bottleneck)

# classification head
clf = Dense(32, activation='relu')(bottleneck)
clf = Dropout(0.5)(clf)
clf = Dense(1, use_bias=False)(clf)
clf_output = Activation("sigmoid", name="clf_head_output")(clf)

# regression head
regr = Dense(32, activation='relu')(bottleneck)
regr = Dropout(0.5)(regr)
regr = Dense(1, use_bias=False)(regr)
regr_output = Activation("relu", name="regr_head_output")(regr) #<- change activation according to your task

# build model
model = Model(inputs=[inputs], outputs=[regr_output, clf_output,])

Typically one would select mean_absolute_error or mean_squared_error as the regression output loss. The problem you may face in the case of ZIP (zero-inflated Poissonian) distribution is that the model's predictions might tend to be lower than the ground truth, especially for the highest regression values. If this is the case, then this is probably because the model gets most punished for predicting the high regression values wrong.

Standardizing the regression outputs is important, and one could try tricks such as boxcox, however, this might be difficult, keeping in mind that zero has a special role in the distribution. If there is some certain lower (L) and upper (U) bound for the regression task, a trick that has worked well for me is to normalize the outputs to the interval [0,1] and use angular regression instead. The idea is that then the original L and U would be identical with each other (the regression values lie on the unit circle) and the model gets equal punishment whether it tries to approach the high/rare values from "above" or "below". Here's also my implementation of angular loss for tf.keras. I suggest using sigmoid or clipped relu (with max_value=1.) with this approach.

import tensorflow as tf
import tensorflow.keras.backend as K

PI = tf.constant(math.pi)
def angular_l1(y_true, y_pred): #y_true and y_pred should be in the inteval [0,1]
    y_true = y_true*2*PI #<- scale values to [0,2pi]
    y_pred = y_pred*2*PI #<- scale values to [0,2pi]
    return K.mean(K.abs(tf.math.atan2(K.sin(y_true - y_pred), K.cos(y_true - y_pred))))