What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data

Question

I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on Google.

Here is an example scenario:

Image you own a business that has an average of 100 walk-in customers every day. One day, you decide you want to increase the number of walk-in customers arriving at your store each day, so you pull a crazy stunt outside your store to get attention. Over the next week, you see on average 125 customers a day.

Over the next few months, you again decide that you want to get some more business, and perhaps sustain it a bit longer, so you try some other random things to get more customers in your store. Unfortunately, you are not the best marketer, and some of your tactics have little or no effect, and others even have a negative impact.

What methodology could I use to determine the probability that any one individual event positively or negatively impacted the number of walk-in customers? I am fully aware that correlation does not necessarily equal causation, but what methods could I use to determine the likely increase or decrease in your business's daily walk in client's following a specific event?

I am not interested in analyzing whether or not there is a correlation between your attempts to increase the number of walk-in customers, but rather whether or not any one single event, independent of all others, was impactful.

I realize that this example is rather contrived and simplistic, so I will also give you a brief description of the actual data that I am using:

I am attempting to determine the impact that a particular marketing agency has on their client's website when they publish new content, perform social media campaigns, etc. For any one specific agency, they may have anywhere from 1 to 500 clients. Each client has websites ranging in size from 5 pages to well over 1 million. Over the course of the past 5 year, each agency has annotated all of their work for each client, including the type of work that was done, the number of webpages on a website that were influenced, the number of hours spent, etc.

Using the above data, which I have assembled into a data warehouse (placed into a bunch of star/snowflake schemas), I need to determine how likely it was that any one piece of work (any one event in time) had an impact on the traffic hitting any/all pages influenced by a specific piece of work. I have created models for 40 different types of content that are found on a website that describes the typical traffic pattern a page with said content type might experience from launch date until present. Normalized relative to the appropriate model, I need to determine the highest and lowest number of increased or decreased visitors a specific page received as the result of a specific piece of work.

While I have experience with basic data analysis (linear and multiple regression, correlation, etc), I am at a loss for how to approach solving this problem. Whereas in the past I have typically analyzed data with multiple measurements for a given axis (for example temperature vs thirst vs animal and determined the impact on thirst that increased temperate has across animals), I feel that above, I am attempting to analyze the impact of a single event at some point in time for a non-linear, but predictable (or at least model-able), longitudinal dataset. I am stumped :(

Any help, tips, pointers, recommendations, or directions would be extremely helpful and I would be eternally grateful!

Answer 1

For the record, I think this is the type of question that's perfect for the data science Stack Exchange. I hope we get a bunch of real world examples of data problems and several perspectives on how best to solve them.

I would encourage you not to use p-values as they can be pretty misleading (1, 2). My approach hinges on you being able to summarize traffic on a given page before and after some intervention. What you care about is the difference in the rate before and after the intervention. That is, how does the number of hits per day change? Below, I explain a first stab approach with some simulated example data. I will then explain one potential pitfall (and what I would do about it).

First, let's think about one page before and after an intervention. Pretend the intervention increases hits per day by roughly 15%:

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

def simulate_data(true_diff=0):
    #First choose a number of days between [1, 1000] before the intervention
    num_before = np.random.randint(1, 1001)

    #Next choose a number of days between [1, 1000] after the intervention
    num_after = np.random.randint(1, 1001)

    #Next choose a rate for before the intervention. How many views per day on average?
    rate_before = np.random.randint(50, 151)

    #The intervention causes a `true_diff` increase on average (but is also random)
    rate_after = np.random.normal(1 + true_diff, .1) * rate_before

    #Simulate viewers per day:
    vpd_before = np.random.poisson(rate_before, size=num_before)
    vpd_after = np.random.poisson(rate_after, size=num_after)

    return vpd_before, vpd_after

vpd_before, vpd_after = simulate_data(.15)

plt.hist(vpd_before, histtype="step", bins=20, normed=True, lw=2)
plt.hist(vpd_after, histtype="step", bins=20, normed=True, lw=2)
plt.legend(("before", "after"))
plt.title("Views per day before and after intervention")
plt.xlabel("Views per day")
plt.ylabel("Frequency")
plt.show()

We can clearly see that the intervention increased the number of hits per day, on average. But in order to quantify the difference in rates, we should use one company's intervention for multiple pages. Since the underlying rate will be different for each page, we should compute the percent change in rate (again, the rate here is hits per day).

Now, let's pretend we have data for n = 100 pages, each of which received an intervention from the same company. To get the percent difference we take (mean(hits per day before) - mean(hits per day after)) / mean(hits per day before):

n = 100

pct_diff = np.zeros(n)

for i in xrange(n):
    vpd_before, vpd_after = simulate_data(.15)
    # % difference. Note: this is the thing we want to infer
    pct_diff[i] = (vpd_after.mean() - vpd_before.mean()) / vpd_before.mean()

plt.hist(pct_diff)
plt.title("Distribution of percent change")
plt.xlabel("Percent change")
plt.ylabel("Frequency")
plt.show()

Now we have the distribution of our parameter of interest! We can query this result in different ways. For example, we might want to know the mode, or (approximation of) the most likely value for this percent change:

def mode_continuous(x, num_bins=None):
    if num_bins is None:
        counts, bins = np.histogram(x)
    else:
        counts, bins = np.histogram(x, bins=num_bins)

    ndx = np.argmax(counts)
    return bins[ndx:(ndx+1)].mean()

mode_continuous(pct_diff, 20)

When I ran this I got 0.126, which is not bad, considering our true percent change is 15. We can also see the number of positive changes, which approximates the probability that a given company's intervention improves hits per day:

(pct_diff > 0).mean()

Here, my result is 0.93, so we could say there's a pretty good chance that this company is effective.

Finally, a potential pitfall: Each page probably has some underlying trend that you should probably account for. That is, even without the intervention, hits per day may increase. To account for this, I would estimate a simple linear regression where the outcome variable is hits per day and the independent variable is day (start at day=0 and simply increment for all the days in your sample). Then subtract the estimate, y_hat, from each number of hits per day to de-trend your data. Then you can do the above procedure and be confident that a positive percent difference is not due to the underlying trend. Of course, the trend may not be linear, so use discretion! Good luck!

Answer 2

Back in my data analyst days this type of problem was pretty typical. Basically, everyone in marketing would come up with a crazy idea that the sold to higher ups as the single event that would boost KPI's by 2000%. The higher ups would approve them and then they would begin their "test". Results would come back, and management would dump it on the data analysts to determine what worked and who did it.

The short answer is you cant really know if it wasn't run as a random A/B style test on like time periods. But I am very aware of how deficient that answer is, especially if the fact that a pure answer doesn't exist is irrelevant to the urgency of future business decisions. Here are some of the techniques I would use to salvage the analysis in this situation, bear in mind this is more of an art then a science.

Handles

A handle is something that exists in the data that you can hold onto. From what you are telling me in your situation you have a lot of info on who the marketing agency is, when they tried a tactic, and to which site they applied it to. These are your starting point and information like this going to be the corner stone of your analysis.

Methodology

The methodology is going to probably hold the strongest impact on which agencies are given credit for any and all gains so you are going to need to make sure that it is clearly outlines and all stake holders agree that it makes sense. If you cant do that it is going to be difficult for people to trust your analysis.

An example of this are conversions. Say the marketing department purchases some leads and they arrive at our landing page, we would track them for 3 days, if they made a purchase within that time we would count them as having been converted. Why 3 days, why not 5 or 1? Thats not important as long as everyone agrees, you now have a definition you can build off of.

Comparisons

In an ideal would you would have a nice A/B test to prove a definitive relationship, I am going to assume that you are running short on those, still, you can learn something from a simple comparison of like data. When companies are trying to determine the efficacy of radio advertising they will often run ads on offset months in the same market, or for several months in one market and compare that with the results in a separate but similar market. Its doesn't pass for science, but even with all that noise a strong results will almost always be noticeable.

I would combine these in your case to determine how long an event is given to register an effect. Once you have the data from that time period run it against your modeled out traffic prediction, week over week growth, month over month etc. Which, can then allow a meaningful comparison between agencies, and across time periods.

Pragmatism

The aspiration is to be able to provide a deep understanding of cause and effect, but it is probably not realistic. Because of how messy outside factors make your analysis, you are constantly going to run up against the question over and over again: Did this event raise volume/sales/click throughs, or would doing anything at all have had the same effect? The best advise I can give for this is set very realistic goals for what you are looking to measure. A good starting point is, within the methodology you have, which event had the largest impact. Once you have those open your aperture from there.

Summary

Once you have reasoned out all of these aspects you can go about building a general solution which can then be automated. The advantage to designing your solution in this manner is that the business logic is already built in. This will make your results much more approachable and intuitive to non-technical business leaders.

Answer 3

Edit: Warning, i leave my message but my answer seems wrong, please check out the comment below!

I'm not an expert but I guess the main problem is to answer this question:

Has an/any event affected the number of hits on a certain day?

But I don't know how to treat multiple events, so I would try to answer this question:

• Does event X affected the number of hits on a certain day?

Which can be answered using hypothesis testing with p-values (what scientist do to evaluate for instance if a medicine affects a disease or not).

By using p-values, you could determinate if the number of hits in a certain day were mere random and acceptable under normal circumstances or that they must correspond to a change in your model.

You can read more about p-values in Open Intro to Statistics Book, I've actually learn about them from there.

Then, the other parts of the problem are how to identify your events and calculate the necessary parameters to answer your question (average/median, variance, etc.) and also how to keep that up-to-date and working.

Answer 4

A few years ago (2015), Google published a research paper about the effect a specific event has in a time-series model. You can read more details here Inferring causal impact using Bayesian structural time-series models.

In this GitHub page, you can find a detailed example and description of how it works. In short,

This R package implements an approach to estimating the causal effect of a designed intervention on a time series. For example, how many additional daily clicks were generated by an advertising campaign? Answering a question like this can be difficult when a randomized experiment is not available.

You train your model with pre-event data and after-event data and you get a plot like this one

The first panel shows the data and a counterfactual prediction for the post-treatment period. The second panel shows the difference between observed data and counterfactual predictions. This is the pointwise causal effect, as estimated by the model. The third panel adds up the pointwise contributions from the second panel, resulting in a plot of the cumulative effect of the intervention.

Running the following summary(impact), you get a summary and the estimated impact the event had in your time-series data.

The same library has been ported to Python as well. For example here