Which learning algorithms to use in what order - dimensionality reduction, bayesian network structure, regression?

Question

The data is a huge set of observations of dozens of variables, all (potentially, somehow) related to a dichotomous outcome variable, and all (potentially) correlated to each other, or to unknown / unobserved things.

I think that "merely" applying logistic regressions or any related extension or improvement (such as Classification And Regression Trees etc.) might not be sufficient, because there is a "basic" bayesian network structure that seems to exist (based on expert opinion) - this means that the observed variables are somehow describing distinct nodes in a bayesian network, but the exact formula HOW they describe these nodes is unclear, as is the eventual predictive power of these nodes towards the outcome.

Here is this exact problem in a highly simplified example:

We have two runners who are going to compete in a 400m sprint against each other. We have a dataset of observations of thousands of such competitions with two runners, each observation containing, for each of the two runners, (a) age, (b) size, (c) number of training runs in the past 6 weeks and (d) average mileage per training run in the past 6 weeks.

Now, we believe that (a) and (b) will determine (e) "talent for sprinting" and (c) and (d) will determine (f) "current form", and that (e) and (f) together will give the probability of winning - but we do not know HOW (a) and (b) determine (e) or how (c) and (d) determine (f) (e.g., is it "age times size" or is it "age times 2 while age lower than 45" etc.). Further, we also do not know how (e) and (f) together determine the probability of winning...

Is there any "known" approach to such a problem, i.e. a way to determine the formula that connects the initial variables to the later variables that determine the outcome, and then further determining the predictive power of the resulting network?

Or is that unnecessary (i.e. using the original variables only in some algorithm - which one? - should give the same result)? Or is it impossible (too many unknown things)? What should I do / read about / learn to solve this problem?

And in addition, is there a way to CHECK whether my original assumption (that (a) and (b) determine (e) etc.) was correct, e.g. via some sort of clustering algorithm, PCA, ...?

What would you do to approach such a question?

Thank you very much indeed in advance for helping a rather new data science learner by pointing me into the right direction!

Answer 1

Is there any "known" approach to such a problem, i.e. a way to determine the formula that connects the initial variables to the later variables that determine the outcome, and then further determining the predictive power of the resulting network?

This is kinda a question about modelling approach as well as different types of models. I think that it is always best to start with simpler models, such as simple logistic regression. In this case you are trying to predict which runner will win, so that is a 0/1 outcome. So you can put all of your variables into a logistic regression model and see how it performs in terms of accuracy. If you have a lot of predictors, you can add a penalty term like L1 or L2. The simplest model serves as a baseline by which to compare more complex models, which might not always work better.

The formulation you describe seems to like a very common one. These are often called Latent Variable models or others might prefer to call them Bayesian network models. The idea is that we cannot measure a particular quantity directly, but we can use indirect measure to estimate them. So if your sense is that "talent for sprinting" and "current form" are better predictors for predicting who wins the race, then you can use a method for Latent Variables. I have not run those in a while but there are common algorithms like the EM algorithm or Markov Chain Monte Carlo sampling.

So by fitting the two different models you can see how much accuracy you gain by adding the latent variable structure versus making no assumptions about the structure. There are methods to try and estimate the bayesian network structure, but they often require a lot of data. In a case like this you might not have enough data to a neural net or something.

I don't think dimensional reduction would necessarily help in this case. You do seem to have time-series data, which you could include directly into the logistic regression equation. You might even be able to try an LSTM or sequence model if you have enough data.

Those are some ideas off the top of my head, but I think the idea of starting with simple models, benchmarking them, and then trying more complex models is the best way to learn and to avoid getting lost. Haha. Happy running.

Answer 2

There exist approaches for learning the structure of Bayesian networks.

They just don't seem to be very popular, so I don't think you'll find them in standard toolkits.

Answer 3

So we face a similar problem when dealing with Word Embedding. There we generate co-relations of words with each other and then using the generated relations, try to predict the related words.

I should say that there could be a statistical way to check the your assumption, but I would rather try and code my idea to check its correctness by checking it on some test-set. This way, we can narrow down our approach's drawbacks and then think in that specific direction.

Whenever we have correlated data, we use algorithm like t-SNE for dimensionality reduction. You can then easily visualize it or use it in any way as per you requirement.

Answer 4

You can use Graph based machine learning: Stellar