0
$\begingroup$

Here is a example dataset that resembles the one I am using.

example dataset

Let's call one row = one experiment.

I'm trying to predict Y based on features X1 through X7

The problem I am facing is very similar to this one, in that the target Y to predict depends on several input rows (several experiments done the same day). But what is different in my case, is that the number of experiments (so the number of rows) varies from 1 to 9.

By using the method suggested in the most upvoted answer (namely, concatenating the features for the same day so I have one row for 1 day, with 7*n_experiments features), I end up with a large dataframe of 100+ columns (I have more than 7 X features in the real dataset). Most importantly, by using this method, the last columns (representing the case of days with a lot of experiments, like 6,7,8,9 experiments on the same day) contains a lot of null values because those cases of days are pretty scarce (they are more days with 1-4 experiments than days with 8-9 experiments, so the latest columns are almost never filled with values).

So by following the method of concatenating my X features to have 1 row = 1 day, I end up with a dataframe that is, to me, useless for my prediction problem.

How would you approach this problem then?

$\endgroup$

2 Answers 2

0
$\begingroup$

You can collect various statistics for each feature (X1, X2, ..., X7). These statistics can be the mean, median, max, standard deviation, # of nulls (if this makes any sense for your problem), etc. Let's say you only consider the mean, max and min. Then you would end up with 7*3=21 features.

$\endgroup$
2
  • $\begingroup$ Actually it doesn't make sense to make summary statistics of my features as they represent physical values like thickness of material or radius of machine. Each experiment use different materials and machines, and so mixing them up doesn't make much sense in my case. $\endgroup$
    – Izem
    Commented Jul 19, 2023 at 14:19
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Jul 25, 2023 at 15:13
0
$\begingroup$

The approach I would go for is to take a step back and ask, "how are the features and target related?" This is the science part of data science; blindly searching for a magic algorithm which your dataset can suck in is not science. We have to understand the problem.

A few questions first:

  1. For each day, there is only 1 target. Is this correct? (I assume yes)

  2. Does the target of each date have any relationship with each other? (I assume no)

  3. For each date there can be multiple experiments. Does the order in which these experiments are carried out affect the target?

  4. Is there any rationale on why each experiment is carried out, e.g. 'first we test with A, if the result is X, then we do B, otherwise do C'? Or completely random?

  5. How does the results of the experiments contribute to the target together? Does each experiment contributes a part, or only a selection of experiments finally count?

  6. Is it possible that some experiments are not related to the target at all?

These questions are a start in order to find the appropriate approach. And as always, any domain knowledge is beneficial.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.