I have a very interesting dataset that I need to use for doing regression. It is production data from stainless steel production and I have about 290 input features, so I need to start reducing the dimensions.
Most of the data is batch-type, ie. there is a single record for every batch. However, some of the sub-processes have multiple cycles (that are repeated n times). On each cycle the data is measured as time-series, but the final measurement is only reported with its statistics for cycle n: mean, standard deviation, max and min. The difference here is that you have a distribution for every single record, not for the entire input variable only.
Is there a recommended approach of using distributions on your input records to your advantage?
If I could use these input records as a distribution as inputs to my selected model, I could essentially reduce the number of variables by a factor of 4 (for these sub-processes only). I looked here, here, here, but the closest question and answer was here. It does not answer my question though.
P.S. I have already trained an ANN using the 290 input features (and a suitable dimensionality reduction), but I kept these variables as separate inputs and got good results. I just think there should be a better way, since these statistics for each parameter are for the same parameter, ie. not independent.
If the above is not clear, here is the hierarchy of the data:
- For each batch
- There is a sub-process
- That goes through multiple cycles where the physical parameters are measured in real-time
- Each cycle has statistics for the physical parameters (mean, standard deviation, max and min)
- Eg. Sub-process X -> Cycle n -> parameter1_mean, parameter1_stddev, parameter1_max, parameter1_min; parameter2_mean, parameter2_stddev, etc