0
$\begingroup$

I'm trying to apply machine learning to pharmaceutical manufacturing to predict whether batches of drug products manufactured are good or not. for the sake of relatability, let's use coffee brewing as an analogous process. Let's imagine that I'm trying to predict the acidity of the coffee that I've brewed.

The dataset that I have contains features such as temperature of water, stirring speed and pressure that are constantly measured (say..on a per second basis) over a variable amount of time (the first cup may be brewed in 5 minutes, the second in 10 etc).

What kind of preprocessing should I perform on such a multidimensional dataset? One stumbling block is that for each observation, the duration is different, which may complicate dimension reduction? Once preprocessed, is there any specific model that would suit the task at hand? I'm looking at something like a regression but alternatively, classifiers seem to be fine as well if I split the acidity(pH) into "<5.5" or ">5.5"?

I hope to get some general directions and if you can paste a few links to texts or examples that'll be good! Also, I'm more familiar with python and scikit learn, so if you can point me in the right section in the documentation that'll be great too!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

I don't know much about coffee or pharmaceuticals but I think the widely varying time samples is a problem. If I brewed one batch of coffee for a minute and another for 5 hours, I'm pretty sure the 5 hour batch would come out burnt-tasting in all cases.

Can you break the samples up into cohorts by duration and then train on each cohort? You'd end up with a model for the "1 minute batch", a model for the "1 hour batch", etc.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I've definitely tried brewing coffee for more than 5 hours and it turned out fine, but its cold brew ;) Anyway back to topic, by cohorts you mean split the say...one sample that took 12 hours to manufacture/brew and convert it into 12 samples? Or do you mean make 12 features with the average value of each? $\endgroup$
    – Impuls3H
    Mar 31, 2017 at 15:06
  • $\begingroup$ Fair enough about cold brew - how about traditional percolators? Maybe this isn't relevant to pharmaceuticals, but the fact that you mentioned the samples have widely different durations makes me worry that your variables wouldn't have the same meaning across the data set. By cohorts, I meant split your problem up into several parts. Say you have 500 observations. I would plot a histogram of the durations and then hopefully some buckets will pop out so you can train a model on "batches 0-1 hours", "batches 1-5 hours", and "batches > 5 hours". $\endgroup$
    – CalZ
    Mar 31, 2017 at 18:26
  • $\begingroup$ Ah I see what you mean. For the actual dataset that I'm working on, it's not that variable. It ranges from 24 hours to the odd 48 hours. On the training for different buckets, does it work like bootstrapping the samples together? $\endgroup$
    – Impuls3H
    Apr 1, 2017 at 3:47
  • $\begingroup$ OK, in that case I would make sure the duration is in there as a feature. I would also see how it is distributed and see if it makes sense to normalize or put it into buckets. $\endgroup$
    – CalZ
    Apr 3, 2017 at 11:53
  • $\begingroup$ Ok I'll try it out and see how it goes? Thanks CalZ! $\endgroup$
    – Impuls3H
    Apr 4, 2017 at 23:40
0
$\begingroup$

You can use panda date time function to create various features representing time; Total time, day etc. Then use these features as your independent variable to train your model

Improve this answer
$\endgroup$
3
  • $\begingroup$ Setting timestamps isn't a problem, I'm stuck on handling the features instead. if I don't do any form of preprocessing or feature engineering, the dimension of the dataset will be too high, bigger than the number of samples I have. for my actual dataset, I have hundreds of samples and hours of per second data, which means I have more features than samples. Any idea on how to reduce the dimension of the dataset? $\endgroup$
    – Impuls3H
    Mar 31, 2017 at 14:40
  • $\begingroup$ Use lasso sparse model to reject few of the unnecessary features and you can also do feature engineering using your domain knowledge to combine few features or just discard few of them. $\endgroup$
    – Vivek Khetan
    Mar 31, 2017 at 14:42
  • $\begingroup$ Lasso sparse model? Ah I'll need to read up on that! for feature engineering, are there any general heuristics? I've previously tried taking averages of the entire period and adding another feature on the total duration. that didn't work out well though, I guess too much information was lost. A higher temperature at the start would definitely affect it more then at the end so simple averaging didn't help. $\endgroup$
    – Impuls3H
    Mar 31, 2017 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.