2
$\begingroup$

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense.

Chemical Dataset

Contains thousands of rows of chemical mixtures with many columns of properties. Example properties: bromine content, density.

Inherently non-applicable data

The chemicals can be gas, liquid or solid but some properties are only applicable to samples of a certain state. An example could be viscosity in liquids, bond type (ionic, molecular, covalent) in solids or density in gas.

So far...

...all research has pointed towards methods of fixing "missing values" via column means, data imputation or something similar. There doesn't seem to be any sense in imputing the freezing point of a gas. A gas mixture does not have a freezing point. I am still in the process of data preparation and unsure how to proceed.

I am working in python and missing data is stored as NaN values. Perhaps there are some models that can deal with such NaN-values.

Side-Note:

The majority of the dataset is comprised of distillation curve datapoints (sequential data describing what percentage of a chemical sample evaporates as temperature is increased). This data is present for all samples.

Follow-up 1: Is there a model that will give me NaN values for the freezing point when I give it something that resembles a gas?

Follow-up 2: Can this be compared to image object detection where the object is partially obscured? or part of the image is corrupt?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Welcome to the site!

The usual approach to missing values is to handle them manually. There are a few algorithms which can do this automatically, such as LightGBM and XGBoost, but in most cases it's better for model performance to decide on how you should indicate that a value is missing in your data.

For example in with a Pandas dataframe in Python, I may decide to replace all NA/NaN values in a particular column which should contain positive integers with -99:

dataframe[column] = dataframe[column].fillna(-99)
Improve this answer
$\endgroup$
3
  • $\begingroup$ Replacing NaNs with fixed, "impossible" values sounds like a good idea. Is there no risk of skewing the model with these real numbers? $\endgroup$
    – DannyDannyDanny
    Commented Feb 28, 2019 at 10:21
  • $\begingroup$ As long as it is a value distinct from 'real' ones to indicate missing, it's more likely to help, especially if there is some relationship between those missing values and the target. $\endgroup$
    – Dee Carter
    Commented Mar 4, 2019 at 11:10
  • $\begingroup$ To follow on from this, another popular approach is to use the median value from that column where there are NaNs when replacing, instead of using an 'impossible' number. Try both, see what works best :) $\endgroup$
    – Dee Carter
    Commented Mar 6, 2019 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.