Skip to main content
added 171 characters in body
Source Link
Erwan
  • 25.9k
  • 3
  • 15
  • 37

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.

From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.

I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful to know what it looks like, in particular to choose an appropriate regression method.

[edit] About dealing with noise.

The standard statistical way to deal with noise is to do nothing at all :)

How much data do you have? Because if you have enough data, statistically the noise should balance itself out. For instance if you have 100 such graphs for a fixed $n$, the mean optimal $p$ should be close enough to the true optimal $p$. Even if for a few values of $n$ the data doesn't contain the optimal value of $p$, across sufficiently many values of $n$ this noise is unlikely to be significant. What matters is for the regression model to correctly represent the general trend, and not to capture the small variations due to noise.

I would start with a graph made of one boxplot for each value of $n$ and varying $n$, where the set of values represented in a boxplot are all the mnimum $p$ in the data for this value of $n$.

[edit 2] Another interesting visualization: you can represent your full data as a heatmap with $n$/$p$ on the X/Y axis and the colour based on the value of $f(n,p$)$.

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.

From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.

I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful to know what it looks like, in particular to choose an appropriate regression method.

[edit] About dealing with noise.

The standard statistical way to deal with noise is to do nothing at all :)

How much data do you have? Because if you have enough data, statistically the noise should balance itself out. For instance if you have 100 such graphs for a fixed $n$, the mean optimal $p$ should be close enough to the true optimal $p$. Even if for a few values of $n$ the data doesn't contain the optimal value of $p$, across sufficiently many values of $n$ this noise is unlikely to be significant. What matters is for the regression model to correctly represent the general trend, and not to capture the small variations due to noise.

I would start with a graph made of one boxplot for each value of $n$ and varying $n$, where the set of values represented in a boxplot are all the mnimum $p$ in the data for this value of $n$.

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.

From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.

I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful to know what it looks like, in particular to choose an appropriate regression method.

[edit] About dealing with noise.

The standard statistical way to deal with noise is to do nothing at all :)

How much data do you have? Because if you have enough data, statistically the noise should balance itself out. For instance if you have 100 such graphs for a fixed $n$, the mean optimal $p$ should be close enough to the true optimal $p$. Even if for a few values of $n$ the data doesn't contain the optimal value of $p$, across sufficiently many values of $n$ this noise is unlikely to be significant. What matters is for the regression model to correctly represent the general trend, and not to capture the small variations due to noise.

I would start with a graph made of one boxplot for each value of $n$ and varying $n$, where the set of values represented in a boxplot are all the mnimum $p$ in the data for this value of $n$.

[edit 2] Another interesting visualization: you can represent your full data as a heatmap with $n$/$p$ on the X/Y axis and the colour based on the value of $f(n,p$)$.

added 856 characters in body
Source Link
Erwan
  • 25.9k
  • 3
  • 15
  • 37

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.

From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.

I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful to know what it looks like, in particular to choose an appropriate regression method.

[edit] About dealing with noise.

The standard statistical way to deal with noise is to do nothing at all :)

How much data do you have? Because if you have enough data, statistically the noise should balance itself out. For instance if you have 100 such graphs for a fixed $n$, the mean optimal $p$ should be close enough to the true optimal $p$. Even if for a few values of $n$ the data doesn't contain the optimal value of $p$, across sufficiently many values of $n$ this noise is unlikely to be significant. What matters is for the regression model to correctly represent the general trend, and not to capture the small variations due to noise.

I would start with a graph made of one boxplot for each value of $n$ and varying $n$, where the set of values represented in a boxplot are all the mnimum $p$ in the data for this value of $n$.

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.

From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.

I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful to know what it looks like, in particular to choose an appropriate regression method.

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.

From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.

I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful to know what it looks like, in particular to choose an appropriate regression method.

[edit] About dealing with noise.

The standard statistical way to deal with noise is to do nothing at all :)

How much data do you have? Because if you have enough data, statistically the noise should balance itself out. For instance if you have 100 such graphs for a fixed $n$, the mean optimal $p$ should be close enough to the true optimal $p$. Even if for a few values of $n$ the data doesn't contain the optimal value of $p$, across sufficiently many values of $n$ this noise is unlikely to be significant. What matters is for the regression model to correctly represent the general trend, and not to capture the small variations due to noise.

I would start with a graph made of one boxplot for each value of $n$ and varying $n$, where the set of values represented in a boxplot are all the mnimum $p$ in the data for this value of $n$.

Source Link
Erwan
  • 25.9k
  • 3
  • 15
  • 37

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.

From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.

I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful to know what it looks like, in particular to choose an appropriate regression method.