How can I improve the precision of my regression model?

Question

For a uni project I'm trying to fit a keras sequential model to a few thousand given datapoints with are approximately in the range $y \in [-0.03,0.03]$ and have the domain $x \in (0, 409.6)$.

I'm using the following model definition

Sequential([
  Input(shape=(1,)),
  feature_normalizer,
  layers.Dense(512, activation='relu'),
  layers.Dense(512, activation='relu'),
  layers.Dense(512, activation='relu'),
  layers.Dense(512, activation='relu'),
  layers.Dense(512, activation='relu'),
  layers.Dense(1, activation="linear")
])

a batch size of 32 and the Adam Optimizer with a learning rate ot 0.0001.

Using this configuration and 10000 epochs some parts of the domain are really poorly fitted: whereas the approximation is better in other areas:

My loss history looks like this

Is there anything I can do to improve the accuracy of my model? Or should I just train it with more epochs. Any tips would by appreciated :)

Answer 1

have you tried feature scaling ? standardizing your features before training the model and try working with regularization techniques like l1,l2

try using different algorithms such as xgboost regressor, random forest, gradient boosting as ensemble methods.

Answer 2

The way you are drawing the points makes it look like you are dealing with a time series forecasting problem. If that is the case I would suggest changing from the feedforward NN architecture you are currently implementing to a recurrent NN type of architecture.

A starting example taken from the tensorflow tutorials is here

Answer 3

Few things you can do

1. Try with the Sequence models architectures
2. Remove stationarity from the time series if possible?
3. With current architectures, try adding more layers too.