1
$\begingroup$

I want to predict bacteria plate count in the water from time series(around 10000 values in a row) of water temperature on a one minute granularity, and other daily climate data including min and max temperature, rainfall, solar exposure, day_of_the_week etc for a sequence of 20 days each before the sample was collected for test. There are 700 different locations in the building.

My initial approach was to join all time series of the same location into one row/one series. I used RNN LSTM but the testing accuracy is not so good(65%-75%). I guess the reason could be that joining different time series(eg. water temperature and rainfall) may have led to gradient explosion due to the contrasting nature of data, thus I tried gradient clipping and there was no improvement in testing accuracy. I understand that in general RNN is good for time series data and CNN is good for image preprocessing. But my case is slightly different as I'm having multiple time series joined together to form one time series. I'm wondering if CNN or GRU would be a better model to use.

The data looks like this:

     x0  x1    x2   x3   x4   x5   ... x10000 Date       max_t1...max_t20 min_t1...min_t20 rf1... rf20 sol1...sol20 d_wk1... d_wk20
1    40 31.05  25.5 25.5 25.5 25   ...  33    2019-01-01 26.2  ...        20.2  ...         0 ...      32.4...       4 ...
2    35  35.75 36.5 36.5 36.5 36.5 ...  29    2019-01-03 24.8. ...        18.4  ...         0 ...      28.8          6 ...
⋮     ⋮   ⋮      ⋮    ⋮    ⋮     ⋮          ⋮

where

max_t1, ..., max_t20 represent max temperature from day1 to day20(Date day);

min_t1, ..., min_t20 represent min temperature from day1 to day20(Date day);

rf1, ..., rf20 represent rainfall from day1 to day20(Date day);

sol1, ..., sol20 represent solar exposure from day1 to day20(Date day).

d_wk1, ..., d_wk20 represent which day it was of the week from day1 to day20(Date day)

These are all the features beside water temperature data(so there are around 100 new columns in total).


Update Of Question:

I have checked many CNN case studies online but very few are on non-image data. I fit the data using CNN on Keras but the accuracy level is very low(<60%). What could go wrong? Is there anything I can do? A friend suggested XGBoost- is there a way to apply the algorithm to my data? Any idea is appreciated.

So far I've used something like this:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(merge.iloc[:,1:10170], merge[['Result_cat','Result_cat1']].values, test_size=0.2) 

import numpy as np
import keras
import tensorflow 
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.optimizers import SGD
from tensorflow.python.keras.optimizer_v2.adam import Adam

model = Sequential()

model.add(Dense(1000, input_shape=(10167,))) 
model.add(Activation("relu"))
model.add(Dense(512, activation='softmax')) 
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax')) 


model.fit(X_train, y_train, batch_size=10000, epochs=1000)
score = model.evaluate(X_test, y_test, batch_size=10000)
$\endgroup$
  • $\begingroup$ Did you gather the climate data on your own or do you use a weather database? $\endgroup$ – Ben Nov 6 '19 at 6:35
  • $\begingroup$ @nilsinelabore what is your data like? Can you share samples of the data you have (and the target values)? $\endgroup$ – serali Nov 6 '19 at 8:39
  • $\begingroup$ @Ben I downloaded the daily climate date from Bureau of Meteorology $\endgroup$ – nilsinelabore Nov 6 '19 at 21:37
  • $\begingroup$ @serali hi I've added a sample of the data in my original question $\endgroup$ – nilsinelabore Nov 6 '19 at 21:54
  • $\begingroup$ Can't you download the data with a different time interval? $\endgroup$ – Ben Nov 7 '19 at 5:44
0
$\begingroup$

You mention having 2 types of data:

  • Water temperature with a minute-granularity
  • Climate data taken 20 days prior

I'm not sure what you mean with "joining the series into one row/series", but I don't think that it make sense to combine these two types as the time-frame is just too different. In this case it would probably be better to combine different layers to deal with the discrepancy. I would use a RNN layer for the water temperature and experiment with different layers (for example regular dense layers) for the remaining input, treating it as fixed inputs (like the location variable) instead of dynamic time-step input like the water temperature.

Since the climate data is 20 days prior, I don't think that it'll really influence the water temperature on a minute basis; the climate data is more like a foundation, similar to the location variable.

$\endgroup$
  • $\begingroup$ thank you for the answer.By "joining the series into one row/series" I mean lay out various time series side by side, eg. water temp time series, rainfall time series etc. I agree that time-frame is just too different when combining time different features and there's great disparity in sample size. I want to mention that climate data consists of time series of max, min daily temperature, daily rainfall and solar exposure, each is a time series. (The hypothesis is that the climate data of all 20days prior to the date when sample was collected would affect bacteria growth) $\endgroup$ – nilsinelabore Nov 6 '19 at 21:31
  • $\begingroup$ My questions are: 1. Is there a way to model the data if I join all features in a row(before that, it is valid to do this?)? 2. Can I put the different input data in separate layers of RNN?(I'm new to neural network) 3. If yes, how can I do it? Could you maybe share some online examples? Thanks $\endgroup$ – nilsinelabore Nov 6 '19 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.