I want to predict bacteria plate count in the water from time series(around 10000 values in a row) of water temperature on a one minute granularity, and other daily climate data including min and max temperature, rainfall, solar exposure, day_of_the_week etc for a sequence of 20 days each before the sample was collected for test. There are 700 different locations in the building.
My initial approach was to join all time series of the same location into one row/one series. I used RNN LSTM but the testing accuracy is not so good(65%-75%). I guess the reason could be that joining different time series(eg. water temperature and rainfall) may have led to gradient explosion due to the contrasting nature of data, thus I tried gradient clipping and there was no improvement in testing accuracy. I understand that in general RNN is good for time series data and CNN is good for image preprocessing. But my case is slightly different as I'm having multiple time series joined together to form one time series. I'm wondering if CNN or GRU would be a better model to use.
The data looks like this:
x0 x1 x2 x3 x4 x5 ... x10000 Date max_t1...max_t20 min_t1...min_t20 rf1... rf20 sol1...sol20 d_wk1... d_wk20
1 40 31.05 25.5 25.5 25.5 25 ... 33 2019-01-01 26.2 ... 20.2 ... 0 ... 32.4... 4 ...
2 35 35.75 36.5 36.5 36.5 36.5 ... 29 2019-01-03 24.8. ... 18.4 ... 0 ... 28.8 6 ...
⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
where
max_t1, ..., max_t20 represent max temperature from day1 to day20(Date
day);
min_t1, ..., min_t20 represent min temperature from day1 to day20(Date
day);
rf1, ..., rf20 represent rainfall from day1 to day20(Date
day);
sol1, ..., sol20 represent solar exposure from day1 to day20(Date
day).
d_wk1, ..., d_wk20 represent which day it was of the week from day1 to day20(Date
day)
These are all the features beside water temperature data(so there are around 100 new columns in total).
Update Of Question:
I have checked many CNN case studies online but very few are on non-image data. I fit the data using CNN on Keras but the accuracy level is very low(<60%). What could go wrong? Is there anything I can do? A friend suggested XGBoost
- is there a way to apply the algorithm to my data? Any idea is appreciated.
So far I've used something like this:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(merge.iloc[:,1:10170], merge[['Result_cat','Result_cat1']].values, test_size=0.2)
import numpy as np
import keras
import tensorflow
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.optimizers import SGD
from tensorflow.python.keras.optimizer_v2.adam import Adam
model = Sequential()
model.add(Dense(1000, input_shape=(10167,)))
model.add(Activation("relu"))
model.add(Dense(512, activation='softmax'))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))
model.fit(X_train, y_train, batch_size=10000, epochs=1000)
score = model.evaluate(X_test, y_test, batch_size=10000)