I want to predict bacteria plate count in the water from time series(around 10000 values in a row) of water temperature on a one minute granularity, and other daily climate data including min and max temperature, rainfall, solar exposure, day_of_the_week etc for a sequence of 20 days each before the sample was collected for test. There are 700 different locations in the building.
My initial approach was to join all time series of the same location into one row/one series. I used RNN LSTM but the testing accuracy is not so good(65%-75%). I guess the reason could be that joining different time series(eg. water temperature and rainfall) may have led to gradient explosion due to the contrasting nature of data, thus I tried gradient clipping and there was no improvement in testing accuracy. I understand that in general RNN is good for time series data and CNN is good for image preprocessing. But my case is slightly different as I'm having multiple time series joined together to form one time series. I'm wondering if CNN or GRU would be a better model to use.
The data looks like this:
x0 x1 x2 x3 x4 x5 ... x10000 Date max_t1...max_t20 min_t1...min_t20 rf1... rf20 sol1...sol20 d_wk1... d_wk20 1 40 31.05 25.5 25.5 25.5 25 ... 33 2019-01-01 26.2 ... 20.2 ... 0 ... 32.4... 4 ... 2 35 35.75 36.5 36.5 36.5 36.5 ... 29 2019-01-03 24.8. ... 18.4 ... 0 ... 28.8 6 ... ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
max_t1, ..., max_t20 represent max temperature from day1 to day20(
min_t1, ..., min_t20 represent min temperature from day1 to day20(
rf1, ..., rf20 represent rainfall from day1 to day20(
sol1, ..., sol20 represent solar exposure from day1 to day20(
d_wk1, ..., d_wk20 represent which day it was of the week from day1 to day20(
These are all the features beside water temperature data(so there are around 100 new columns in total).
Update Of Question:
I have checked many CNN case studies online but very few are on non-image data. I fit the data using CNN on Keras but the accuracy level is very low(<60%). What could go wrong? Is there anything I can do? A friend suggested
XGBoost- is there a way to apply the algorithm to my data? Any idea is appreciated.
So far I've used something like this:
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(merge.iloc[:,1:10170], merge[['Result_cat','Result_cat1']].values, test_size=0.2) import numpy as np import keras import tensorflow from keras.models import Sequential from keras.layers import Dense, Flatten from keras.optimizers import SGD from tensorflow.python.keras.optimizer_v2.adam import Adam model = Sequential() model.add(Dense(1000, input_shape=(10167,))) model.add(Activation("relu")) model.add(Dense(512, activation='softmax')) model.add(Dense(256, activation='relu')) model.add(Dropout(0.5)) model.add(Dense(2, activation='softmax')) model.fit(X_train, y_train, batch_size=10000, epochs=1000) score = model.evaluate(X_test, y_test, batch_size=10000)