0
$\begingroup$

I'm performing my first NN with my own data and while I was already tuning the parameters I stumbled over an aspect which confuses me now such that I'm not sure what is right and what is wrong..

Given this (head of the) data df_train1_raw:

                     timestamp       A_phsA
2018-02-05 14:00:00 1.517839e+09    856.436487
2018-02-05 15:00:00 1.517843e+09    859.653339
2018-02-05 16:00:00 1.517846e+09    836.635463
2018-02-05 17:00:00 1.517850e+09    801.097284
2018-02-05 18:00:00 1.517854e+09    794.855960
                       (...)

The timestamp column is basically the index just converted into numeric so I can use these time information for the nn model.

Goal: Predict A_phsA on timestamp

First, I create the train and test sets:

# Prepare data
X_train1_raw = df_train1_raw.values
y_train1_raw = X_train1_raw

# Split data into appropriate sets
## Standardize and scale data

scaler = StandardScaler()
tscv = TimeSeriesSplit(n_splits = 5)
pyplot.figure(1)
index = 1

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();

Split-plots

This looks reasonable. When I plot the loss and val_loss values of the nn later it also looks reasonable.

But what struggles me actually is this line at the beginning:

y_train1_raw = X_train1_raw

I can't tell if it is plain stupid or if I can't get my head around it anymore. The reason is, when I look for example at KFold:

X =  list(range(10))
print (X)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
y = [x*x for x in X]
print (y)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

kf = KFold(n_splits=5)
X = np.array(X)
y = np.array(y)
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    print(“X_test: ", X_test)

they have different X and y. Which makes sense I would say. But when I adjust my code accordingly

X_train1_raw = df_train1_raw.iloc[:, 1].values
y_train1_raw = df_train1_raw.iloc[:, 2].values

I get an error when performing

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();

ValueError: Expected 2D array, got 1D array instead: array=[1.5178392e+09 1.5178428e+09 1.5178464e+09 1.5178500e+09 1.5178536e+09

And I don't understand why. Which part is not correct?

edit: Is it scaler.fit_transform() ?

$\endgroup$
3
  • $\begingroup$ Despite any programming mistake, the assumption "y_train1_raw = X_train1_raw" is.. dumb, right? $\endgroup$
    – Ben
    Oct 8 '19 at 14:23
  • 1
    $\begingroup$ Yes, you only want your target feature to be your independent variable, not your dependent variables. $\endgroup$
    – Danny
    Oct 8 '19 at 14:29
  • $\begingroup$ I just figured out, why I initially did that: I am about to implement an Autoencoder and so far, besides the topology, it is done by using model.fit(X, X). Then it should be correct, or? $\endgroup$
    – Ben
    Oct 8 '19 at 17:42
1
$\begingroup$
fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
     X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(X_train1_raw[test_index].reshape(-1,1))
     y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(y_train1_raw[test_index].reshape(-1,1))
     pyplot.subplot(510 + index)
     pyplot.plot(X_train1[:, 1])
     pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
     index +=1
pyplot.show();

This should fix your code and this happens because the Standard Scaler expects inputs as samples where each sample is an array of n_features to transform. The reshape(-1,1) does that for you if you have got a single feature to scale.

$\endgroup$
1
  • $\begingroup$ awesome, this worked. Thank you! Just have to scramble with the rest of the code now.. :) $\endgroup$
    – Ben
    Oct 8 '19 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.