I would start by graphing the time variable vs other variables and looking for trends.
In this case there is a periodic weekly trend and a long term upwards trend. So you would want to encode two time variables:
There are several common time frames that trends occur over:
For RNNs (e.g., LSTMs and GRUs), the layer input is a list of timesteps, and each timestep is a feature tensor. That means that you could have a input tensor like this (in Pythonic notation):
# Input tensor to RNN
# Timestep 1
[ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
# Timestep 2
[ temperature_in_paris, value_of_nasdaq,...
In general time series are not really different from other machine learning problems - you want your test set to 'look like' your training set, because you want the model you learned on your training set to still be appropriate for your test set. That's the important underlying concept regarding stationarity. Time series have the additional complexity that ...
Statement 1 is correct, statement 2 is correct, but requires elaboration, and statement 3 is incorrect for seasonal ARIMA:
The following might point you in the right direction but hopefully you'll get a few more answers with more depth in the arena of LSTM.
You mention that you have tried both algorithms and that you are simply trying to figure out which ...
I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks.
The package is in Python and its name is pyod. It is published in JMLR.
It has multiple algorithms for following individual approaches:
Linear Models for Outlier Detection (PCA,vMCD,vOne-Class, and SVM)
Proximity-Based Outlier Detection ...
You can specify the unit of a pandas to_datetime call.
Stolen from here:
# assuming `df` is your data frame and `date` is your column of timestamps
df['date'] = pandas.to_datetime(df['date'], unit='s')
Should work with integer datatypes, which makes sense if the unit is seconds since the epoch.
After reading your question, I became curious about the topic of time series clustering and dynamic time warping (DTW). So, I have performed a limited search and came up with basic understanding (for me) and the following set of IMHO relevant references (for you). I hope that you'll find this useful, but keep in mind that I have intentionally skipped ...
The problem with models like KNN is that they do not take into account seasonality (time-dependent variations in trend). To take those into account, you should use Time Series analysis.
For count data, such as yours, you can use generalized linear auto-regressive moving average models (GLARMA). Fortunately, there is an R package that implements them (...
Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system.
To construct a reinforcement learning (RL) problem where it is worth using an RL prediction or control algorithm, then you need to identify some components:
An environment that be in one of many states that ...
Seaborn uses inter-quartile range to detect the outliers. What you need to do is to reproduce the same function in the column you want to drop the outliers. It's quite easy to do in Pandas.
If we assume that your dataframe is called df and the column you want to filter based AVG, then
Q1 = df['AVG'].quantile(0.25)
Q3 = df['AVG'].quantile(0.75)
IQR = Q3 - ...
This is a fun problem. This is a time series and from this time series you want to identify the trigger of a certain event. So it is a binary classification problem. Based on the information from the specified window will a spike occur? Yes or No.
The first step is to set up your database. What you will have is a set of instances (which can have some overlap ...
I wouldn't consider DTW to be outdated at all. In 2006 Xi et al. showed that
[...] many algorithms have been proposed for the problem of time series
classification. However, it is clear that one-nearest-neighbor with
Dynamic Time Warping (DTW) distance is exceptionally difficult to
The results of this paper are summarized in the book "...
I would not use the word "best" but LSTM-RNN are very powerful when dealing with timeseries, simply because they can store information about previous values and exploit the time dependencies between the samples. That said, it is definitely worth going for it.
It has been proven that their performance can be boosted significantly if they are ...
Fancy animations are cool
I was very impressed when I saw this animation of the discourse git repository. They used Gourse which is specifically for git. But it may give ideas about how to represent the dynamics of growth.
You can create animations with matplotlib
This stackoverflow answer seems to point at a python/networkx/matplotlib solution.
But D3.js ...
Random forest (as well as most of supervised learning models) accepts a vector $x=(x_1,...x_k)$ for each observation and tries to correctly predict output $y$. So you need to convert your training data to this format. The following pandas-based function will help:
import pandas as pd
def table2lags(table, max_lag, min_lag=0, separator='_'):
""" Given a ...
Directly, this is not possible. However, if you model it in a different way you can get out confidence intervals. You could instead of a normal regression approach it as estimating a continuous probability distribution. By doing this for every step you can plot your distribution. Ways to do this are Kernel Mixture Networks (https://janvdvegt.github.io/2017/...
Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar. Using an autoencoder lets you re-represent high dimensional points in a lower-dimensional space. It doesn't do clustering per se - but it is a useful preprocessing step for a secondary clustering step. You would map each input vector $x_i$ to a ...
I think Christopher's answers above are entirely sensible. As an alternate approach (or perhaps just in addition to the advise he's given), I might start by just visualizing the data a bit to try get a rough sense of what's going on.
If you haven't already done this, you might try adding a date's month and day of week as features -- if you end up sticking ...
For another alternative approach, you can take a look at the PyMC library.
There is a good gist created by Fonnesbeck which walks you through the HMM creation.
And if you become really eager about the PyMC, there is an awesome open-source book about Bayesian Modeling. It does not explicitly describe Hidden Markov Processes, but it gives a very good tutorial ...
Multivariate time series is an active research topic you will find a lot of recent paper tackling the subject.
To answer your questions, you can use a single RNN. You can input one value for each time step. Nothing keeps you from adding another value at each time step (if your sensor are synchronized). Your model will then learn how to classify with a two ...
LSTM layers require data of a different shape.
From your description, I understand the starting dataset to have 3125 rows and 1000 columns, where each row is one time-step. The target variable should then have 3125 rows and 1 column, where each value can be one of three possible values. So it sounds like you're doing a classification problem. To check this ...
One more thing to consider, beyond everything that Ben Haley said, is to convert to user local time. For example, if you are trying to predict something that occurs around 8pm for all users, if you look at UTC time, it will be harder to predict from.
when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ?
For your timeseries analysis you should do both: get to the highest granularity possible with the daily dataset, and also repeat the analysis with the monthly dataset. With the monthly dataset you have 120 data points, which is sufficient to get ...
You are facing a very common problem: handling imbalanced data. For neural networks, typical procedures are:
Having the proper metrics: global accuracy should not be used.
Oversampling the minority class: randomly generate replicas of the minority class until the imbalance disappears.You can also perform data augmentation on the minority class. Synthetic ...
Divide the data into windows and find features for those windows like autocorrelation coefficients, wavelets, etc. and use those features for learning.
For example, if you have temperature and pressure data, break it down to individual parameters and calculate features like number of local minima in that window and others, and use these features for your ...
There are two solutions that are worth looking at:
InfluxDB is an open source database platform specifically designed for time series data. The platform includes many optimized functions related to time and you can collect data on any interval and compute rollups/aggregations when reporting. The company recently launched a query app called Chronograf. I ...
As an update on this question, I believe the accepted answer is not the best as of 2017.
As suggested in comments by Kyle, hmmlearn is currently the library to go with for HMMs in Python.
Several reasons for this:
The up-to-date documentation, that is very detailed and includes tutorial
The _BaseHMM class from which custom subclass can inherit for ...
Based on all the good answers of this thread, I wrote a library to condition on auxiliary inputs. It abstracts all the complexity and has been designed to be as user-friendly as possible:
Hope it helps!
As for stateful LSTM and its understanding, refer to here. Quoting an answer from there:
"I’m given a big sequence (e.g. Time Series) and I split it into smaller sequences to construct my input matrix X. Is it possible that the LSTM may find dependencies between the sequences?
No it’s not possible unless you go for the stateful LSTM. Most of the problems ...