1
$\begingroup$

Our csv contains 36 columns

  • 1 date time column collected every 30 mins
  • 3 variables (count,latency,Totaltime) x 10 Features(user io, serverio ,concurrency ..etc ) Of different data points from the server: example user io count,user io latency, userio totaltime.
  • other 5 are more static : server name,ip address..

We know when there is a peak in 1 of the features(userio, serverio) there is a issue in the server .. there is also times when it can be a combination of the features.

In past one year data we can see if latency in any 1 feature : the outlier has some chances to become a bigger outlier(meaning we see the outlier keep increasing in 30 mins bucket and the biggest one matches our issue time) that causes a issue on the server.

after loads of blogs, was able to get something like below working for 1 feature.. But we want to combine the other features as well to give us a consolidated result.

# from sklearn.ensemble import IsolationForest
# model =  IsolationForest(contamination=0.004)
# model.fit(df[['Concurrency Latency']])
# df['outliers']=pd.Series(model.predict(df[['Concurrency Latency']])).apply(lambda x: 'yes' if (x == -1) else 'no' )
# df.query('outliers=="yes"')
# fig = px.scatter(df.reset_index(), x='STime', y='Concurrency Latency', color='outliers')
# fig.update_xaxes(
#     rangeslider_visible=True,
# )
# fig.show()

Any idea which library can help to achieve this . Need it work with timeseries to understand trends and seasonality.

Background : new in ML, have started python some time back.

Any help is appreciated. The goal is to catch the outlier when it started off with a small alert.. so we are prepared few hours before hand.

$\endgroup$

1 Answer 1

0
$\begingroup$

Prophet is quite well adapted to multivariate time series with or without seasonality, including events.

Here are several codes that could be useful:

https://grabngoinfo.com/multivariate-time-series-forecasting-with-seasonality-and-holiday-effect-using-prophet-in-python/

https://medium.com/mlearning-ai/multivariate-time-series-forecasting-using-fbprophet-66147f049e66

https://www.kaggle.com/code/bagavathypriya/multivariate-time-series-using-fb-prophet

https://www.analyticsvidhya.com/blog/2022/04/an-end-to-end-guide-on-time-series-forecasting-using-fbprophet/

However, Prophet only works on time-series data and cannot handle missing data very well.

Therefore, you can use Random Forest which can handle any type of data, but the time-series predictions could be worse than Prophet:

https://www.analyticsvidhya.com/blog/2021/06/random-forest-for-time-series-forecasting/

$\endgroup$
5
  • $\begingroup$ i had taken a look in prophet.. correct me if im wrong.. but.. my data doesnt have a target y column.. We want to catch if a outlier is getting raised looking at historical data.. from my multivariate columns.. i cannot say one of them is Y.. $\endgroup$
    – trent
    Commented Dec 17, 2022 at 15:18
  • $\begingroup$ A good way to detect outliers is to use good prediction tools and see if the real value is too far from the predicted one. The advantage of using Prophet is that you can make good predictions (in most cases) and see if the situation is normal. $\endgroup$ Commented Dec 17, 2022 at 16:37
  • $\begingroup$ does this make sense : i predict each features latency.. using the other 2 variables. So when a new data comes, i provide the 2 variables.. predict latency. and validate against the new one. If not normal its outlier ? Repeat 10 times for each feature ? $\endgroup$
    – trent
    Commented Dec 17, 2022 at 16:42
  • $\begingroup$ Yes, it makes sense from my point of view. You just have to be careful in choosing the right threshold value to detect an outlier to avoid false positives. Maybe it is possible by calculating the differences' mean value of the former outliers. $\endgroup$ Commented Dec 17, 2022 at 16:49
  • $\begingroup$ Does it answer your question? If not, please let me know to provide additional information or other solutions. $\endgroup$ Commented Dec 27, 2022 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.