0
$\begingroup$

We want to know how to apply Feature engineering(or any other ways) to time series data to capture a specific pattern like the blue line shows, the raw data is: time stamp, and value. And we got a few samples with labelled data like y representing if the current time is in that desired phase. But performance is bad when applied to test data. Now thinking about applying a regression model to it in a sliding window and how close the value in that sliding window fits the desired pattern linearly.

enter image description here

Below is some information we know before the training, that most of the label==True fell in the green boxed area. So that's why we are happy to pay more attention to feature engineering to capture this pattern.

enter image description here

The ideal result I want is a classification prediction: if the value is in the charging phase

TLDR version:

input(ts, value, label) --> feature eng --> model --> output: pred_label

This is what some of our time-series data looks like:

enter image description here

enter image description here enter image description here enter image description here

Some motifs it found and the motif we want : enter image description here enter image description here

$\endgroup$
4
  • $\begingroup$ What algorithm are you using? Can't you multiply red & blue values to build a new feature that could answer your need? $\endgroup$ Nov 21, 2022 at 8:40
  • $\begingroup$ Hi @NicolasMartin, the figure just for illustration of the idea, in the real world we only have two columns of data, 1 timestamp 2. power values. Which is in blue shape. We want to find some battery patterns from power consumption. And we've created some lagged features, and date time features. So I want to derive more features from those two raw feature. $\endgroup$
    – Yiffany
    Nov 21, 2022 at 9:23
  • $\begingroup$ @NicolasMartin, we have a small amount of data that is labelled, e.g. constant charging and faded charging phases are labelled. But performance is bad when apply to test data. So... $\endgroup$
    – Yiffany
    Nov 21, 2022 at 9:32
  • $\begingroup$ @NicolasMartin, Can refer to my other question for dataset from here $\endgroup$
    – Yiffany
    Nov 21, 2022 at 9:39

2 Answers 2

1
$\begingroup$

This seems to be a perfect use case for Matrix Profile. With this approach you won't need to do any feature engineering at all, but just use the raw input power series and its related Matrix Profile to identify the "charging phase" motif.

Edit in response to comments:

  1. it can be manually set up using your given, known patterns as a label - I would refer you to the paper "Using Weakly Labelled Time Series to Predict Outcomes" in the linked page.
  2. if you get this to work, it is a classification method in its own right so you will not need to extract features for another classification algorithm.
$\endgroup$
4
  • $\begingroup$ Thanks, does it works well in noised data as well? such those motifs are not exactly the same in every subspecies? $\endgroup$
    – Yiffany
    Nov 23, 2022 at 22:30
  • $\begingroup$ @Yiffany It does work with noisy data. You can threshold the Market Profile value for a potential match to allow for such noisy data. $\endgroup$ Nov 25, 2022 at 9:46
  • $\begingroup$ After some experiments with MP, got some questions 1).I can find some visualization but not full motif after running the auto-analysis function, is there a way to manually set it up? 2) how to transfer motifs into features so we can input data into some classification model? Thanks $\endgroup$
    – Yiffany
    Nov 29, 2022 at 0:53
  • $\begingroup$ could you provide more info? $\endgroup$
    – Yiffany
    Nov 29, 2022 at 3:50
0
$\begingroup$

First, some noise reduction is necessary to make data clean and comparable.

Therefore, you should define what clean data is and how to identify noise efficiently.

It could be done thanks to Kalman filters, but not sure if they work with false peaks noise.

Then, perhaps some data normalization could be necessary to make subspecies comparable.

About shape detection, you can use models based on 1D CNN for example, but you have to define the starting and ending points for each shape detection.

https://www.kaggle.com/code/rsmits/cnn-1d-model/script

$\endgroup$
2
  • $\begingroup$ Thanks, It's hard to find start and end points now. There are 50k non-labelled data but we only got 3 labelled data for training. $\endgroup$
    – Yiffany
    Nov 25, 2022 at 4:19
  • $\begingroup$ It's difficult to help because your problem seems very complex (a lot of uncertainty) and I can't connect the dots. My best advice is to cut down into smaller problems and deal with one problem at a time with priorities (ex: noise reduction, shape detection, lack of labeled data,...). Otherwise, you wouldn't be able to reach your objective fast enough. $\endgroup$ Nov 25, 2022 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.