3
$\begingroup$

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear and another to appear in the data as a whole. Occasionally, a widget will just disappear and so the dataset is incomplete, invalidating the whole dataset for that day.

What I am looking for is a method of comparing the current set of widgets with another set of widgets to detect if any widgets are missing. I am not trying to create the values, just identify that they are missing. I could do time-series, but that feels like overkill on so many widgets and there are multiple attributes on which data might be missing. I was hoping for something more set based that might account for the regular changes in widgets but detecting the unusual dropouts. I am sure I just need to adjust the way I am thinking about the problem.

Any ideas would be much appreciated?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ concept is good and any sample data? $\endgroup$
    – sai saran
    Nov 15, 2018 at 3:49
  • 1
    $\begingroup$ Unfortunately, it is proprietary data, but I'll try to document a proxy of the data. $\endgroup$
    – Skiddles
    Nov 15, 2018 at 14:29
  • 1
    $\begingroup$ Something I don't get: what happens to your data when a widget "disappears"? $\endgroup$
    – anymous.asker
    Nov 17, 2018 at 18:47
  • 1
    $\begingroup$ When the widgets disappear, part of the whole is missing. $\endgroup$
    – Skiddles
    Nov 18, 2018 at 1:25

1 Answer 1

Reset to default
0
$\begingroup$

One option is hashing, assigning a numerical value to each widget. The best hashing option is if each widget had a unique id, something like a serial number. If a widget does not have inherent unique, then a hash value can be created by applying a hash function to the features of widget.

After creating hash values for each widget, set comparison can be used to see if any hashed values are different between the two sets of widgets.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.