3
$\begingroup$

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear and another to appear in the data as a whole. Occasionally, a widget will just disappear and so the dataset is incomplete, invalidating the whole dataset for that day.

What I am looking for is a method of comparing the current set of widgets with another set of widgets to detect if any widgets are missing. I am not trying to create the values, just identify that they are missing. I could do time-series, but that feels like overkill on so many widgets and there are multiple attributes on which data might be missing. I was hoping for something more set based that might account for the regular changes in widgets but detecting the unusual dropouts. I am sure I just need to adjust the way I am thinking about the problem.

Any ideas would be much appreciated?

$\endgroup$
4
  • 1
    $\begingroup$ concept is good and any sample data? $\endgroup$
    – sai saran
    Nov 15 '18 at 3:49
  • 1
    $\begingroup$ Unfortunately, it is proprietary data, but I'll try to document a proxy of the data. $\endgroup$
    – Skiddles
    Nov 15 '18 at 14:29
  • 1
    $\begingroup$ Something I don't get: what happens to your data when a widget "disappears"? $\endgroup$ Nov 17 '18 at 18:47
  • 1
    $\begingroup$ When the widgets disappear, part of the whole is missing. $\endgroup$
    – Skiddles
    Nov 18 '18 at 1:25
0
$\begingroup$

One option is hashing, assigning a numerical value to each widget. The best hashing option is if each widget had a unique id, something like a serial number. If a widget does not have inherent unique, then a hash value can be created by applying a hash function to the features of widget.

After creating hash values for each widget, set comparison can be used to see if any hashed values are different between the two sets of widgets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.