Questions tagged [data-stream-mining]
An activity that seeks patterns in a continuous stream of data elements, usually involving summarizing the stream in some way.
23
questions
0
votes
0
answers
73
views
How to visualize a data drift?
I want to show that my data distribution changes between data windows. Is it enough to visualize the mean and variance for every window? Is there any other solution? thank you
- 21
2
votes
1
answer
164
views
What is the differenc between Real concept drift, virtual concept drift and feature drift
As far as I know, the real concept drift is caused by changes in the decision boundary while virtual drift occurs because of changes in data distribution. Some researchers mention that virtual drift ...
- 21
2
votes
1
answer
27
views
Newbie questions: real-time clustering of messages
I'm very much a newbie in NLP, so please accept my apologies if this is an obvious question, the wrong place to ask it or any other error I could be making.
I am considering using NLP for some subset ...
- 121
1
vote
1
answer
58
views
How to do an incremental update for the mean and standard deviation of tensor data?
I have a big dataset (some 400Gb) consisting of tensor data (shape is $(600, 600, 10)$) and I want to normalize this dataset before feeding it to a neural network but this dataset can't fit in my ...
- 13
1
vote
1
answer
31
views
Cannot figure out how to make THIS particular data set long-form (theory, not even code)
I am a tableau developer, but I know Python, stats, and, in short, I think you all will be best able to solve my problem.
There is a universal filter on Facility. This means that any dataset/sheet ...
- 21
0
votes
1
answer
41
views
How to create a cctv monitoring ML solution to classify images?
I have a cctv that records video of products that move over a factory line.
I want to stream the cctv into a ML solution that will classify the products, and if it detects that it is a certain type ...
- 187
1
vote
2
answers
49
views
What would be the right tool for gathering data structure analytics in a data stream?
We are processing pretty big number of JSON objects (hundreds of thousands daily) and we need to gather insights about the processed objects. We are interested in gathering the following analytics of ...
- 11
0
votes
1
answer
14
views
How would you count top repetitions in a stream (grouping and summing similar strings too)
I would look in an arbitrary window (already relative so unbased), count each element (repetitions) then rank them. Continuously look, group and sum similar "strings".
There are two options:
...
- 453
0
votes
1
answer
161
views
How to find anomalies in (almost) constant stream of data?
I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database.
More detailed explanation would be that process starts, collects data (which takes some ...
- 111
1
vote
1
answer
92
views
reduction of sample from videos sample
Well, I post the same question in the main stack before finding the right place, sorry.
A friend of mine is working with more than a 100 videos as sample for his neural network. Each video last more ...
- 23
3
votes
0
answers
34
views
High dimensional data stream summarization and processing
Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection?
In fact, I investigated the different methods for data ...
- 57
0
votes
2
answers
61
views
Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?
I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
- 57
2
votes
1
answer
4k
views
What are the approaches to aggregate categorical variables?
I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem:
...
- 123
1
vote
0
answers
22
views
modelling multirotor aerodynamics using datalogs of flights
I am trying to find a vector that would describe the effects of wind on a multirotor. I have a bunch of datalogs from a single frame of multirotor and am of the mind to digg.
The idea is that during ...
- 111
4
votes
1
answer
9k
views
Real time noise removal using Savitzky-Golay Method
I would like to ask if Savitzky-Golay can be implemented on real-time data.
I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone ...
- 161
1
vote
2
answers
138
views
Online learning w/ feature weighting/adjusting
Let's say I have a supervised learning problem with a sequence of features and labels. First, I learn on the training data and then I decide to stream in data, point by point and do online learning. ...
- 13
2
votes
0
answers
39
views
Is there a counting sketch optimized for intersections?
Popular counting sketches(loglog, hyperloglog, etc) feature natural union operations. Are there any known counting sketches that feature natural intersection operations?
- 121
3
votes
1
answer
111
views
Designing a ConvNet to facilitate game playing
For fun I want to design a convolutional neural net to recognize enemy NPCs in a first person shooter. I have captured 100 jpegs of the npcs as well as 100 jpegs of not-NPCs. I have successfully ...
- 133
2
votes
0
answers
63
views
local regression with streaming data
From a data stream i'm receiving a pair of measurements consisting of a current consumption and a current percentage every second. By accumulating the consumption over time it will represent ...
- 251
2
votes
1
answer
494
views
Analysis of Real-Time Bidding
I'm totally new to the topic of real-time bidding in which I know Machine Learning algorithms are used pretty often.
Can somebody explain me the system in a plain language i.e. a language for a non-...
- 163
4
votes
2
answers
2k
views
Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib
I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose:
Storm alone and implement streaming naive bayes on my own in storm ...
- 141
12
votes
2
answers
146
views
Opensource tools for help in mining stream of leader board scores
Consider a stream containing tuples (user, new_score) representing users' scores in an online game. The stream could have 100-1,000 new elements per second. The ...
- 315
6
votes
1
answer
184
views
Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?
(Note: Pulled this question from the list of questions in Area51, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able ...
- 1,922