Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [data-stream-mining]

An activity that seeks patterns in a continuous stream of data elements, usually involving summarizing the stream in some way.

Filter by
Sorted by
Tagged with
2 votes
0 answers
42 views

How to build this data pipeline?

I don't have much experience in data engineering, so I'm here to ask for advice. I am working on a project which consists of building a dashboard for the IT department of a bank. the dashboard should ...
Wissem Boujlida's user avatar
0 votes
0 answers
75 views

How to visualize a data drift?

I want to show that my data distribution changes between data windows. Is it enough to visualize the mean and variance for every window? Is there any other solution? thank you
Imen F's user avatar
  • 21
2 votes
1 answer
319 views

What is the differenc between Real concept drift, virtual concept drift and feature drift

As far as I know, the real concept drift is caused by changes in the decision boundary while virtual drift occurs because of changes in data distribution. Some researchers mention that virtual drift ...
Imen F's user avatar
  • 21
2 votes
1 answer
29 views

Newbie questions: real-time clustering of messages

I'm very much a newbie in NLP, so please accept my apologies if this is an obvious question, the wrong place to ask it or any other error I could be making. I am considering using NLP for some subset ...
Yoric's user avatar
  • 121
1 vote
1 answer
67 views

How to do an incremental update for the mean and standard deviation of tensor data?

I have a big dataset (some 400Gb) consisting of tensor data (shape is $(600, 600, 10)$) and I want to normalize this dataset before feeding it to a neural network but this dataset can't fit in my ...
Toshi Mint's user avatar
1 vote
1 answer
32 views

Cannot figure out how to make THIS particular data set long-form (theory, not even code)

I am a tableau developer, but I know Python, stats, and, in short, I think you all will be best able to solve my problem. There is a universal filter on Facility. This means that any dataset/sheet ...
Caitlin's user avatar
  • 21
0 votes
1 answer
44 views

How to create a cctv monitoring ML solution to classify images?

I have a cctv that records video of products that move over a factory line. I want to stream the cctv into a ML solution that will classify the products, and if it detects that it is a certain type ...
variable's user avatar
  • 187
1 vote
2 answers
49 views

What would be the right tool for gathering data structure analytics in a data stream?

We are processing pretty big number of JSON objects (hundreds of thousands daily) and we need to gather insights about the processed objects. We are interested in gathering the following analytics of ...
Fedor's user avatar
  • 11
0 votes
1 answer
14 views

How would you count top repetitions in a stream (grouping and summing similar strings too)

I would look in an arbitrary window (already relative so unbased), count each element (repetitions) then rank them. Continuously look, group and sum similar "strings". There are two options: ...
bacloud14's user avatar
  • 453
0 votes
1 answer
183 views

How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
user3225309's user avatar
1 vote
1 answer
99 views

reduction of sample from videos sample

Well, I post the same question in the main stack before finding the right place, sorry. A friend of mine is working with more than a 100 videos as sample for his neural network. Each video last more ...
T.Dunglas's user avatar
3 votes
0 answers
36 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
I Sui's user avatar
  • 57
0 votes
2 answers
62 views

Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
I Sui's user avatar
  • 57
2 votes
1 answer
4k views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
Amir's user avatar
  • 123
1 vote
0 answers
22 views

modelling multirotor aerodynamics using datalogs of flights

I am trying to find a vector that would describe the effects of wind on a multirotor. I have a bunch of datalogs from a single frame of multirotor and am of the mind to digg. The idea is that during ...
Karl Uibo's user avatar
  • 111
5 votes
1 answer
10k views

Real time noise removal using Savitzky-Golay Method

I would like to ask if Savitzky-Golay can be implemented on real-time data. I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone ...
Abdullah Nazir's user avatar
1 vote
2 answers
139 views

Online learning w/ feature weighting/adjusting

Let's say I have a supervised learning problem with a sequence of features and labels. First, I learn on the training data and then I decide to stream in data, point by point and do online learning. ...
Jeremy's user avatar
  • 13
2 votes
0 answers
41 views

Is there a counting sketch optimized for intersections?

Popular counting sketches(loglog, hyperloglog, etc) feature natural union operations. Are there any known counting sketches that feature natural intersection operations?
Newbie's user avatar
  • 121
3 votes
1 answer
112 views

Designing a ConvNet to facilitate game playing

For fun I want to design a convolutional neural net to recognize enemy NPCs in a first person shooter. I have captured 100 jpegs of the npcs as well as 100 jpegs of not-NPCs. I have successfully ...
aquagremlin's user avatar
2 votes
0 answers
65 views

local regression with streaming data

From a data stream i'm receiving a pair of measurements consisting of a current consumption and a current percentage every second. By accumulating the consumption over time it will represent ...
R. Doe's user avatar
  • 251
2 votes
1 answer
496 views

Analysis of Real-Time Bidding

I'm totally new to the topic of real-time bidding in which I know Machine Learning algorithms are used pretty often. Can somebody explain me the system in a plain language i.e. a language for a non-...
DanielWelke's user avatar
4 votes
2 answers
2k views

Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib

I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose: Storm alone and implement streaming naive bayes on my own in storm ...
Raman's user avatar
  • 141
12 votes
2 answers
148 views

Opensource tools for help in mining stream of leader board scores

Consider a stream containing tuples (user, new_score) representing users' scores in an online game. The stream could have 100-1,000 new elements per second. The ...
Tahir Akhtar's user avatar
6 votes
1 answer
189 views

Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?

(Note: Pulled this question from the list of questions in Area51, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able ...
blunders's user avatar
  • 1,932