Questions tagged [scalability]
The scalability tag has no usage guidance.
33 questions
2
votes
0
answers
415
views
Is there a need to use scaling for Age attribute?
What is the good way working with 'Age' attribute? Don't touch it or should it be scaled?
Below photo shows my results 'Before' and 'After' standardization.
- 121
1
vote
1
answer
351
views
How does compute required scale with number of model parameters?
GPT-3 has 175 billion parameters, required ~$3.114 * 10^{23}$ FLOPS, and took approximately one month to train on ~10k Tesla V100 GPUs. It seems commonly stated that the brain has the equivalent of ~...
2
votes
1
answer
140
views
Where and how to do large scale supervised machine learning?
I'm beginner in ML and I have a large dataset that has 15 features with 6M rows, so it becomes challenging to work on it locally. I can train one model locally but to perform hyper parameter tuning ...
- 35
1
vote
1
answer
258
views
How many video streams can single GPU handle for object detection
I need to detect objects from multiple video streams at realtime (or close to it, like 10 FPS).
How many GPUs do I need to detect objects using YOLOv3 or MobileNet for, say, 10 video streams?
Is it ...
- 111
3
votes
0
answers
160
views
Clustering large set of images
I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
- 31
1
vote
1
answer
328
views
Is it possible to update data and retrain just one of several data series in bigquery model
I am building something very similar to this BigQuery ML example project.
My system is different in two ways:
Firstly it will need several thousand time-series so I would prefer to use the multiple-...
- 121
2
votes
0
answers
23
views
Resource-unintensive (low complexity) methods for large-scale unsupervised clustering?
I'm working on an issue where I need to cluster user types on a scale in an unsupervised manner. I've been looking at the basics like KNN and K-means etc., but I found it hard to scale, as these ...
- 1,359
1
vote
1
answer
88
views
Ways to speed up Python code for data science purposes
Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the ...
- 2,734
4
votes
3
answers
365
views
How can one quickly look up people from a large database?
Vocabulary
Face detection: Finding all faces in an image.
Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
- 19.2k
4
votes
1
answer
3k
views
LSTM Time series prediction for multiple multivariate series
I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and ...
- 345
2
votes
5
answers
362
views
Face Recognition (Scalability Issue)
Background
I would like to build a face recognition model for registration and login for some kind of service. For example, using this approach (CNN + SVM).
When a new user wants to register a ...
- 121
2
votes
1
answer
2k
views
DBSCAN - Space complexity of O(n)?
According to Wikipedia, "the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation ...
1
vote
1
answer
1k
views
Scaling DBSCAN clustering - minHash?
Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results.
However, scaling this to $10$ million data points requires a creatively ...
- 103
1
vote
2
answers
84
views
What is the easiest way to scale a data science project based on scikit stack?
This is an issue for all Data Scientists who have worked with this stack:
python
scikit-learn
scipy-stats
matplotlib
etc.
We are looking for ways to have a project already implemented in the ...
0
votes
1
answer
103
views
Distance measure calculation addresses for record linking
At the moment we use different methods for record linking locations in different datasets.
Theoretically given two locations we can give a prediction on how well they match (are the same). This is ...
- 109
2
votes
0
answers
276
views
Scalable training/updating of many small LSTM models
My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
- 53
2
votes
0
answers
205
views
How to create Self learning data product
I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below.
Professional service: Statistician involvement to build regression model or any ...
- 865
0
votes
1
answer
57
views
Memory efficient structure for membership checking without false positive
The initial task can be described like this:
I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable
All I need to ...
- 103
2
votes
1
answer
442
views
Scalable open source machine learning library written in python
I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being ...
- 571
0
votes
2
answers
115
views
Scaling thousands of automated forecasts in R
Hello and thanks in advance! I'd like some advice on a scalability issue and the best way to resolve. I'm writing an algorithm in R to produce forecasts for several thousand entities. One entity ...
- 101
0
votes
1
answer
312
views
When it is time to use Hadoop? [closed]
Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
4
votes
1
answer
6k
views
Use Cases of Neo4J and Spark GraphX
I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive.
Looking at scaling to a larger data set, I am not confident No4J + Cypher will ...
- 1,169
15
votes
4
answers
612
views
Data Science Tools Using Scala
I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
- 1,169
2
votes
2
answers
527
views
Geospatial Social Network Analysis Visualization
I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record.
What will be the best way to visualize this kind of data set?...
3
votes
1
answer
145
views
scikit-learn OMP mem error
I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error.
The machine has ...
- 31
10
votes
3
answers
353
views
How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?
Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
- 229
9
votes
1
answer
267
views
Learning signal encoding
I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
- 1,834
14
votes
4
answers
2k
views
Looking for example infrastructure stacks/workflows/pipelines
I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
- 143
4
votes
2
answers
894
views
How to measure execution time on distributed system
I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.
I have limited number of machines nearly ten ...
- 4,117
8
votes
3
answers
185
views
How to compare experiments run over different infrastructures
I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
- 4,117
6
votes
2
answers
145
views
How to best accomplish high speed comparison of like data?
I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
- 3,128
11
votes
3
answers
711
views
Can map-reduce algorithms written for MongoDB be ported to Hadoop later?
In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
- 1,393
94
votes
12
answers
20k
views
How big is big data?
Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...
- 4,117