Questions tagged [scalability]
The scalability tag has no usage guidance.
32
questions
0
votes
0
answers
10
views
Discussion: Is hyperparameter tuning on linux/ubuntu virtual machine a usefull approach?
In order to get an R/Shiny forecasting app ready for production, I am concerned about speeding up the model tuning process.
I am already using parallel processing on Windows 10. There are some other ...
- 143
2
votes
1
answer
127
views
Where and how to do large scale supervised machine learning?
I'm beginner in ML and I have a large dataset that has 15 features with 6M rows, so it becomes challenging to work on it locally. I can train one model locally but to perform hyper parameter tuning ...
- 35
0
votes
1
answer
51
views
How many video streams can single GPU handle for object detection
I need to detect objects from multiple video streams at realtime (or close to it, like 10 FPS).
How many GPUs do I need to detect objects using YOLOv3 or MobileNet for, say, 10 video streams?
Is it ...
- 101
2
votes
0
answers
57
views
Clustering large set of images
I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
- 21
1
vote
1
answer
81
views
Is it possible to update data and retrain just one of several data series in bigquery model
I am building something very similar to this BigQuery ML example project.
My system is different in two ways:
Firstly it will need several thousand time-series so I would prefer to use the multiple-...
- 121
2
votes
0
answers
20
views
Resource-unintensive (low complexity) methods for large-scale unsupervised clustering?
I'm working on an issue where I need to cluster user types on a scale in an unsupervised manner. I've been looking at the basics like KNN and K-means etc., but I found it hard to scale, as these ...
- 1,248
1
vote
1
answer
56
views
Ways to speed up Python code for data science purposes
Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the ...
- 2,294
4
votes
3
answers
163
views
How can one quickly look up people from a large database?
Vocabulary
Face detection: Finding all faces in an image.
Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
- 17.4k
3
votes
1
answer
3k
views
LSTM Time series prediction for multiple multivariate series
I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and ...
- 325
2
votes
5
answers
302
views
Face Recognition (Scalability Issue)
Background
I would like to build a face recognition model for registration and login for some kind of service. For example, using this approach (CNN + SVM).
When a new user wants to register a ...
- 121
1
vote
1
answer
1k
views
DBSCAN - Space complexity of O(n)?
According to Wikipedia, "the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation ...
1
vote
1
answer
742
views
Scaling DBSCAN clustering - minHash?
Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results.
However, scaling this to $10$ million data points requires a creatively ...
- 93
1
vote
2
answers
81
views
What is the easiest way to scale a data science project based on scikit stack?
This is an issue for all Data Scientists who have worked with this stack:
python
scikit-learn
scipy-stats
matplotlib
etc.
We are looking for ways to have a project already implemented in the ...
0
votes
1
answer
66
views
Distance measure calculation addresses for record linking
At the moment we use different methods for record linking locations in different datasets.
Theoretically given two locations we can give a prediction on how well they match (are the same). This is ...
- 109
1
vote
0
answers
256
views
Scalable training/updating of many small LSTM models
My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
- 33
2
votes
0
answers
202
views
How to create Self learning data product
I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below.
Professional service: Statistician involvement to build regression model or any ...
- 825
0
votes
1
answer
51
views
Memory efficient structure for membership checking without false positive
The initial task can be described like this:
I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable
All I need to ...
- 103
2
votes
1
answer
426
views
Scalable open source machine learning library written in python
I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being ...
- 571
0
votes
2
answers
101
views
Scaling thousands of automated forecasts in R
Hello and thanks in advance! I'd like some advice on a scalability issue and the best way to resolve. I'm writing an algorithm in R to produce forecasts for several thousand entities. One entity ...
- 101
0
votes
1
answer
305
views
When it is time to use Hadoop? [closed]
Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
4
votes
1
answer
6k
views
Use Cases of Neo4J and Spark GraphX
I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive.
Looking at scaling to a larger data set, I am not confident No4J + Cypher will ...
- 1,149
15
votes
4
answers
578
views
Data Science Tools Using Scala
I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
- 1,149
2
votes
2
answers
501
views
Geospatial Social Network Analysis Visualization
I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record.
What will be the best way to visualize this kind of data set?...
2
votes
1
answer
126
views
scikit-learn OMP mem error
I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error.
The machine has ...
- 21
10
votes
3
answers
339
views
How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?
Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
- 229
9
votes
1
answer
220
views
Learning signal encoding
I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
- 1,804
14
votes
4
answers
2k
views
Looking for example infrastructure stacks/workflows/pipelines
I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
- 143
4
votes
2
answers
741
views
How to measure execution time on distributed system
I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.
I have limited number of machines nearly ten ...
- 4,027
8
votes
3
answers
157
views
How to compare experiments run over different infrastructures
I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
- 4,027
6
votes
2
answers
138
views
How to best accomplish high speed comparison of like data?
I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
- 3,078
11
votes
3
answers
696
views
Can map-reduce algorithms written for MongoDB be ported to Hadoop later?
In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
- 1,383
92
votes
12
answers
19k
views
How big is big data?
Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...
- 4,027