Questions tagged [scalability]
The scalability tag has no usage guidance.
27
questions
1
vote
1answer
36 views
Ways to speed up Python code for data science purposes
Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the ...
0
votes
0answers
13 views
How to scale out a python based model in the cloud?
This is a question about architecture choice, tools and scalability.
Let's say I have a modeling problem that runs in python and uses some libraries moduleX, moduleY, and moduleZ. We could further ...
4
votes
1answer
90 views
How can one quickly look up people from a large database?
Vocabulary
Face detection: Finding all faces in an image.
Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
2
votes
1answer
2k views
LSTM Time series prediction for multiple multivariate series
I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and ...
2
votes
5answers
166 views
Face Recognition (Scalability Issue)
Background
I would like to build a face recognition model for registration and login for some kind of service. For example, using this approach (CNN + SVM).
When a new user wants to register a ...
1
vote
1answer
639 views
DBSCAN - Space complexity of O(n)?
According to Wikipedia, "the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation ...
1
vote
1answer
532 views
Scaling DBSCAN clustering - minHash?
Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results.
However, scaling this to $10$ million data points requires a creatively ...
1
vote
2answers
77 views
What is the easiest way to scale a data science project based on scikit stack?
This is an issue for all Data Scientists who have worked with this stack:
python
scikit-learn
scipy-stats
matplotlib
etc.
We are looking for ways to have a project already implemented in the ...
-1
votes
1answer
59 views
Distance measure calculation addresses for record linking
At the moment we use different methods for record linking locations in different datasets.
Theoretically given two locations we can give a prediction on how well they match (are the same). This is ...
1
vote
0answers
228 views
Scalable training/updating of many small LSTM models
My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
2
votes
0answers
199 views
How to create Self learning data product
I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below.
Professional service: Statistician involvement to build regression model or any ...
0
votes
1answer
49 views
Memory efficient structure for membership checking without false positive
The initial task can be described like this:
I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable
All I need to ...
2
votes
1answer
395 views
Scalable open source machine learning library written in python
I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being ...
0
votes
2answers
99 views
Scaling thousands of automated forecasts in R
Hello and thanks in advance! I'd like some advice on a scalability issue and the best way to resolve. I'm writing an algorithm in R to produce forecasts for several thousand entities. One entity ...
0
votes
1answer
262 views
When it is time to use Hadoop? [closed]
Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
4
votes
1answer
5k views
Use Cases of Neo4J and Spark GraphX
I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive.
Looking at scaling to a larger data set, I am not confident No4J + Cypher will ...
15
votes
4answers
524 views
Data Science Tools Using Scala
I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
2
votes
2answers
457 views
Geospatial Social Network Analysis Visualization
I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record.
What will be the best way to visualize this kind of data set?...
2
votes
0answers
110 views
scikit-learn OMP mem error
I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error.
The machine has ...
10
votes
3answers
283 views
How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?
Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
9
votes
1answer
178 views
Learning signal encoding
I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
14
votes
4answers
2k views
Looking for example infrastructure stacks/workflows/pipelines
I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
4
votes
2answers
568 views
How to measure execution time on distributed system
I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.
I have limited number of machines nearly ten ...
8
votes
3answers
126 views
How to compare experiments run over different infrastructures
I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
6
votes
2answers
136 views
How to best accomplish high speed comparison of like data?
I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
11
votes
3answers
661 views
Can map-reduce algorithms written for MongoDB be ported to Hadoop later?
In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
88
votes
12answers
14k views
How big is big data?
Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...
