Questions tagged [scalability]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

How to scale out a python based model in the cloud?

This is a question about architecture choice, tools and scalability. Let's say I have a modeling problem that runs in python and uses some libraries moduleX, moduleY, and moduleZ. We could further ...
4
votes
1answer
80 views

How can one quickly look up people from a large database?

Vocabulary Face detection: Finding all faces in an image. Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
2
votes
1answer
1k views

LSTM Time series prediction for multiple multivariate series

I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and ...
2
votes
4answers
129 views

Face Recognition (Scalability Issue)

Background I would like to build a face recognition model for registration and login for some kind of service. For example, using this approach (CNN + SVM). When a new user wants to register a ...
1
vote
1answer
541 views

DBSCAN - Space complexity of O(n)?

According to Wikipedia, "the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation ...
1
vote
1answer
476 views

Scaling DBSCAN clustering - minHash?

Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results. However, scaling this to $10$ million data points requires a creatively ...
1
vote
2answers
76 views

What is the easiest way to scale a data science project based on scikit stack?

This is an issue for all Data Scientists who have worked with this stack: python scikit-learn scipy-stats matplotlib etc. We are looking for ways to have a project already implemented in the ...
-1
votes
1answer
58 views

Distance measure calculation addresses for record linking

At the moment we use different methods for record linking locations in different datasets. Theoretically given two locations we can give a prediction on how well they match (are the same). This is ...
1
vote
0answers
221 views

Scalable training/updating of many small LSTM models

My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
2
votes
0answers
199 views

How to create Self learning data product

I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below. Professional service: Statistician involvement to build regression model or any ...
0
votes
1answer
49 views

Memory efficient structure for membership checking without false positive

The initial task can be described like this: I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable All I need to ...
2
votes
1answer
393 views

Scalable open source machine learning library written in python

I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being ...
0
votes
2answers
99 views

Scaling thousands of automated forecasts in R

Hello and thanks in advance! I'd like some advice on a scalability issue and the best way to resolve. I'm writing an algorithm in R to produce forecasts for several thousand entities. One entity ...
0
votes
1answer
250 views

When it is time to use Hadoop? [closed]

Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
4
votes
1answer
4k views

Use Cases of Neo4J and Spark GraphX

I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive. Looking at scaling to a larger data set, I am not confident No4J + Cypher will ...
15
votes
4answers
504 views

Data Science Tools Using Scala

I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
2
votes
2answers
452 views

Geospatial Social Network Analysis Visualization

I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record. What will be the best way to visualize this kind of data set?...
1
vote
0answers
107 views

scikit-learn OMP mem error

I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error. The machine has ...
10
votes
3answers
269 views

How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
9
votes
1answer
176 views

Learning signal encoding

I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
14
votes
4answers
2k views

Looking for example infrastructure stacks/workflows/pipelines

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
4
votes
2answers
527 views

How to measure execution time on distributed system

I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals. I have limited number of machines nearly ten ...
7
votes
3answers
121 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
5
votes
2answers
134 views

How to best accomplish high speed comparison of like data?

I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
11
votes
3answers
654 views

Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
87
votes
12answers
13k views

How big is big data?

Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...