Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [scalability]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Discussion: Is hyperparameter tuning on linux/ubuntu virtual machine a usefull approach?

In order to get an R/Shiny forecasting app ready for production, I am concerned about speeding up the model tuning process. I am already using parallel processing on Windows 10. There are some other ...
user avatar
2 votes
1 answer
127 views

Where and how to do large scale supervised machine learning?

I'm beginner in ML and I have a large dataset that has 15 features with 6M rows, so it becomes challenging to work on it locally. I can train one model locally but to perform hyper parameter tuning ...
user avatar
  • 35
0 votes
1 answer
51 views

How many video streams can single GPU handle for object detection

I need to detect objects from multiple video streams at realtime (or close to it, like 10 FPS). How many GPUs do I need to detect objects using YOLOv3 or MobileNet for, say, 10 video streams? Is it ...
user avatar
  • 101
2 votes
0 answers
57 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
user avatar
1 vote
1 answer
81 views

Is it possible to update data and retrain just one of several data series in bigquery model

I am building something very similar to this BigQuery ML example project. My system is different in two ways: Firstly it will need several thousand time-series so I would prefer to use the multiple-...
user avatar
  • 121
2 votes
0 answers
20 views

Resource-unintensive (low complexity) methods for large-scale unsupervised clustering?

I'm working on an issue where I need to cluster user types on a scale in an unsupervised manner. I've been looking at the basics like KNN and K-means etc., but I found it hard to scale, as these ...
user avatar
  • 1,248
1 vote
1 answer
56 views

Ways to speed up Python code for data science purposes

Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the ...
user avatar
  • 2,294
4 votes
3 answers
163 views

How can one quickly look up people from a large database?

Vocabulary Face detection: Finding all faces in an image. Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
user avatar
  • 17.4k
3 votes
1 answer
3k views

LSTM Time series prediction for multiple multivariate series

I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and ...
user avatar
  • 325
2 votes
5 answers
302 views

Face Recognition (Scalability Issue)

Background I would like to build a face recognition model for registration and login for some kind of service. For example, using this approach (CNN + SVM). When a new user wants to register a ...
user avatar
  • 121
1 vote
1 answer
1k views

DBSCAN - Space complexity of O(n)?

According to Wikipedia, "the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation ...
user avatar
1 vote
1 answer
742 views

Scaling DBSCAN clustering - minHash?

Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results. However, scaling this to $10$ million data points requires a creatively ...
user avatar
  • 93
1 vote
2 answers
81 views

What is the easiest way to scale a data science project based on scikit stack?

This is an issue for all Data Scientists who have worked with this stack: python scikit-learn scipy-stats matplotlib etc. We are looking for ways to have a project already implemented in the ...
user avatar
0 votes
1 answer
66 views

Distance measure calculation addresses for record linking

At the moment we use different methods for record linking locations in different datasets. Theoretically given two locations we can give a prediction on how well they match (are the same). This is ...
user avatar
  • 109
1 vote
0 answers
256 views

Scalable training/updating of many small LSTM models

My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
user avatar
  • 33
2 votes
0 answers
202 views

How to create Self learning data product

I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below. Professional service: Statistician involvement to build regression model or any ...
user avatar
0 votes
1 answer
51 views

Memory efficient structure for membership checking without false positive

The initial task can be described like this: I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable All I need to ...
user avatar
  • 103
2 votes
1 answer
426 views

Scalable open source machine learning library written in python

I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being ...
user avatar
  • 571
0 votes
2 answers
101 views

Scaling thousands of automated forecasts in R

Hello and thanks in advance! I'd like some advice on a scalability issue and the best way to resolve. I'm writing an algorithm in R to produce forecasts for several thousand entities. One entity ...
user avatar
  • 101
0 votes
1 answer
305 views

When it is time to use Hadoop? [closed]

Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
user avatar
4 votes
1 answer
6k views

Use Cases of Neo4J and Spark GraphX

I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive. Looking at scaling to a larger data set, I am not confident No4J + Cypher will ...
user avatar
15 votes
4 answers
578 views

Data Science Tools Using Scala

I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
user avatar
2 votes
2 answers
501 views

Geospatial Social Network Analysis Visualization

I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record. What will be the best way to visualize this kind of data set?...
user avatar
2 votes
1 answer
126 views

scikit-learn OMP mem error

I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error. The machine has ...
user avatar
  • 21
10 votes
3 answers
339 views

How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
user avatar
9 votes
1 answer
220 views

Learning signal encoding

I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
user avatar
  • 1,804
14 votes
4 answers
2k views

Looking for example infrastructure stacks/workflows/pipelines

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
user avatar
  • 143
4 votes
2 answers
741 views

How to measure execution time on distributed system

I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals. I have limited number of machines nearly ten ...
user avatar
  • 4,027
8 votes
3 answers
157 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
user avatar
  • 4,027
6 votes
2 answers
138 views

How to best accomplish high speed comparison of like data?

I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
user avatar
11 votes
3 answers
696 views

Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
user avatar
92 votes
12 answers
19k views

How big is big data?

Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...
user avatar
  • 4,027