Skip to main content

Questions tagged [efficiency]

Efficiency, in algorithmic processing, is usually associated to resource usage. The metrics to evaluate the efficiency of a process are commonly account for execution time, memory/disk or storage requirements, network usage and power consumption.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Literature suggestions: Guarantees for GPs with subset of data

I am currently trying to solve the following problem: Given a fixed set of query points (or if possible even better: a region where I want to obtain accurate predictions), approximate the GP such that ...
L208's user avatar
  • 1
0 votes
0 answers
784 views

Choosing the number of episodes and iterations when training a RL model

I know the parameters chosen for training a RL model depend heavily on the model itself as well as the problem. Nevertheless, I am trying to train a bunch of these agents on different environments, ...
user152104's user avatar
0 votes
1 answer
13 views

feature engineering mechanism

why do we need to rescale some feature having large range I know we do it for faster rate of gradient descent ,but still how does rescaling works? and it doesn't break the model and does rescaling ...
rushi jhala's user avatar
2 votes
3 answers
2k views

What's the fastest clustering package in Python?

I'd like to perform clustering analysis on a dataset with 1,300 columns and 500,000 rows. I've seen that clustering algorithms are available in SciKit-Learn. But I'm worried that the algorithms will ...
Connor's user avatar
  • 661
0 votes
1 answer
42 views

Efficient ways of clustering for big data

I have a task which is customer segmentation with 120k users and a record of their purchases which is +3 million records of data, the approach I want to use is to use clustering algorithms like kmeans ...
F.Hazratian's user avatar
1 vote
1 answer
725 views

Why does SciKit-Learn's OneHotEncoder take so long on a Large Dataset?

I'm using an older version of SciKit-Learn, version 1.0.2, to try and OneHotEncode some data. My dataset is fairly large, 184 columns going to 311 after the ...
Connor's user avatar
  • 661
0 votes
1 answer
465 views

parallel work on KNN in python

I have a question, related to parallel work on python How I can use Processers =1,2,3... on k nearest neighbor algorithm when K=1, 2, 3,.. to find the change in time spent, speedup, and efficiency. ...
Salman Al-haddad's user avatar
0 votes
0 answers
19 views

find speedup for different number of processes

I am new to data science I need to create code to find speedup compared with the number of processes while using a k-nearest neighbor. which (k=1,2,3,4,5,6,7). this process should be after downloading ...
Salman Al-haddad's user avatar
8 votes
3 answers
1k views

Levenshtein distance vs simple for loop

I have recently begun studying different data science principles, and have had a particular interest as of late in fuzzy matching. For preface, I'd like to include smarter fuzzy searching in a ...
Jadon Steinmetz's user avatar
1 vote
0 answers
26 views

More efficient way to create frequency column based on different groupings

I have code below that calculates a frequency for each column element (respective to it's own column) and adds all five frequencies together in a column. The code works but is very slow and the ...
Curt's user avatar
  • 11
0 votes
0 answers
26 views

Inbetween CNN and MLP: neural network architecture for "close to convolutional" problem?

I am looking to approximate an (expensive to calculate precisely) forward problem using a NN. Input and output are vectors of identical length. Although not linear, the output somewhat resembles a ...
Mav's user avatar
  • 1
1 vote
0 answers
43 views

Efficient method of performing within matrix similarity

I want to compute a similarity comparison for each entry in a dataset to every other entry that is labeled as class 1 (excluding the current entry if it has a label of 1). So, consider a matrix of ...
CopyOfA's user avatar
  • 167
1 vote
1 answer
1k views

Set value for column based on two other columns in pandas dataframe

I have a dataframe that has contracts with different order dates and I need to create a new column that assign a number to each contract if it has more than one order date. For example my sample ...
rv0723's user avatar
  • 13
2 votes
0 answers
280 views

What is the difference in computational cost at inference time between object detection and semantic segmentation?

I am aware that YOLO (v1-5) is a real-time object detection model with moderately good overall prediction performance. I know that UNet and variants are efficient semantic segmentation models that are ...
JStrahl's user avatar
  • 121
1 vote
0 answers
32 views

Can I say that a trained neural network model with less parameters requires less resources during real world inference?

Let us imagine that we have two trained neural network models with different architectures (e.g., type of layers). The first model (a) uses 1D convolutional layers with fully-connected layers and has ...
user3352632's user avatar
1 vote
2 answers
36 views

Deep learning on cloud

I am trying to implement some deep learning models with large amount of data around 10gigabyte. Although, my Laptop and Collab-free crashes when it tries to load them. Do you think it worths to buy ...
mlds1337's user avatar
2 votes
1 answer
25 views

Memory efficient encoding logic for group categories

I have a huge dataset with categorical data. It is comprised of alerts having multiple properties. Each alert belongs to a group, and some even belong to multiple groups. It looks somewhat like this: ...
redguy's user avatar
  • 43
1 vote
1 answer
79 views

Ways to speed up Python code for data science purposes

Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the ...
German C M's user avatar
  • 2,724
3 votes
1 answer
396 views

How can calculate Efficiency for predictive models based on accuracy or error over time?

I was wondering if I could express the efficiency of prognostic models according to their accuracy(error, e.g. MAPE or MSE) over time [sec]. So let's imagine I have the following results for different ...
Mario's user avatar
  • 400
1 vote
0 answers
11 views

Finding synergies among observations of equal length

Assume we have a set $I$ with 20 different items (we call them $I_0$, $I_1$ up to $I_{19}$). Also we have $n$ observations $O \in I^{n\times 8}$; so each observation is a subset of $I$ with exactly 8 ...
Matmarbon's user avatar
  • 111
0 votes
1 answer
37 views

Do DBMS decrease Memory requirements?

I finished my Economics thesis using RStudio, but my script was very slow due to massive RAM consumption during the process. My Case I had a massive dataset (stock prices in daily frequency for 10 ...
Commissar Vasili Karlovic's user avatar
10 votes
2 answers
5k views

Kernel trick explanation

In support vector machines, I understand it would be computationally prohibitive to calculate a basis function at every point in the data set. However, it is possible to find this optimal solution due ...
user1717828's user avatar
22 votes
1 answer
33k views

XGBRegressor vs. xgboost.train huge speed difference?

If I train my model using the following code: ...
user1566200's user avatar
0 votes
1 answer
315 views

Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)

I am using a Dice Coefficient based function to calculate the similarity of two strings: ...
PythonNoob's user avatar
4 votes
1 answer
515 views

Computationally Inexpensive Imputation Techniques in R

I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers. I'd like to use a ...
neural-nut's user avatar
  • 1,783
0 votes
1 answer
2k views

Implementation of Association Rules in Javascript

I have implemented an interactive visualization with d3.js, javascript to explore the frequency and various combinations of co-occurring item sets. I want to complement the interactive exploration ...
ee2Dev's user avatar
  • 188
0 votes
1 answer
312 views

When it is time to use Hadoop? [closed]

Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
Aleksandro M Granda's user avatar
3 votes
1 answer
42 views

Timing sequence in MapReduce

I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution ...
syed's user avatar
  • 31
4 votes
1 answer
2k views

Scikit Learn Logistic Regression Memory Leak

I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a ...
DaveA's user avatar
  • 181
10 votes
3 answers
353 views

How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
Bridgeburners's user avatar
4 votes
2 answers
2k views

How to implement Brown Clustering Algorithm in O(|V|k^2)

I am trying to implement the Brown Clustering Algorithm. Paper details: "Class-Based n-gram Models of Natural Language" by Brown et al The algorithm is supposed to in ...
Ofir's user avatar
  • 123
13 votes
2 answers
2k views

Is FPGrowth still considered "state of the art" in frequent pattern mining?

As far as I know the development of algorithms to solve the Frequent Pattern Mining (FPM) problem, the road of improvements have some main checkpoints. Firstly, the Apriori algorithm was proposed in ...
Rubens's user avatar
  • 4,117
5 votes
2 answers
931 views

How to speedup message passing between computing nodes

I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire ...
Rubens's user avatar
  • 4,117
14 votes
4 answers
2k views

Looking for example infrastructure stacks/workflows/pipelines

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
chrshmmmr's user avatar
  • 143
11 votes
3 answers
8k views

Best languages for scientific computing [closed]

It seems as though most languages have some number of scientific computing libraries available. Python has Scipy Rust has <...
ragingSloth's user avatar
  • 1,824
8 votes
2 answers
376 views

Filtering spam from retrieved data

I once heard that filtering spam by using blacklists is not a good approach, since some user searching for entries in your dataset may be looking for particular information from the sources blocked. ...
Rubens's user avatar
  • 4,117
8 votes
3 answers
181 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
Rubens's user avatar
  • 4,117
65 votes
6 answers
43k views

When is a Model Underfitted?

Logic often states that by underfitting a model, it's capacity to generalize is increased. That said, clearly at some point underfitting a model cause models to become worse regardless of the ...
blunders's user avatar
  • 1,932
6 votes
2 answers
142 views

How to best accomplish high speed comparison of like data?

I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
Steve Kallestad's user avatar
12 votes
2 answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
mbbce's user avatar
  • 347
10 votes
1 answer
1k views

What is the most efficient data indexing technique

As we all know, there are some data indexing techniques, using by well-known indexing apps, like Lucene (for java) or Lucene.NET (for .NET), MurMurHash, B+Tree etc. For a No-Sql / Object Oriented ...
sihirbazzz's user avatar
10 votes
4 answers
226 views

Why is it hard to grant efficiency while using libraries?

Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend ...
Rubens's user avatar
  • 4,117
5 votes
2 answers
725 views

Running an R script programmatically [closed]

I have an R script that generates a report based on the current contents of a database. This database is constantly in flux with records being added/deleted many times each day. How can I ask my ...
rnorberg's user avatar
  • 203
94 votes
12 answers
20k views

How big is big data?

Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...
Rubens's user avatar
  • 4,117