Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [efficiency]

Efficiency, in algorithmic processing, is usually associated to resource usage. The metrics to evaluate the efficiency of a process are commonly account for execution time, memory/disk or storage requirements, network usage and power consumption.

1
vote
0answers
28 views

Efficiency of a supercomputer. Alternatives to rent one to process large amount of data

I want to live from poker as I don't find a job as geologist, and I am making a trainning system to train myself. I need to process some data to calculate the mathematical chances for each 270.000 ...
0
votes
1answer
26 views

Do DBMS decrease Memory requirements?

I finished my Economics thesis using RStudio, but my script was very slow due to massive RAM consumption during the process. My Case I had a massive dataset (stock prices in daily frequency for 10 ...
3
votes
2answers
3k views

Kernel trick explanation

In support vector machines, I understand it would be computationally prohibitive to calculate a basis function at every point in the data set. However, it is possible to find this optimal solution due ...
11
votes
1answer
17k views

XGBRegressor vs. xgboost.train huge speed difference?

If I train my model using the following code: ...
0
votes
1answer
131 views

Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)

I am using a Dice Coefficient based function to calculate the similarity of two strings: ...
4
votes
1answer
310 views

Computationally Inexpensive Imputation Techniques in R

I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers. I'd like to use a ...
0
votes
1answer
965 views

Implementation of Association Rules in Javascript

I have implemented an interactive visualization with d3.js, javascript to explore the frequency and various combinations of co-occurring item sets. I want to complement the interactive exploration ...
0
votes
1answer
225 views

When it is time to use Hadoop? [closed]

Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
3
votes
0answers
33 views

Timing sequence in MapReduce

I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution ...
4
votes
1answer
1k views

Scikit Learn Logistic Regression Memory Leak

I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a ...
10
votes
3answers
261 views

How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
4
votes
2answers
2k views

How to implement Brown Clustering Algorithm in O(|V|k^2)

I am trying to implement the Brown Clustering Algorithm. Paper details: "Class-Based n-gram Models of Natural Language" by Brown et al The algorithm is supposed to in ...
10
votes
2answers
1k views

Is FPGrowth still considered “state of the art” in frequent pattern mining?

As far as I know the development of algorithms to solve the Frequent Pattern Mining (FPM) problem, the road of improvements have some main checkpoints. Firstly, the Apriori algorithm was proposed in ...
5
votes
2answers
738 views

How to speedup message passing between computing nodes

I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire ...
15
votes
4answers
2k views

Looking for example infrastructure stacks/workflows/pipelines

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
11
votes
3answers
3k views

Best languages for scientific computing [closed]

It seems as though most languages have some number of scientific computing libraries available. Python has Scipy Rust has <...
7
votes
2answers
347 views

Filtering spam from retrieved data

I once heard that filtering spam by using blacklists is not a good approach, since some user searching for entries in your dataset may be looking for particular information from the sources blocked. ...
7
votes
3answers
118 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
55
votes
5answers
29k views

When is a Model Underfitted?

Logic often states that by underfitting a model, it's capacity to generalize is increased. That said, clearly at some point underfitting a model cause models to become worse regardless of the ...
5
votes
2answers
132 views

How to best accomplish high speed comparison of like data?

I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
12
votes
2answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
10
votes
1answer
748 views

What is the most efficient data indexing technique

As we all know, there are some data indexing techniques, using by well-known indexing apps, like Lucene (for java) or Lucene.NET (for .NET), MurMurHash, B+Tree etc. For a No-Sql / Object Oriented ...
10
votes
4answers
187 views

Why is it hard to grant efficiency while using libraries?

Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend ...
6
votes
2answers
565 views

Running an R script programmatically [closed]

I have an R script that generates a report based on the current contents of a database. This database is constantly in flux with records being added/deleted many times each day. How can I ask my ...
87
votes
12answers
12k views

How big is big data?

Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...