92

To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far). "Big Data" and "Bad Data" are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require "...


34

As you rightly note, these days "big data" is something everyone wants to say they've got, which entails a certain looseness in how people define the term. Generally, though, I'd say you're certainly dealing with big data if the scale is such that it's no longer feasible to manage with more traditional technologies such as RDBMS, at least without ...


22

Total amount of data in the world: 2.8 zetabytes in 2012, estimated to reach 8 zetabytes by 2015 (source) and with a doubling time of 40 months. Can't get bigger than that :) As an example of a single large organization, Facebook pulls in 500 terabytes per day, into a 100 petabyte warehouse, and runs 70k queries per day on it as of 2012 (source) Their ...


17

The concept to understand is that the conditional is actually a vector. So, you can simply define the conditions, and then combine them logically, like: condition1 = (df.col1 == 10) & (df.col2 <= 15) condition2 = (df.col3 == 7) & (df.col4 >= 4) # at this point, condition1 and condition2 are vectors of bools df1 = df[condition1] df2 = df[...


13

To me Big Data is primarily about the tools (after all, that's where it started); a "big" dataset is one that's too big to be handled with conventional tools - in particular, big enough to demand storage and processing on a cluster rather than a single machine. This rules out a conventional RDBMS, and demands new techniques for processing; in particular, ...


13

I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS. H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several APIs such as their R API. The benefit of combining R and H2O is that H2O is very good at exploiting multi-cores or clusters with minimal effort of the user. It ...


13

Not every seed is the same. Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility: def seed_everything(seed=42): """" Seed everything. """ random.seed(seed) os.environ['PYTHONHASHSEED'] = str(seed) np.random.seed(seed) torch.manual_seed(seed) ...


12

Rational business people don't pay for accuracy, they pay to either save money on a profitable process (thereby making it more profitable), or by creating new money (creating new profitable processes). So any project that is undertaken has to be couched in terms that reflect this. The first step is always understanding which of the two processes ...


11

Data becomes "big" when a single commodity computer can no longer handle the amount of data you have. It denotes the point at which you need to start thinking about building supercomputers or using clusters to process your data.


10

It depends on your data and what you're doing with it. For example, if the processing you have to do requires transactions to synchronize across nodes, it will likely be faster to use transactions implemented in an RDBMS rather than implementing it yourself on top of NoSQL databases which don't support it natively.


10

Gather competitive counterparts. Try and determine a state-of-the-art and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven models are not created statically, they develop dynamically because a good scientist will always try to find ways to improve it. Upper management personnel ...


10

The two slightly-smaller models will perform exactly the same, on average. There is no difference baked in to the different trees: "the last tree will be the best trained" is not true. The only difference among the trees is the random subsample they work with and random effects while building the tree (feature subsetting, e.g.). Gradient boosted ...


8

Unfortunately, parallelization is not yet implemented in pandas. You can join this github issue if you want to participate in the development of this feature. I don't know any "magic unicorn package" for this purposes, so the best thing will be write your own solution. But if you still don't want to spend time on that and want to learn something new – you ...


8

There are a lot of parameters which matter when using GPU's for machine learning, some of them are: CUDA core count Memory bandwidth (GB/s) Memory per core (MB) Raw Speed (MHz) Total Memory available (GB) Performance on 16-bit, 32-bit floating ops/sec Tim Dettmers has an excellent (frequently updated) blog where he's compared different cards, near the end ...


8

First of all, just to be clear, you shouldn't evaluate the performance of your models on the balanced data set. What you should do is to split your dataset into a train and a test set with ideally the same degree of imbalance. The evaluation should be performed exclusively on the test set, while the balancing on the training set. As for your question, any ...


7

Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be ...


7

I don't think that everyone reaches for C/C++ when performance is an issue. The advantage to writing low-level code is using fewer CPU cycles, or sometimes, less memory. But I'd note that higher-level languages can call down to lower-level languages, and do, to get some of this value. Python and JVM languages can do this. The data scientist using, for ...


7

Some factors you might consider: Developer familiarity: go with whatever you or your developers are familiar with. Mongo, Couch, Riak, DynamoDB etc all have their strengths but all should do ok here, so rather than going for an unfamiliar solution that might be slightly better go for familiar and save a bunch of development time. Ease of cloud deployment: ...


7

Big Data is defined by the volume of data, that's right, but not only. The particularity of big data is that you need to store a lots of various and sometimes unstructured stuffs all the times and from a tons of sensors, usually for years or decade. Furthermore you need something scalable, so that it doesn't take you half a year to find a data back. So ...


7

This is a hard problem and researchers are making a lot of progress. If you're looking for supervised feature selection, I'd recommend LASSO and its variants. Evaluation of the algorithm is very straightforward with supervised learning: the performance of whichever metric you choose on test data. Two major caveats of LASSO are that (1) the selected ...


7

F1 will never be zero, but very near to zero for a bad classifier. If TP or TN is zero then there isn't any need to check F1.


7

FP32 and FP16 mean 32-bit floating point and 16-bit floating point. GPUs originally focused on FP32 because these are the calculations needed for 3D games. Nowadays a lot of GPUs have native support of FP16 to speed up the calculation of neural networks. If you look at some benchmarks (https://blog.slavv.com/titan-rtx-quality-time-with-the-top-turing-gpu-...


6

I'll share what Big Data is like in genomics, in particular de-novo assembly. When we sequence your genome (eg: detect novel genes), we take billions of next-generation short reads. Look at the image below, where we try to assemble some reads. This looks simple? But what if you have billion of those reads? What if those reads contain sequence errors? What ...


6

I think your best bet would be rosetta. I'm finding it extremely useful and easy. Check its pandas methods. You can get it by pip.


6

The differences in speed between Naive Bayes and SVM simply boils down to the formulation and the assumptions of each model, and has little to do with the particular library or implementation. Not only is naive bayes a simple probabilistic classifier, it also makes an additional assumption of independence between its features, so that parameter estimates ...


6

Have you timed which line of your code is most time consuming? I suspect that the line df = df[~df.isin(df1)].dropna() would take a long time. Would it be faster if you simply use the negation of the condition you applied to obtain df1, when you want to filter away rows in df1 from df? That is, use df = df[(df.col1 != 10) | (df.col2 > 15)].


6

Label flipping is a training technique where one selectively manipulates the labels in order to make the model more robust against label noise and associated attacks - the specifics depend a lot on the nature of the noise. Label flipping bears no benefit only under the assumption that all labels are (and will always be) correct and that no adversaries exist. ...


5

You need to look at the confidence interval of the statistic. This helps measure how much uncertainty in the statistic, which is largely a function of sample size.


5

There is the useful dask library for parallel numpy/pandas jobs


Only top voted, non community-wiki answers of a minimum length are eligible