90

To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far). "Big Data" and "Bad Data" are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require "...


34

As you rightly note, these days "big data" is something everyone wants to say they've got, which entails a certain looseness in how people define the term. Generally, though, I'd say you're certainly dealing with big data if the scale is such that it's no longer feasible to manage with more traditional technologies such as RDBMS, at least without ...


22

Total amount of data in the world: 2.8 zetabytes in 2012, estimated to reach 8 zetabytes by 2015 (source) and with a doubling time of 40 months. Can't get bigger than that :) As an example of a single large organization, Facebook pulls in 500 terabytes per day, into a 100 petabyte warehouse, and runs 70k queries per day on it as of 2012 (source) Their ...


15

The concept to understand is that the conditional is actually a vector. So, you can simply define the conditions, and then combine them logically, like: condition1 = (df.col1 == 10) & (df.col2 <= 15) condition2 = (df.col3 == 7) & (df.col4 >= 4) # at this point, condition1 and condition2 are vectors of bools df1 = df[condition1] df2 = df[...


13

To me Big Data is primarily about the tools (after all, that's where it started); a "big" dataset is one that's too big to be handled with conventional tools - in particular, big enough to demand storage and processing on a cluster rather than a single machine. This rules out a conventional RDBMS, and demands new techniques for processing; in particular, ...


13

I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS. H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several APIs such as their R API. The benefit of combining R and H2O is that H2O is very good at exploiting multi-cores or clusters with minimal effort of the user. It ...


12

Rational business people don't pay for accuracy, they pay to either save money on a profitable process (thereby making it more profitable), or by creating new money (creating new profitable processes). So any project that is undertaken has to be couched in terms that reflect this. The first step is always understanding which of the two processes ...


10

Data becomes "big" when a single commodity computer can no longer handle the amount of data you have. It denotes the point at which you need to start thinking about building supercomputers or using clusters to process your data.


10

It depends on your data and what you're doing with it. For example, if the processing you have to do requires transactions to synchronize across nodes, it will likely be faster to use transactions implemented in an RDBMS rather than implementing it yourself on top of NoSQL databases which don't support it natively.


10

Gather competitive counterparts. Try and determine a state-of-the-art and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven models are not created statically, they develop dynamically because a good scientist will always try to find ways to improve it. Upper management personnel ...


8

Unfortunately, parallelization is not yet implemented in pandas. You can join this github issue if you want to participate in the development of this feature. I don't know any "magic unicorn package" for this purposes, so the best thing will be write your own solution. But if you still don't want to spend time on that and want to learn something new – you ...


8

There are a lot of parameters which matter when using GPU's for machine learning, some of them are: CUDA core count Memory bandwidth (GB/s) Memory per core (MB) Raw Speed (MHz) Total Memory available (GB) Performance on 16-bit, 32-bit floating ops/sec Tim Dettmers has an excellent (frequently updated) blog where he's compared different cards, near the end ...


8

First of all, just to be clear, you shouldn't evaluate the performance of your models on the balanced data set. What you should do is to split your dataset into a train and a test set with ideally the same degree of imbalance. The evaluation should be performed exclusively on the test set, while the balancing on the training set. As for your question, any ...


7

Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be ...


7

I don't think that everyone reaches for C/C++ when performance is an issue. The advantage to writing low-level code is using fewer CPU cycles, or sometimes, less memory. But I'd note that higher-level languages can call down to lower-level languages, and do, to get some of this value. Python and JVM languages can do this. The data scientist using, for ...


7

Some factors you might consider: Developer familiarity: go with whatever you or your developers are familiar with. Mongo, Couch, Riak, DynamoDB etc all have their strengths but all should do ok here, so rather than going for an unfamiliar solution that might be slightly better go for familiar and save a bunch of development time. Ease of cloud deployment: ...


7

Big Data is defined by the volume of data, that's right, but not only. The particularity of big data is that you need to store a lots of various and sometimes unstructured stuffs all the times and from a tons of sensors, usually for years or decade. Furthermore you need something scalable, so that it doesn't take you half a year to find a data back. So ...


7

Not every seed is the same. Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility: def seed_everything(seed=42): """" Seed everything. """ random.seed(seed) os.environ['PYTHONHASHSEED'] = str(seed) np.random.seed(seed) torch.manual_seed(seed) torch.cuda.manual_seed(seed) ...


6

I'll share what Big Data is like in genomics, in particular de-novo assembly. When we sequence your genome (eg: detect novel genes), we take billions of next-generation short reads. Look at the image below, where we try to assemble some reads. This looks simple? But what if you have billion of those reads? What if those reads contain sequence errors? What ...


6

I think your best bet would be rosetta. I'm finding it extremely useful and easy. Check its pandas methods. You can get it by pip.


6

The differences in speed between Naive Bayes and SVM simply boils down to the formulation and the assumptions of each model, and has little to do with the particular library or implementation. Not only is naive bayes a simple probabilistic classifier, it also makes an additional assumption of independence between its features, so that parameter estimates ...


5

Ok, this ended up being an RTFM situation, although in this case it was RTF error message. While running this, I kept getting the following error: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel(). I assumed that, since this had to do with the target ...


5

You need to look at the confidence interval of the statistic. This helps measure how much uncertainty in the statistic, which is largely a function of sample size.


5

There is the useful dask library for parallel numpy/pandas jobs Link: https://github.com/blaze/dask


5

There is special thing to graph algorithms, you original questions which makes then special, which is about he ability to partition the data essentially. For some things, like sorting numbers on an array it is not too difficult to partition the problem on the data structure into smaller disjunctive pieces, e.g. Here: Parallel in place merge sort For ...


5

I like this question because it gets at the politics that exist in every organization. In my view and to a significant degree, expectations about model performance are a function of the org culture and degree to which an organization is "technically literate." One way to make clear what I mean is to consider the differences between the 4 big "data science" ...


5

I think the term "sensitivity" comes from the world of medical tests. A very sensitive test will test positive for most or all people who take the test and really have a disease, as well as for many people who don't. This corresponds to high recall, which means the query retrieves most or all of the relevant documents, as well as many that may not be ...


5

I get about this same utilization rate when I train models using Tensorflow. The reason is pretty clear in my case, I'm manually choosing a random batch of samples and calling the optimization for each batch separately. That means that each batch of data is in main memory, it's then copied into GPU memory where the rest of the model is, then forward/back ...


5

Have you timed which line of your code is most time consuming? I suspect that the line df = df[~df.isin(df1)].dropna() would take a long time. Would it be faster if you simply use the negation of the condition you applied to obtain df1, when you want to filter away rows in df1 from df? That is, use df = df[(df.col1 != 10) | (df.col2 > 15)].


5

I guess differences in accuracies between class 0 and class 1 come from the class_weight parameter you have used. Class 1 will benefit from this overweighting towards class 0. You could try to play on this parameter to re-balance your results in class 0 and class 1. An other idea could be to play on probabilities outputs and decision boundary threshold. ...


Only top voted, non community-wiki answers of a minimum length are eligible