16

Hadoop is not a database, hadoop is an entire ecosystem. Most people will refer to mapreduce jobs while talking about hadoop. A mapreduce job splits big datasets in some little chunks of data and spread them over a cluster of nodes to get proceed. In the end the result from each node will be put together again as one dataset. Let's assume you load into ...


6

Although your question is not very specific so I'll try to give you some generic solutions. There are couple of things you can do here: Check sparseMatrix from Matrix package as mentioned by @Sidhha Try running your model in parallel using packages like snowfall, Parallel. Check this list of packages on Cran which can help you runnning your model in ...


5

NoSQL is a way to store data that does not require there to be some sort of relation. The simplicity of its design and horizontal scale-ability, one way they store data is the key : value pair design. This lends itself to processing that is similar to Hadoop. The use of a NoSQL db really depends on the type of problem that one is after. Here is a good ...


4

I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your broader categories; description, methodology, classifier, application, and example. That is, I would manually label some transcripts at the paragraph level. If you ...


3

This is a post that starts from data science and walks you through options for completing common tasks: http://www.drbunsen.org/explorations-in-unix/.


3

I would recommend this one - http://www.cheat-sheets.org/saved-copy/ubunturef.pdf Its comprehensive, but not data science specific. There are a number of great cheat sheets here that may be what you are looking for:https://dzone.com/refcardz If you are wanting to create your own, this is a very nice tool: http://www.cheatography.com/


3

The answer to both data sets is an OCR application with some post-processing, but a more specialized program than a generic low-quality or an open source OCR. Essentially the harder the problem, the more capable and advanced tools need to be used to solve it. There will be two major stages in this task: digitizing the data (image to text, i.e. OCR) and ...


2

Since you mention you are building a recommendation system, I believe you have a sparse matrix which you are working on. Check sparseMatrix from Matrix package. This should be able to help you with storing your large size matrix in memory and train your model.


2

To be reproducible without being just a replication, you would need to redo the experiment with new data, following the same technique as before. The work flow is not as important as the techniques used. Sample data in the same way, use the same type of models. It doesn't matter if you switch from one language to another, so long as the models and the ...


2

It looks like the Tech Web Sites that reported Meltdown, may have overblown the negative impacts. According to Google, in their cloud infrastructure, there seems to be little to worry about: There has been speculation that the deployment of KPTI causes significant performance slowdowns. Performance can vary, as the impact of the KPTI mitigations depends ...


1

This could actually be done quickly and intuitively using linear algebra. So consider your player as label binarized array (can be done with MultiLabelBinarizer) so you would expect an array of size (2060590, 39) containing 0 an 1, rearrange the columns similar as how you order the your player table (or the other-way around which ever is easier), basically ...


1

Using xgb.train function, you can set nthread in params. xgb.train({'nthread': 3}, dtrain) In the xgboost documentation, afaict, there is no clear way how to set the nthread parameter. Some global params are set by xgb.set_config and some are not, like nthread. But when I walk through their test script, I found nthread is set by params in xgb.train.


1

You can use the --timeout flag to increase the worker timeout. Run gunicorn --help for more information about the available flags


1

Fun with Group Theory! There are only 8 unique rotation-inversion operations for a square matrix. The four rotation operators are (0,90,180,270). Further rotation or rotation in the reverse direction is the same as these four. Two successive rotations just yields one of the rotation operators, so we will only consider these four rotations applied one ...


1

The reason your OCR program is not recognizing anything is that the font size is very small. If you resize your image to increase its size, you will obtain better results. For example, using ImageMagick to apply a fixed threshold to remove the background on your first image and increase its size: convert -density 500 -threshold 40% 29-1891a.gif -resize 250% ...


1

I thought about it this way: the training and test sets are both a sample of the unknown population. We assume that the training set is representative of the population we're studying. That is, whatever transformations we make to the training set are what we would make to the overall population. In addition, whatever subset of the training data we use, we ...


Only top voted, non community-wiki answers of a minimum length are eligible