29

A column-oriented database (=columnar data-store) stores the data of a table column by column on the disk, while a row-oriented database stores the data of a table row by row. There are two main advantages of using a column-oriented database in comparison with a row-oriented database. The first advantage relates to the amount of data one’s need to read in ...


12

What I am ending up using is a sort of hybrid solution: backup of the raw data git of the workflow manual snapshots of workflow + processed data, that are of relevance, e.g.: standard preprocessing really time-consuming for publication I believe it is seldom sensible to have a full revision history of large amount of binary data, because the time required ...


12

The one thing that you can say for sure is: Nobody can say this for sure. And it might indeed be opinion-based to some extent. The introduction of terms like "Big Data" that some people consider as "hypes" or "buzzwords" don't make it easier to flesh out an appropriate answer here. But I'll try. In general, interdisciplinary fields often seem to have the ...


11

It depends on what you do. Column stores have two key benefits: whole columns can be skipped run-length compression works better on columns (for certain data types; in particular with few distinct values) However they also have drawbacks: many algorithms will need all columns, and only record at a time (e.g. k-means) or may even need to compute a ...


10

It depends on your data and what you're doing with it. For example, if the processing you have to do requires transactions to synchronize across nodes, it will likely be faster to use transactions implemented in an RDBMS rather than implementing it yourself on top of NoSQL databases which don't support it natively.


10

For windows, use the task scheduler to set the task to run for example daily at 4:00 AM It gives you many other options regarding frequency etc. http://en.wikipedia.org/wiki/Windows_Task_Scheduler


10

Try looking at Git Large File Storage (LFS). It is new, but might be the thing worth looking at. As I see, a discussion on Hacker News mentions a few other ways to deal with large files: git-annex (and e.g. using it with Amazon S3) Mercurual Largefiles extension


10

For data load, Postgre outperforms MongoDB. MongoDB is almost always faster when returning query counts. PostgreSQL is almost always faster for queries using indexes. Check out this website and this one too for more info. They have very detailed explanations.


10

You may have a look at nexus or dvc or datmo. There was recently a presentation at at meetup in berlin, zalandos AI data engineering meetup.


9

This is a pretty common problem. I had this pain when I did research projects for a university and now - in industrial data science projects. I've created and recently released an open source tool to solve this problem - DVC. It basically combines your code in Git and data in your local disk or clouds (S3 and GCP storage). DVC tracks dependency between ...


9

I have used Versioning on Amazon S3 buckets to manage 10-100GB in 10-100 files. Transfer can be slow, so it has helped to compress and transfer in parallel, or just run computations on EC2. The boto library provides a nice python interface.


9

How can I ask my computer to run this every night at 4 am so that I have an up to date report waiting for me in the morning? You can set up a cronjob on a Linux system. These are run at the set time, if the computer is on. To do so, open a terminal and type: crontab -e And add: 00 4 * * * r source(/home/FilePath/.../myRscript.R) Source: Stack Overflow


9

See Lucene NGramTokenizer Are you sure you can't just use lucene or similar indexing techniques? Inverted indexes will store the n-gram only once, then just the document ids that contain the ngram; they don't store this as highly redundant raw text. As for finding ngrams that contain your query sub-n-gram, I would build an index on the observed ngrams, e....


8

The data set definitions are on the page here: Attribute Information at the bottom or you can see inside the ZIP folder the file named activity_labels, that has your column headings inside of it, make sure you read the README carefully, it has some good info in it. You can easily bring in a .csv file in R using the read.csv command. For example if you ...


8

Looking at this document called Anatomy of Facebook I note that the median is 100. Looking at the cumulative function plot I can bet that the average is higher, near 200. So 50 seems to not be the best number here. However I think that this is not the main issue here. The main issue is the lack of information on how the database was used. It seems ...


8

I think the premise of your question has a problem. Pandas is not a "datastore" in the way an RDBMS is. Pandas is a Python library for manipulating data that will fit in memory. Disadvantages: Pandas does not persist data. It even has a (slow) function called TO_SQL that will persist your pandas data frame to an RDBMS table. Pandas will only handle results ...


8

The ImageNet dataset consists of more than 14M images, divided into approximately 22k different labels/classes. However the ImageNet challenge is conducted on just 1k high-level categories (probably because 22k is just too much). ImageNet Stats When people mention results on the ImageNet, they almost always mean the 1k labels (if some paper uses the ...


7

Three tables: animal, observation, and sibling. The observation has an animal_id column which links to the animal table, and the sibling table has animal_1_id and animal_2_id columns that indicates two animals are siblings for each row. Even with 5000 animals and 100000 observations I don't think query time will be a problem for something like PostgreSQL ...


7

I would like to suggest 2 more approaches. Store them in document storage (eg. mongoDB) - this method is recommended when your model files are less then 16Mb (or the joblib shards are), then you can store model as binary data. in addition, some ML libraries support model export and import in json (eg. LightGBM), which makes it a perfect candidate for ...


6

We don't version control the actual data files. We wouldn't want to even if we stored it as CSV instead of in a binary form. As Riccardo M. said, we're not going to spend our time reviewing row-by-row changes on a 10M row data set. Instead, along with the processing code, I version control the metadata: Modification date File size Row count Column names ...


6

I suppose you could do this, but if your goal is simply to store 15 boolean values in a single column you are complicating things unnecessarily. Instead of going to all the trouble to compute the prime factors of the stored value, why don't you just store the flags as a bit string? Your example of 15 different possible values could be stored in a single ...


6

I faced this problem (and still face it today) for many years. I really thing that, if you don't provide detailed requirements, you can't expect a serious answer. I explain myself with examples of my work: I regularly try multiple variations of the same model to find what parameters work best. It takes several day to train one single model which produces ...


5

It looks like this (or very similar data set) is used for Coursera courses. Cleaning this dataset is task for Getting and Cleaning Data, but it is also used for case study for Exploratory Data analysis. Video from this case study is available in videos for week 4 of EDA course-ware. It might help you with starting with this data.


5

If you need to scale beyond 1000 entries in the future, a brute-force approach to find the exact neighbors will become increasingly prohibitive from a computational standpoint. To future-proof your solution, I would recommend looking into the well-researched field of approximate nearest neighbors (ANN) techniques. Obviously there is a speed/accuracy tradeoff,...


4

Especially when your main goal is learning, I would break it into several phases: Get familiar with pandas dataframes and visualization using matplotlib. Try loading subsets of your datasets using pandas and visualize them using matplotlib, for example plot the time-series, or word count histograms. This will come in handy lateron to be able to understand ...


4

There are good/fast ways to model graphs in RDBMS, and dumb/slow ways. Some use clever indexing and Stored Procs, trading CPU load and tuned temp tables on RAM disks for faster graph retrieval speed. Some use precomputed graph paths (this may be less feasible in social network scenario, but in a tree with majority of nodes being leaf nodes, it's a pretty ...


4

'SQL' on Hadoop is very much a thing, though I use quotes since it's probably more accurate to say it's SQL-like. Some options for bringing SQL-like capabilities to Hadoop include Hue, Hive/bee (Heading towards Stinger? So punny Apache), Impala, SparkSQL (probably not a great solution for a bank given the possibility of concurrency issues), among others (...


4

Well, they are absolutely different things but that are somehow linked. I gonna go through each of them. Data Base Think of a data base (DB from now) like a computer which only purpose is to store data accesable to be read. By data, and focusing only in SQL-like DB, I mean basically tables of information like and excel file with columns and rows. You can ...


4

"Premature optimization is the root of all evil". --Knuth You could do this, but why? You definitely don't want to do this, if you're going to feed the result into a classifier: most classifiers will perform worse after this transformation. There's no point to do this, if you're trying to save space in a database: hard disks can store hundreds of ...


Only top voted, non community-wiki answers of a minimum length are eligible