So you can integrate with the rest of the code base. It seems your company uses a mix of Java and python. What are you going to do if a little corner of the site needs machine learning; pass the data around with a database, or a cache, drop to R, and so on? Why not just do it all in the same language? It's faster, cleaner, and easier to maintain.
Know any ...
The topic of reproducible research (RR) is very popular today and, consequently, is huge, but I hope that my answer will be comprehensive enough as an answer and will provide enough information for further research, should you decide to do so.
While Python-specific tools for RR certainly exist out there, I think it makes more sense to focus on more ...
Most non-technical people often use Excel as a database replacement. I think that's wrong but tolerable. However, someone who is supposedly experienced in data analysis simply can not use Excel as his main tool (excluding the obvious task of looking at the data for the first time). That's because Excel was never intended for that kind of analysis and as a ...
A column-oriented database (=columnar data-store) stores the data of a table column by column on the disk, while a row-oriented database stores the data of a table row by row.
There are two main advantages of using a column-oriented database in comparison
with a row-oriented database. The first advantage relates to the amount of data one’s
need to read in ...
There may be a lot of reasons like:
Workforce flexibility: One Java / Python programmers can be moved to other tasks or projects easily.
Candidates availability: there are plenty of Java / Python programmers. You do not want to introduce a new programming language to later find out that there are no qualified workers or they are just too expensive.
The best reproducibility tool is to make a log of your actions, something like this:
experiment/input ; expected ; observation/output ; current hypothesis and if supported or rejected
exp1 ; expected1 ; obs1 ; some fancy hypothesis, supported
This can be written down on a paper, but, if your experiments fit in a computational framework, you can use ...
Hadoop is not a database, hadoop is an entire ecosystem.
Most people will refer to mapreduce jobs while talking about hadoop. A mapreduce job splits big datasets in some little chunks of data and spread them over a cluster of nodes to get proceed. In the end the result from each node will be put together again as one dataset.
Let's assume you load into ...
IntelliJ supports R via this plugin:
It's a recent project, so RStudio is still more powerful, including its focus on data-friendly environment (plots and data are always in sight).
Do experienced data scientists use Excel?
I've seen some experienced data scientists, who use Excel - either due to their preference, or due to their workplace's business and IT environment specifics (for example, many financial institutions use Excel as their major tool, at least, for modeling). However, I think that most experienced data scientists ...
Be sure to check out docker! And in general, all the other good things that software engineering has created along decades for ensuring isolation and reproductibility.
I would like to stress that it is not enough to have just reproducible workflows, but also easy to reproduce workflows. Let me show what I mean. Suppose that your project uses Python, a ...
RIDE - R-Brain IDE (RIDE) for R & Python, Other Data Science R IDEs, Other Data Science Python IDEs. Flexible layout. Multiple language support.
Jupyter notebook - The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser. The Jupyter Notebook App can be executed ...
It is in general true that for purely data science and statistics exercises R offers the best and fastest (especially if using the data.table package) tools and methods, that otherwise would be heavier to implement in Python (I assume by Python we all mean Pandas, though). Most data scientists do in fact use R to perform their models and calculations, or ...
I personally have used Julia for a good number of professional projects, and while, as Dirk mentioned, this is purely conjecture, I can give some insights on where Julia really stands out. The question of whether or not these reasons will prove enough to have Julia succeed as a language is anyone's guess.
Distributed Systems: Julia is the easiest language I'...
There is another choice which popular recently: docker(https://www.docker.com). Docker is a container and let you create/maintain a working environment very easily and fast.
install essential tools for data science in python
use r language to do data science
I think most people are answering without having a good knowledge of excel. Excel (since 2010) has an in memory columnar [multi table] database , called power pivot (which allows input from csv/databases etc), allowing it to store millions of rows (it doesn't have to be loaded on a spreadsheet).
It also has an ETL tool called power query allowing you to read ...
This is a pretty massive question, so this is not intended to be a full answer, but hopefully this can help to inform general practice around determining the best tool for the job when it comes to data science. Generally, I have a relatively short list of qualifications I look for when it comes to any tool in this space. In no particular order they are:
You may try using R with Jupyter notebook. It requires installation of jupyter R kernel, IRkernel which will allow you to open a new jupyter notebook with option to choose R instead of default python kernel.
See https://www.continuum.io/blog/developer/jupyter-and-conda-r and https://irkernel.github.io/installation/ for installation steps.
I will try to answer your questions, but before I'd like to note that using term "large dataset" is misleading, as "large" is a relative concept. You have to provide more details. If you're dealing with bid data, then this fact will most likely affect selection of preferred tools, approaches and algorithms for your data analysis. I hope that the following ...
It depends on what you do.
Column stores have two key benefits:
whole columns can be skipped
run-length compression works better on columns (for certain data types; in particular with few distinct values)
However they also have drawbacks:
many algorithms will need all columns, and only record at a time (e.g. k-means) or may even need to compute a ...
In general any tool that many researchers use is by definition useful/helpful.
For the particular case of Keras and other neural network frameworks (like PyTorch, TensorFlow, etc), a lot of people use them. You can see this by reading papers in the topic, as it is usually mentioned which framework the implementation is made in. You can also check "...
How can I ask my computer to run this every night at 4 am so that I have an up to date report waiting for me in the morning?
You can set up a cronjob on a Linux system. These are run at the set time, if the computer is on. To do so, open a terminal and type:
00 4 * * * r source(/home/FilePath/.../myRscript.R)
Source: Stack Overflow
VisualStudio added syntax highlighting for R a few days ago: https://www.visualstudio.com/news/2015-mar-10-vso
The current RStudio preview is pretty cool as well - you can switch to a dark theme, code completion is working well, you can filter in the viewer, etc.
This isn't a full solution, but you may want to look into OrientDB as part of your stack. Orient is a Graph-Document database server written entirely in Java.
In graph databases, relationships are considered first class citizens and therefore traversing those relationships can be done pretty quickly. Orient is also a document database which would allow you ...
Unfortunately, I do not have enough reputation points to answer to the post by Plank, so have to answer to the whole thread - sorry about that.
I am actually the developer of the open-source Collective Knowledge Framework mentioned above. It attempts to simplify sharing of artifacts and experimental workflows as reusable and reproducible Python components ...
While Docker images are now more trendy, I personally find Docker technology not user-friendly, even for advanced users. If you are OK with using non-local VM images and can use Amazon Web Services (AWS) EC2, consider R-focused images for data science projects, pre-built by Louis Aslett. The images contain very recent, if not the latest, versions of Ubuntu ...
There is an entire course that is devoted to reproducible research.
This course is based on R, but the underlying idea can be learnt.
One simple way is to have an Ipython notebook and keep saving every dirty work you do, be it cleaning the data, exploratory analysis or building the model.