40

This is a situation that many blogs, companies and papers acknowledge as something real in many cases. In this paper Data Wrangling for Big Data: Challenges and Opportunities, there is a quote about it data scientists spend from 50 percent to 80 percent of their time collecting and preparing unruly digital data. Also, you can read the source of that quote ...


27

Feels like most of the work is not related to data science at all. Is this accurate? Yes I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that data science requires minimum levels of data accessibility. Am I wrong? You're not wrong, but such are the realities of real life. Is this type of setup ...


26

Feels like most of the work is not related to data science at all. Is this accurate? This is the reality of any data science project. Google actually measured it and published a paper "Hidden Technical Debt in Machine Learning Systems" https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf Result of the paper reflects my ...


10

Does this do what you want it to? from pandas import DataFrame df = DataFrame([['A', 123, 1], ['B', 345, 5], ['C', 712, 4], ['B', 768, 2], ['A', 318, 9], ['C', 178, 6], ['A', 321, 3]], columns=['name', 'value1', 'value2']) d = {} for i in df['name'].unique(): d[i] = [{df['value1'][j]: df['value2'][j]} for j in df[df['name']==i].index] This returns ...


7

Feels like most of the work is not related to data science at all. Is this accurate? Wrangling data is most definitely in the Data Scientist job description. At some level you have to understand the data generating process in order to use it to drive solutions. Sure, someone specialized in ETL could do it faster/more efficient, but being given data dumps ...


6

As another recent starter in Data Science, I can only add that I don't think you're experience is unique, my team of about 10 apparently hasn't done any DS in over a year (one small project that occupied 2 of the team). This is due to the promise of an effective pipeline the team's been working on, but still just isn't quite delivering the data. Apparently ...


5

If you look at this from the perspective of "this isn't my job, so why should I do it" then that's a fairly common, general problem not specific to data science. Ultimately, your job is to do whatever the boss tells you to do, but in practice there is little reason for the boss to be dictatorial about this and usually they can be persuaded. Or at least they ...


4

The to_dict() method sets the column names as dictionary keys so you'll need to reshape your DataFrame slightly. Setting the 'ID' column as the index and then transposing the DataFrame is one way to achieve this. The same can be done with the following line: >>> df.set_index('ID').T.to_dict('list') {'p': [1, 3, 2], 'q': [4, 3, 2], 'r': [4, 0, 9]} ...


3

df.groupby('name')[['value1','value2']].apply(lambda g: g.values.tolist()).to_dict() if you need a list of tuples explicitly: df.groupby('name')[['value1','value2']].apply(lambda g: list(map(tuple, g.values.tolist()))).to_dict()


3

This is an interesting question. I also see people mentioning Spark as de-facto. Here are my two cents on the same Big Data: The size of the data and the goal is the key here. Goal could be defined as Decrease computation time for data wrangling Efficient storage Handling multiple file formats coming from different sources Ability to provide data ...


3

Wrong question. Big data is not a question of this or that language, but cluster computing. For me it's implicit in the definition; if you can find a way to process your data on your laptop it just isn't big data. Spark is the de facto standard today for cluster computing. It comes with many of its own munging primitives, borrowed from numerous languages (...


3

It really depends on what you mean by 'big data'. A truly big dataset cannot fit in memory, in which case local python and R really only work for smaller scale experimentation and prototyping. For the purpose of data wrangling, you'll want a picture of the whole dataset by either slicing based on cuts, sampling, or aggregation. In any case, you'll need to ...


3

I think this may help you: Is the R language suitable for Big Data I think it really depends on what you are comfortable with and what your objectives are. In Python, I can execute SQL queries directly into Pandas data frames. From there, data cleaning and visualizing (I like seaborn) is fairly straightforward. I can easily perform matrix manipulations ...


3

Yes, this is a well-studied problem: rank aggregation. Here is a solution with code. The problem is that the quantity you are trying to estimate, the "score" of the item, is subject to noise. The fewer votes you have the greater the noise. Therefore you want to consider the variance of your estimates when ranking them.


3

Perhaps to put it simply: When creating variables and binning numerics, would you be doing that blindly, or after analysing your data? When peers review your findings, if they had questions about particular bits of data, would it embarrass you to not know them? You need to work with and understand your data - which includes simple stuff from fixing ...


3

1) Feels like most of the work is not related to data science at all. Is this accurate? In my opinion, Data Science cannot pull out from Data wrangling. But, as you said, the question would come on how much percentage of Data Wrangling is required to do by a Data Scientist. It depends on Organizations bandwidth and the person interest in doing such work. In ...


3

In his talk "Big Data is four different problems", Turing award winner Michael Stonebraker mentions this particular issue as a big problem (video, slides) He says that there are a number of open problems in this area: Ingest, Transform(e.g. euro/dollar), Clean(e.g.-99/Null), Schema mapping (e.g. wages/salary), Entity consolidation (e.g. Mike ...


3

Try nunique(). That should do it. Here is a toy example: import pandas as pd df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'foo'], 'B' : ['one', 'one', 'two', 'three', 'two', 'two', 'one', 'two']}) print(df) gives A B 0 foo one 1 bar one ...


2

There are many tools that are able to support such queries (as you mentioned Hive or Spark), and it is really up to your requirements in terms of number of queries, number of people who are going to query the data, what kind of BI or reporting tools you might want to use with it, etc. More than that, your requirements are probably going to change in the ...


2

You can do this using the xtabs function! Here's how I did it using your example data: # Create example data... name <- c("Maria", "Thomas", "Maria", "Maria", "Thomas", "Maria") sex <- c("f", "m", "m", "f", "m", "m") count <- c(97, 12, 5, 97, 8, 4) data <- data.frame("name"=name, "sex"=sex, "count"=count) # Create table... xtabs(formula=count~...


2

Have a look at the example on "How to order Reddit comments" using their up- & down-votes in Cam Davidson Pilon's book. $$ \frac{a}{a+b} - 1.65\sqrt{\frac{a b}{(a+b)^2(a+b+1)}} $$ where $$ a = 1 + u $$ $$ b = 1 + d $$ $u$ is the number of yes votes and $d$ is the number of no votes. Sorting your data using the score obtained from that formula results ...


2

It is usually called reshaping! For a great description of the process, see this walkthrough, or read up on Hadley Wickham's documentation for the reshape package!


2

As far as working with data depends on one's education, expertise, goal and favorite tools, I would answer it within my narrow scope - and trying to keep your track. Framing the problem is an important starting point a lot of people neglect. Even-though it is only the beginning, this should result in first strategies to explore the data. Translate "What I ...


2

That is a very good framework of solving a question you have. According to me, it has multiple answers. I'll give you the one which I relate to. After cleaning the data or rather while cleaning it, we have to be clear about the task ahead and about our results. The work on the data follows the below steps mostly: Feature Detection Training using the above ...


2

A simple approach could be the following: import numpy as np import pandas as pd counter = 0 def conditional_cumulative_sum(x): global counter if x['status'] == 'EXPIRED': return np.nan elif x['side'] == 'BUY': temporal = counter counter += 1 return temporal elif x['side'] == 'SELL': counter = 0 ...


2

Let me try to explain by intuitively. First let me take the easy one. Data being tidy As per definition Tidy means Arranged in Order, Neat, Uncluttered. All of these explain the physical aspects of the data representation. For example, data arranged in proper columns, with good headings, with relevance etc. You can think of this being syntactic in nature ...


2

from pandas import read_csv, concat from ast import literal_eval df = read_csv('file.csv',header=None,names=['name','value']) split = df.value.apply(literal_eval).apply(Series).set_index(df.name) part1 = split.ix[:,:2] part2 = split.ix[:,3:5] part3 = split.ix[:,6:] part2.columns=part3.columns=range(3) stacked = concat([part1,part2,part3]) Note that this ...


2

The answer given by @Aditya is very good and educational despite the fact that the how is deprecated. So the right answer would be as follows: df.resample("15T").agg({'count':'sum'})


2

Some thoughts: Your data is highly imbalanced. This is a critical issue which should be dealt with. Possible solutions include simple under-/over-sampling to more complicated synthetic approaches like SMOTE. Decision trees and random forests do not require feature scaling - this means that normalisation is not needed (unless perhaps you plan on using some ...


2

I'm getting timeit results of about 1/4 of the time using: flatX = X.apply(lambda x: x.flatten()) pd.DataFrame(item for item in flatX) See also https://stackoverflow.com/questions/45901018/convert-pandas-series-of-lists-to-dataframe for some possibly better options for the second line. (Regarding my earlier comment, I don't get any real savings by just ...


Only top voted, non community-wiki answers of a minimum length are eligible