Questions tagged [data-wrangling]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
3answers
397 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
-1
votes
1answer
13 views

Data wrangling dates

I have a feature with data creation dates. I have normalized them all to the same format and split them to 'day', 'month' and 'year' columns. But now I have a question. Should I apply normalization or ...
4
votes
2answers
166 views

Data wrangling for a big set of docx files advice!

I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (...
11
votes
4answers
43k views

Export pandas to dictionary by combining multiple row values

I have a pandas dataframe df that looks like this ...
1
vote
1answer
65 views

Group_by 2 variables and pivot_wider distribution based on 2 others

Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all ...
2
votes
1answer
43 views

What is the difference between 'if the data is of good quality' and 'if the data is tidy'?

I'm doing Data Analyst nanodegree from Udacity. I'm confused between the difference even after going through the lecture a few times.
2
votes
1answer
91 views

What should I do with the NaN values on this stock quote data?

I concatenated 3 stock quote data-frames all with date-time indexes. However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
2
votes
1answer
56 views

How to preprocess an ordered categorical variable to feed a machine learning algorithm?

I have a categorical variable that measures the income of a family: A: no income B: Up to $500 C: $500-$700 … P: $5000-$6000 Q: More than \\\$6000 It seems odd to ...
3
votes
2answers
3k views

Mean across every several rows in pandas

I have a table of features and labels where each row has a time stamp. Labels are categorical. They go in a batch where one label repeats several times. Batches with the same label do not have a ...
1
vote
2answers
6k views

When to choose character instead of factor in R?

I am currently working on a dataset which contains a name attribute, which stands for a person's first name. After reading the csv file with ...
-1
votes
1answer
24 views

Similar values cleaning [closed]

can someone know algorithm how to identify account names that are similar enough to be potentially merged and imported as one Duplicates with different values: Geico val1 NaN =====>>...
1
vote
0answers
23 views

Issue with miscount on test train split in Python

Problem Hey everyone. I think I am missing something incredibly easy, but because I am wokring in resampling for the first time it is giving me all sorts of inadequacy. I performed an ADASYN up sample ...
0
votes
1answer
29 views

Encoding of high cardinality multi-label categorical feature?

This is the problem of binary classification: "1" - the subscriber is a driver (belongs to the segment of drivers), "0" - the subscriber is not a driver (does not belong to the ...
1
vote
2answers
3k views

how to calculate number of datapoints within a given time interval?

I have a dataframe with one column which is a timestamp. I have converted that column to a datetime object using pandas to_datetime() method. However what I want is to ...
0
votes
0answers
13 views

R - using heatmaply for a 2d histogram / density

I have a rather long series of x, y coordinates representing a character in 2d space. Let's say that space is 200 x 400. I want to represent the amount of time the character is in each x, y coordinate ...
3
votes
2answers
710 views

Inputting (a lot of )data into a dataframe one row at a time

I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this: ...
0
votes
0answers
83 views

How do I extract album and song titles from this plain text file?

Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the ...
0
votes
1answer
91 views

How to obtain unique count of categorical variable based on another categorical variable?

I have a dataframe with date and cat_1 variables among other variables. I want to get number of unique values of ...
1
vote
0answers
141 views

Grouping by 2 Columns in R to Sum, Count, Percentage, Weighted mean, mode [closed]

I am trying to use dplyr or data.table to calculate sums, percentages, counts, and weighted means for various sub groups. Each Name can have subgroups High Medium and low, and potentially others, and ...
45
votes
9answers
8k views

How much of data wrangling is a data scientist's job?

I'm currently working as a data scientist at a large company (my first job as a DS, so this question may be a result of my lack of experience). They have a huge backlog of really important data ...
1
vote
2answers
83 views

How to make tool that's robust to user-generated typos?

Background: A client is generating datasets (excel files). They asked I make an app to analyze the datasets, e.g. summary tables and figures. I'm doing this in R shiny. Problem: There are many typos ...
4
votes
3answers
282 views

How to deal with count data in random forest

I am working on a classification model where my target class is a biased class with the class shape as 0 1 20694 101 Most of my features are the ...
2
votes
0answers
146 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
0
votes
3answers
361 views

Can we use median to replace all the missing values from a column

If we can use the median to replace all of the missing values from a column, then what is the advantage of using median over mean for replacing the missing values in a column with numeric values?
2
votes
0answers
111 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
1
vote
1answer
72 views

How to run KNN (or other) on nested features? Image metadata

I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I ...
1
vote
0answers
127 views

Cleaning Excel Data in Merged Cells [closed]

I need to use an Excel file that have many merged cells with no specific order. I guess that the file creator thought that it would be more readable to merge the cells that have the same value. ...
1
vote
3answers
69 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
3
votes
1answer
17k views

Populate column based on previous row with a twist

I'm struggling with a Pandas problem. I have the following data. ...
2
votes
2answers
209 views

Technical name for this data wrangling process? Multiple columns into multi-factor single column

What is the technical name for the following data wrangling process? I want to collapse Table A into Table B. (To make the data suitable for ANOVA.) Table A: ...
1
vote
0answers
22 views

How to work with string data with a lot of NAs in an aggregation task with R

I have recently taken up a data wrangling task and inherited a data from my predecessor which contains keyphrases extracted by hand like the following : ...
1
vote
1answer
9k views

removing special character from CSV file

I read my csv file as pandas dataframe. Originally it's a dict with multiple entries per keys. Its looks like this after reading as pandas dataframe: ...
1
vote
2answers
450 views

How do I split number string with digit pattern?

I am trying to split number string to two to digit numbers How do I get two different Numbers out that string Example: I want separate two numbers ...
0
votes
1answer
395 views

Unable to print in Jupyter Notebook using Pandas [closed]

I am doing basic data analysis on an csv file in jupyter notebook def answer_two(): return (df['Gold']-df['Gold.1']).argmax() answer_two The above code ...
0
votes
2answers
92 views

R Programming rearranging rows and colums from timeline data [closed]

I am new to R programming and data analytics. This is probably a trivial question but I could not find an answer. I'm also limited with my ontology of data science and data manipulation. So, I'm not ...
0
votes
2answers
80 views

Adoptive(Machine wise) ML packages

What kind of Machine Learning/Data Mining packages are available that can easily be scaled on a cluster. For instance H2O is one ...
1
vote
4answers
2k views

Which one is better performer on wrangling big data, R or Python?

I want to know which language/packages performs better & faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and ...
2
votes
2answers
131 views

How do you define the steps to explore the data? [closed]

I'm falling in love with data science and I'm spending a lot of time studying it. It seems that a common data science workflow is: Frame the problem Collect the data Clean the data Work on the data ...
0
votes
2answers
40 views

Analytics term for turning row values into column names and count its assigned values

Do we have a data mining/analysis term for turning row values into column names and count its assigned values?
1
vote
2answers
34k views

Sum up counts in a data.frame grouped by multiple variables

This is a snippet of the dataset I am currently working on: ...
2
votes
2answers
246 views

Sort by average votes/ratings

I have a data set that's a dictionary of tuples. Each key represents an ID number and each tuple is (yesvotes, totalvotes). Example: ...
4
votes
2answers
434 views

Tools to perform SQL analytics on 350TB of csv data

In short, what would be the best method/tricks/techniques/tools for performing ad hoc sql (style) queries on 350TB of csv data? Would there be other options, tool wise that would do it faster if we ...
1
vote
1answer
284 views

Detecting boilerplate in text samples

I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example: DESCRIPTION PROVIDED BY AUTHOR: The goal ...