Questions tagged [data-wrangling]
The data-wrangling tag has no usage guidance.
54
questions
0
votes
0
answers
12
views
How to use text-to-columns or an equivalent operation on records in 1 field with varying lengths & no clear delimiters built into them
If this question in inappropriate for Data Science Stack Exchange, but there is another one within this stack it would be appropriate for, please let me know! It looks borderline to me.
A member of my ...
- 157
0
votes
0
answers
30
views
How to multiply the boolean/binary values in 1 row in a dataframe by the chr values in another row with the product landing in the original row
Suppose that I already know which subset out of a set of 30 candidate regressor columns are the true regressors included in the structural equation describing that dataset (because I do by ...
- 157
0
votes
1
answer
25
views
Tool/package for merging tables with inconsistent column names and categorical variable encoding?
I have 10s of spreadsheets with facility-level rows. Each spreadsheet corresponds to a month. They each contain approximately the same variables (10s of them), but often with different column naming ...
- 111
0
votes
1
answer
1k
views
Export pandas dataframe to dictionary as tuple keys and value [closed]
I have a pandas dataframe df that looks like this:
col1 col2 col3
A X 1
B Y 2
C Z 3
...
- 3
2
votes
0
answers
68
views
Dealing with class imbalance in test set
I am building a machine that tries to predict which ISP customers will complain due to issues with the network. I am having some difficulties.
The idea is to use network metrics of ~300K customers as ...
- 133
3
votes
3
answers
1k
views
Giving each person in order their top choice which is still available in Google Sheets
The problem I want to solve is my residential building's garage choices.
There will be a random distribution of parking spaces.
I thought that it would be better if each person writes down which ...
- 131
1
vote
1
answer
348
views
Data Wrangling and data cleaning
I found some information about Data Wrangling and they say different things.
In this one, they say data cleaning is a subcategory of data wrangling link
In this PDF, data wrangling is EDA and model ...
- 481
0
votes
1
answer
38
views
Advantages to combining similarly-named columns for supervised ML?
Is there any benefit to combining similarly named columns either for an improvement in accuracy or for speeding up training/prediction in case of logistic regression, random forest or neural network ...
- 3
0
votes
1
answer
2k
views
Filter for top 10 highest values of group in dataset (in R)
Context: I am trying to find the top 10 highest values of count in my data frame conditional on them falling within the years 1970-1979. My data frame looks as below:
...
- 39
1
vote
1
answer
291
views
Data cleaning in Pandas, where the csv file has all data of each row in 1 field [closed]
I have really messy data that looks like this:
As you can see all the data in each row is contained in 1 column separated by a semi colon.
How do I arrange this data so that they are spread out over ...
0
votes
1
answer
4k
views
Compare multiple values from a DataFrame against a single row from another
I'm trying to compare address values for inaccuracies, for example, given multiple records like:
Reference
Apartment
Address
PostCode
AS097
NaN
00 Name Road
BH1 4HB
AS097
Flat 1 Building Name
00 ...
- 113
1
vote
0
answers
21
views
What is a good way to handle nominal spatial data with a changing number of categories to use in prediction model?
For a project I'm going to be working with spatial data with a nominal attribute (land use). Every year the number of categories for this attribute changes because categories split or merge. I do have ...
-1
votes
1
answer
38
views
Data wrangling dates
I have a feature with data creation dates. I have normalized them all to the same format and split them to 'day', 'month' and 'year' columns. But now I have a question. Should I apply normalization or ...
- 101
-1
votes
1
answer
30
views
Similar values cleaning [closed]
can someone know algorithm how to identify account names that are similar enough to be potentially merged and imported as one
Duplicates with different values:
Geico val1 NaN =====>>...
1
vote
0
answers
90
views
Issue with miscount on test train split in Python
Problem
Hey everyone. I think I am missing something incredibly easy, but because I am wokring in resampling for the first time it is giving me all sorts of inadequacy. I performed an ADASYN up sample ...
- 11
0
votes
1
answer
163
views
Encoding of high cardinality multi-label categorical feature?
This is the problem of binary classification:
"1" - the subscriber is a driver (belongs to the segment of drivers),
"0" - the subscriber is not a driver (does not belong to the ...
2
votes
1
answer
154
views
How to preprocess an ordered categorical variable to feed a machine learning algorithm?
I have a categorical variable that measures the income of a family:
A: no income
B: Up to $500
C: $500-$700
…
P: $5000-$6000
Q: More than \\\$6000
It seems odd to ...
- 21
0
votes
0
answers
206
views
How do I extract album and song titles from this plain text file?
Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the ...
- 101
0
votes
1
answer
1k
views
How to obtain unique count of categorical variable based on another categorical variable?
I have a dataframe with date and cat_1 variables among other variables. I want to get number of unique values of ...
- 1,091
1
vote
1
answer
488
views
Group_by 2 variables and pivot_wider distribution based on 2 others
Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all ...
- 31
1
vote
0
answers
244
views
Grouping by 2 Columns in R to Sum, Count, Percentage, Weighted mean, mode [closed]
I am trying to use dplyr or data.table to calculate sums, percentages, counts, and weighted means for various sub groups.
Each Name can have subgroups High Medium and low, and potentially others, and ...
- 31
1
vote
2
answers
228
views
How to make tool that's robust to user-generated typos?
Background: A client is generating datasets (excel files). They asked I make an app to analyze the datasets, e.g. summary tables and figures. I'm doing this in R shiny.
Problem: There are many typos ...
- 141
2
votes
0
answers
243
views
Tools for reading data from large, irregular csv files (aka excel file hell)
I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
- 141
3
votes
0
answers
190
views
How to use zero-inflated negative binomial regression for binary classification task?
I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
- 321
0
votes
3
answers
544
views
Can we use median to replace all the missing values from a column
If we can use the median to replace all of the missing values from a column, then what is the advantage of using median over mean for replacing the missing values in a column with numeric values?
- 45
1
vote
3
answers
670
views
How to deal with highly skewed (on counts) dependent variables?
I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
- 321
4
votes
2
answers
594
views
Data wrangling for a big set of docx files advice!
I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (...
- 41
1
vote
1
answer
112
views
How to run KNN (or other) on nested features? Image metadata
I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I ...
1
vote
3
answers
76
views
What is the correct procedure when "joining" data takes ~6 hours?
I am dealing with bike-share data. I have 2 DataFrames:
trips_df (subset shown), total entries = ...
- 195
2
votes
1
answer
198
views
What should I do with the NaN values on this stock quote data?
I concatenated 3 stock quote data-frames all with date-time indexes.
However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
- 37
46
votes
9
answers
8k
views
How much of data wrangling is a data scientist's job?
I'm currently working as a data scientist at a large company (my first job as a DS, so this question may be a result of my lack of experience). They have a huge backlog of really important data ...
- 569
2
votes
1
answer
5k
views
R Combine Multiple Rows of DataFrame by creating new columns and union values
I have a dataframe in R that looks like this
...
user69420
3
votes
2
answers
1k
views
Inputting (a lot of )data into a dataframe one row at a time
I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this:
...
- 31
4
votes
3
answers
509
views
How to deal with count data in random forest
I am working on a classification model where my target class is a biased class with the class shape as
0 1
20694 101
Most of my features are the ...
- 143
4
votes
2
answers
8k
views
Mean across every several rows in pandas
I have a table of features and labels where each row has a time stamp. Labels are categorical. They go in a batch where one label repeats several times. Batches with the same label do not have a ...
- 167
2
votes
2
answers
6k
views
how to calculate number of datapoints within a given time interval?
I have a dataframe with one column which is a timestamp. I have converted that column to a datetime object using pandas to_datetime() method. However what I want is to ...
- 1,091
2
votes
1
answer
130
views
What is the difference between 'if the data is of good quality' and 'if the data is tidy'?
I'm doing Data Analyst nanodegree from Udacity. I'm confused between the difference even after going through the lecture a few times.
- 121
1
vote
0
answers
29
views
How to work with string data with a lot of NAs in an aggregation task with R
I have recently taken up a data wrangling task and inherited a data from my predecessor which contains keyphrases extracted by hand like the following :
...
- 11
1
vote
1
answer
10k
views
removing special character from CSV file
I read my csv file as pandas dataframe. Originally it's a dict with multiple entries per keys. Its looks like this after reading as pandas dataframe:
...
- 135
12
votes
4
answers
55k
views
Export pandas to dictionary by combining multiple row values
I have a pandas dataframe df that looks like this
...
- 223
1
vote
2
answers
732
views
How do I split number string with digit pattern?
I am trying to split number string to two to digit numbers
How do I get two different Numbers out that string
Example:
I want separate two numbers
...
- 13
3
votes
1
answer
20k
views
Populate column based on previous row with a twist
I'm struggling with a Pandas problem. I have the following data.
...
- 131
0
votes
1
answer
470
views
Unable to print in Jupyter Notebook using Pandas [closed]
I am doing basic data analysis on an csv file in jupyter notebook
def answer_two():
return (df['Gold']-df['Gold.1']).argmax()
answer_two
The above code ...
- 111
0
votes
2
answers
101
views
R Programming rearranging rows and colums from timeline data [closed]
I am new to R programming and data analytics. This is probably a trivial question but I could not find an answer. I'm also limited with my ontology of data science and data manipulation. So, I'm not ...
- 11
0
votes
2
answers
83
views
Adoptive(Machine wise) ML packages
What kind of Machine Learning/Data Mining packages are available that can easily be scaled on a cluster. For instance H2O is one ...
- 670
1
vote
4
answers
3k
views
Which one is better performer on wrangling big data, R or Python?
I want to know which language/packages performs better & faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and ...
2
votes
2
answers
144
views
How do you define the steps to explore the data? [closed]
I'm falling in love with data science and I'm spending a lot of time studying it. It seems that a common data science workflow is:
Frame the problem
Collect the data
Clean the data
Work on the data
...
- 123
0
votes
2
answers
44
views
Analytics term for turning row values into column names and count its assigned values
Do we have a data mining/analysis term for turning row values into column names and count its assigned values?
- 103
2
votes
2
answers
226
views
Technical name for this data wrangling process? Multiple columns into multi-factor single column
What is the technical name for the following data wrangling process? I want to collapse Table A into Table B. (To make the data suitable for ANOVA.)
Table A:
...
1
vote
2
answers
35k
views
Sum up counts in a data.frame grouped by multiple variables
This is a snippet of the dataset I am currently working on:
...
- 55