Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

1
vote
1answer
7 views

data duplication optimisation

I am working in python3 cleaning data. I have a large number of midi files from scraped from a variety of sources using beautiful soup. Many of them may be duplicates. Some of them have only numbers ...
0
votes
2answers
19 views

Understanding missing values in dataset

I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) ...
0
votes
0answers
8 views

Dict with features and classification -> two seperate but aligned lists

For my ML project, feature set extraction is expensive, but I need to be able to retrain the model on a fairly regular basis. For this reason I'm extracting my features from all of my documents at ...
1
vote
3answers
41 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
0
votes
0answers
9 views

Removing ambiguous data: Same input variables with different class labels

Background: I'm working with a tree-based ensemble model on a large data set. The target variable y is a binary attribute that have two classes (True and False). I noticed that some of the ...
1
vote
0answers
12 views

Term for an identifier that has been superseded

Is there a 'proper' term for an ID (or IDs) that have been superseded by (or merged into) another ID? My use case: rsIDs are used by geneticists to refer to a SNP. These take the form of a string '...
0
votes
2answers
22 views

Tips for checking data integrity / data sanity?

I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets. I am interested in ...
0
votes
0answers
10 views

Script to convert non-numeric elements that are numbers to numeric

I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward. I downloaded some Google Play Store ...
2
votes
0answers
43 views

Jaccard Similarity with Binary Data

I have 5400 rows of data and 3211 columns of attributes. The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity ...
1
vote
2answers
14 views

Keeping part of a string in R [closed]

I have a dataframe with the following column city <- c("Sydney NSW", "Newcastle NSW", "Liverpool NSW", "Broken Hill NSW") I want to maintain everything prior ...
3
votes
1answer
59 views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
0
votes
0answers
5 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
0
votes
0answers
14 views

Apache Nifi out of memory

I've set up a data pipeline using Apache nifi. It processes 3GB scale json data. When I try to flatten these Json files it always outputs: ...
1
vote
0answers
23 views

Keras Binary Classification val_acc won't go past ~67; Full data and code included

I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
1
vote
1answer
28 views

How to normalize data from multiple sources?

I am trying to model an individuals' purchasing behavior using different data sources (ex: Zalando, Otto, etc.,). When I combine data sources, I see that the data across these channels is very ...
1
vote
1answer
44 views

Is this a data issue, or a model issue? A Keras binary classification model

I've been trying to create a binary classification model that predicts wether there will be a train delay based on the train and time. Here is a link to the data The issue I'm having is that my ...
0
votes
0answers
43 views

creating a dataframe out of nested loop data [on hold]

I have three parameter arrays ,Parameter1 = [6,7,8],Parameter2 = [11,12] and Parameter3 which can be calculated using Parameter1 and Parameter2. There is a mapping between parameter 1 and 2. I want ...
2
votes
2answers
49 views

Possible Challenges for a Data Science Escape Room

Dear Data Science Community, as my project for my bachelorthesis I am working on a concept for a physical Data Science Escape Room. The goal of the escape room is to create awareness for data science ...
0
votes
0answers
28 views

Saving multiple numpy arrays to multiple image files

I have some CSV data that I have extracted in chunks of 375. Since I have 20 columns of data (7500 elements total), I have then reshaped these chunks into an image in the form of (3,50,50) to ...
0
votes
0answers
11 views

How can I cluster data by using multiple views of the data with a common column using OpenRefine?

I need to cluster my data, but there are multiple ways to view the data such that I can't really place it all into a single table. I have been working with Open Refine to cluster this data but I keep ...
1
vote
0answers
15 views

How to concatenate many .psv files in google collaboratory?

I have a folder named 'training' in my local drive which has 20000 .psv files. I zipped it and uploaded to google collaboratory, with the upload option in the Files section. I unzipped it with the ...
0
votes
0answers
24 views

Accurately choosing a model with sequential data

The dataset I'm working on is mapping journeys - breaking them down into entry & exit coordinates, and entry & exit times, for each part of the journey. My goal is to predict the final exit ...
2
votes
1answer
39 views

Python - Create many dummy variables from one text variable?

I'm trying to create dummy variables for a variable that has text data in rows. Data in 1st row is: ...
1
vote
1answer
24 views

Is there a model that can adapt to additional new training data with different columns?

My training data comes in batches. Sometimes, new batches (completely new samples) come with new columns that are not in old batches, or they may be missing some of the old columns. For example, ...
0
votes
2answers
38 views

Loops in R programming,

I want to update this Remaining column in this table below: How do I do this in R programming? I tried while (n.Remaining>0) { n.remaining <- n.total-n.expense} Desired Output:
0
votes
1answer
21 views

Can you apply PCA to part of your dataset?

I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
0
votes
0answers
25 views

Filtering unique row values in SQL, Advanced

My data looks like this: Number(String), Number2(String), TransactionType(String), Cost(Integer) For number 1, Cost 10 and -10 cancel out so the remaining cost is 100 For number 2, Cost 50 and -...
0
votes
3answers
44 views
0
votes
0answers
28 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...
1
vote
1answer
26 views

R Combine Multiple Rows of DataFrame by creating new columns and union values

I have a dataframe in R that looks like this ...
0
votes
2answers
43 views

Dataset Merging

I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
0
votes
2answers
83 views
0
votes
0answers
38 views

XGBoost feature significance and feature importance

In a regression model it is possible to judge at a specified significance level (often alpha = 5%) whether a variable has a significant influence on the target attribute. With XGBoost, you can use ...
0
votes
1answer
40 views

Normalization and Outlier on Target variable which is continuous

I have doubt that should I perform outlier analysis and normalization even on target variable which is continuous ?
2
votes
1answer
24 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
2
votes
2answers
59 views

Dealing with a dataset with a mix of continuous and categorical variables

How do the choice of machine learning algorithm and preprocessing change when some of the independent variables are categorical while others are continuous? Can such data be directly applied to the ...
4
votes
1answer
80 views

How do I find the relevant features out of 11,000+ possibilities?

While working on Kaggle Competition, I ended up with 11,726 columns which are mostly "dummies" (one hot encoding). Is this too many? I know that we need to find out which features are relevant, but ...
2
votes
1answer
41 views
6
votes
2answers
504 views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does as per the Sci-Kit documentation. Normalizer - https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Normalizer.html#...
1
vote
0answers
16 views

Visulazing a specific value of a column with the corresponding values

I have recently started using data studio to make visualizations before diving into analysis and data prep! I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
0answers
37 views

How to encode H3 geohash in regression model

I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3. I'm struggling ...
3
votes
0answers
39 views

How to deal with count data in random forest

I am working on a classification model where my target class is a biased class with the class shape as 0 1 20694 101 Most of my features are the ...
1
vote
0answers
30 views

preparing time series data for building a rnn

I am preparing time series data for to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a ...
3
votes
0answers
37 views

How to filter this signal to get a heartbeat signal?

I am giving my first steps in data analysis, gathering/cleaning. To learn, I am trying to create a simple code that can detect heartbeats from color variations from the image coming from the camera ...
2
votes
1answer
164 views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
1
vote
0answers
7 views

How to a object type data which contains a string of unique id for feeding in my model?

I am having a dataset in which data type of unique id of a user is in object form. I need to convert it into Int for feeding this data into my model. here is first rows of my dataset. ...
2
votes
2answers
432 views

Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity. For example : string one : 'Pair of women's ...
1
vote
1answer
15 views

How can I create a new column of binary values from my TfidfVectorizer sparse matrix?

I currently have a sparse matrix object of TfidfVectorizer which is of 1000 length. Right now it is displayed like this : ...
1
vote
1answer
14 views

How can I extract the top words from a string in my dataframe column?

I have a column in my dataframe which is in a string providing description of a product. For example : This is a shirt. It is blue in color. The sizes available are large, small. The shirt is tight ...
1
vote
1answer
17 views

How can I see a long string in my dataframe?

I have a column in my dataframe in which there are sentences which are too long. I want to see them as a whole but every time I perform even a simple iloc operation i get output like 'i am going to...'...