Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Extract only numbers within parentheses from a string

I am trying to extract only the numbers within parentheses from a string and extract them as numeric values. My current code keeps the % sign and extracts as a character value. For example, the column ...
0
votes
0answers
4 views

Identifying data distribution for my data and model building strategy

i am a beginner for panel data econometrics. Need help of experts of the field deciding if it is panel or pooled data or should i use any other methodology. I have data from wind mill and we use wind ...
0
votes
0answers
31 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
-1
votes
0answers
14 views

Convert dataset to table format [closed]

I am trying to understand the given dataset. I want it to simplified in table format. Basically, there is no primary key here. SF and SR are values. Y1, L1, Y2, L2, Y3, L3 are columns. Can someone ...
1
vote
0answers
16 views

Find the mode value and frequency in R

I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...
0
votes
1answer
14 views

How to transform and cummulate data in python

I would like to transform dataframe and cummulate them using pandas. year country value 1999 JAPAN 10 2000 KOREA 15 2000 USA 20 2001 USA 13 2002 JAPAN 30 * I want to transform ...
0
votes
0answers
13 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...
0
votes
2answers
35 views

Data Cleaning, refining etc

I have data as below: Carpanter Carepnter Carpentor Labourer Labor Labour Housewife House Wife housewife. I want to clean data and rectify the spelling mistakes ...
14
votes
10answers
3k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform analysis of it. In this dataset, the gender field ...
0
votes
0answers
61 views

Since is not possible test all the possible combination, what is the correct procedure to follow on building Machine Learning?

Sorry, I'm a little confused and this is a general question. How can I be sure that the procedure that I am following is the correct one? Following the steps for building a machine learning model, we ...
1
vote
1answer
13 views

Data Under fitting and its handling

Regularization is used to handle over-fitting problem. Similarly can we have some methodology to overcome under fitting problem or merely adding new features or training data will help us in reducing ...
0
votes
0answers
17 views

One Hot Encoding where all sequences don't have all values

Is there a way (other than manually creating dictionaries) to one hot encode sequences in which not all values can be present in a sequence? sklearn's OneHotEncoder ...
0
votes
1answer
18 views

redundancy or functional dependency of two columns

Being a beginner in Data Analysis I ask you not to be much judgemental about this question. I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a ...
0
votes
0answers
12 views

Trouble understanding how I could use multivariate time series to predict when an error will occur?

First off, I have very limited knowledge statistics-wise and am more of a coder. I was thrown into a large scale project and could use some guidance. I have a large multivariate time series dataset ...
0
votes
1answer
15 views

Why normalize function has a different result on a matrix vs single value?

I have a matrix like: B=[ 1.5035; 1.5728; 1.6485; 1.5369; 1.5467; 1.572; 1.5374; 1.787; 1.5825; 1.6905]; Using normalize function like normalize(B,'range') ...
2
votes
4answers
555 views

Is it good practice to convert columns with a number to a range between 0 and 1?

Relatively new to data science. I heard something about converting columns which contain integers into a range between 0 and 1. I think the reasoning was that so all the columns will be more similar ...
0
votes
0answers
17 views

Input missing data with interpolation [duplicate]

When dealing with missing values there are several strategies that can be used such as: imputing with the mean, median, a fixed value... I have read that you can do things such as interpolation. ...
0
votes
1answer
16 views

Python how to add a condition to the groupby while calculating the median of a field?

I need to add a new condition to the following line: train["Age"].fillna(train.groupby("Ticket")["Age"].transform("median"), inplace=True) Currently, its taking ...
0
votes
1answer
47 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
0
votes
0answers
14 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
0
votes
0answers
45 views

Normalizing Feature/Label with Negative Values

I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object ...
-3
votes
1answer
28 views

How to do feature selection for classification problem? Which technique will work? [closed]

I have 200 variables with 200000 records. How to find best features from this variables? I have tried correlation technique via Heatmap but all the variables have near to same correlation score < 0....
1
vote
0answers
29 views

Group_by 2 variables and pivot_wider distribution based on 2 others

Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all ...
2
votes
1answer
61 views

When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?

I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould. This completely changed my ...
0
votes
1answer
19 views

How to group a unique column and find the mean value in a separate column in R

My table of data looks like this image (but with many more lines): I want to be able to find the mean age of each region, so the mean age for region SSC20790 and so on. I believe I can use the <...
-1
votes
1answer
68 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
2
votes
1answer
55 views

Fixed-Width vs Adaptive Binning

I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
2
votes
1answer
41 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
1
vote
0answers
16 views

Replace values before and after a certain value [closed]

I have a variable like this: V1: 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 I want to change the 0 before the 1 to 1, and the 0 after the 2 to 2. It is supposed to look like this: V1: 1 1 1 1 1 1 1 0 0 0 0 ...
2
votes
2answers
40 views

Data Quality Assessment in Image Data

I'm working on a CNN model that classifies images. After scraping image files from the Internet, I found that many of them didn't look this way as described by the searching keyword (for example ...
1
vote
1answer
17 views

Replacing missing values with mean of feature calculated from previously replaced values

I am not sure how to ask this, but I will try my best. I have replaced some missing values in a feature with the mean of the feature conditional on a second categorical feature. However, not all ...
1
vote
0answers
18 views

Need some advice on approach to select only the informative emojis from the data set?

I have a giant data set from a local elections, which contains hashtags, emojis, and comments. I wanted to make a network analysis using only emojis. So far I have a network analysis graph made in R ...
2
votes
0answers
15 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
2
votes
4answers
50 views

Skewed Data doesnt follow normal distribution

My data (continuous) is highly skewed and it doesn't follow the normal distribution. Using sns.distplot I found out that exponweib fits the data better. How to ...
1
vote
1answer
211 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
1
vote
0answers
57 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
3
votes
0answers
17 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
0
votes
1answer
16 views

How to order the data with respect to data type

I am having large data set (82 variables) Is there any way to arrange data such a way that I have to get all numerical variables firstly then categorical variables so that I can run hypothesis ...
1
vote
3answers
67 views

Use of Standardizer to handle outliers?

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers. Actually I don't wish to drop the data as ...
1
vote
1answer
32 views

One scaler for all features or one scaler per feature?

I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?
2
votes
2answers
68 views

Recommended data cleaning techniques for multivariate time series prediction?

I have to predict the next step(s) in a multivariate time series with about 30 features and 50.000 samples. I am thinking of using LSTM. Which techniques are usually recommended for cleaning the data ...
0
votes
2answers
25 views

Dealing with data with no classes

I am new for data science subject and I am trying to learn on my own. Please bear with me if this is a stupid question. I have a dataset that has no classes. The data set comprises people's activity ...
0
votes
2answers
33 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
1
vote
2answers
74 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
0
votes
0answers
32 views

Linear regression plotting abline()

I have a ufc data set which i got and started cleaning for my own practice This the link to data:-https://www.kaggle.com/rajeevw/ufcdata#raw_fighter_details.csv and using ...
1
vote
0answers
13 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
2answers
60 views

How to handle addresses of the restaurants to feed the data-set in the ML model?

I have data from different restaurants which have also address of the restaurants now I want to predict the food delivery timing based on the given data, now the restaurant address is one of the ...
0
votes
1answer
16 views

How much of a disadvantage is a small sample size?

I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my ...
0
votes
0answers
22 views

Transforming Data for Classification

I have a dataset of the type: Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...

1
2 3 4 5
10