Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
1answer
51 views

Dummy Variable Trap

In my course about machine learning I'm studying multiple linear regression and we talked about dummy variable trap. I have a data set which contains country, height, weight, gender of every person ...
1
vote
0answers
47 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
0
votes
1answer
61 views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
0
votes
0answers
7 views

Manipulating the relative weight of tokens inside CountVectorizer

This is transparently a classic IMDB data recommender question. I'm trying to build a recommender system that suggests movies that a user is likely to enjoy. If I have my terminology correct I am ...
0
votes
0answers
21 views

How could you predict customer churn using transactions date?

I am new to machine learning and I would like to predict churn using dates of transactions. I tried to prepare my data and I couldn't obtain good results. I would like to predict unique Customers, ...
1
vote
0answers
113 views

Encoding with OrdinalEncoder : how to give levels as user input?

I am trying to do ordinal encoding using: from sklearn.preprocessing import OrdinalEncoder I will try to explain my problem with a simple dataset. ...
3
votes
5answers
234 views

How can I handle a column with list data?

I have a dataset which I processed and created six features: ['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList', 'totalProducts'] And the ...
1
vote
1answer
33 views

How do I change the location of Google BigQuery

So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks ...
0
votes
1answer
15 views

Is it acceptable to make new independent variables out of old ones?

I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or ...
1
vote
1answer
21 views

What is the best way to match street addresses in a dataset in Python?

I have a dataset of land parcels owned by the government and I am attempting to match street addresses to an existing list of government agencies. I've used fuzzy matching and used a regex that ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
1answer
25 views

Standardizing features by one specific feature

I am working on a project with a dataset that looks something like the following: ...
0
votes
1answer
81 views

how to use feature engineering on a label

I found a dataset about DDOS attacks id like to use but the tool used to extract features from pcap files seems to have indented the columns a little weird as seen below there are indents in unusual ...
0
votes
0answers
7 views

Identifying data distribution for my data and model building strategy

i am a beginner for panel data econometrics. Need help of experts of the field deciding if it is panel or pooled data or should i use any other methodology. I have data from wind mill and we use wind ...
4
votes
2answers
120 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
1
vote
1answer
24 views

Find the mode value and frequency in R

I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...
0
votes
1answer
22 views

How to transform and cummulate data in python

I would like to transform dataframe and cummulate them using pandas. year country value 1999 JAPAN 10 2000 KOREA 15 2000 USA 20 2001 USA 13 2002 JAPAN 30 * I want to transform ...
0
votes
0answers
25 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...
0
votes
2answers
48 views

Data Cleaning, refining etc

I have data as below: Carpanter Carepnter Carpentor Labourer Labor Labour Housewife House Wife housewife. I want to clean data and rectify the spelling mistakes ...
14
votes
10answers
3k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform analysis of it. In this dataset, the gender field ...
0
votes
0answers
71 views

Since is not possible test all the possible combination, what is the correct procedure to follow on building Machine Learning?

Sorry, I'm a little confused and this is a general question. How can I be sure that the procedure that I am following is the correct one? Following the steps for building a machine learning model, we ...
1
vote
1answer
16 views

Data Under fitting and its handling

Regularization is used to handle over-fitting problem. Similarly can we have some methodology to overcome under fitting problem or merely adding new features or training data will help us in reducing ...
0
votes
0answers
29 views

One Hot Encoding where all sequences don't have all values

Is there a way (other than manually creating dictionaries) to one hot encode sequences in which not all values can be present in a sequence? sklearn's OneHotEncoder ...
0
votes
1answer
28 views

redundancy or functional dependency of two columns

Being a beginner in Data Analysis I ask you not to be much judgemental about this question. I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a ...
0
votes
0answers
15 views

Trouble understanding how I could use multivariate time series to predict when an error will occur?

First off, I have very limited knowledge statistics-wise and am more of a coder. I was thrown into a large scale project and could use some guidance. I have a large multivariate time series dataset ...
0
votes
1answer
16 views

Why normalize function has a different result on a matrix vs single value?

I have a matrix like: B=[ 1.5035; 1.5728; 1.6485; 1.5369; 1.5467; 1.572; 1.5374; 1.787; 1.5825; 1.6905]; Using normalize function like normalize(B,'range') ...
2
votes
4answers
566 views

Is it good practice to convert columns with a number to a range between 0 and 1?

Relatively new to data science. I heard something about converting columns which contain integers into a range between 0 and 1. I think the reasoning was that so all the columns will be more similar ...
0
votes
0answers
27 views

Input missing data with interpolation [duplicate]

When dealing with missing values there are several strategies that can be used such as: imputing with the mean, median, a fixed value... I have read that you can do things such as interpolation. ...
-1
votes
1answer
11k views
0
votes
1answer
22 views

Python how to add a condition to the groupby while calculating the median of a field?

I need to add a new condition to the following line: train["Age"].fillna(train.groupby("Ticket")["Age"].transform("median"), inplace=True) Currently, its taking ...
0
votes
2answers
31 views

Replace Values with corresponding numbers in the same dataframe

Please help me to replace NR values with corresponding values as in the dataframe.
1
vote
2answers
175 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
0
votes
0answers
20 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
0
votes
0answers
100 views

Normalizing Feature/Label with Negative Values

I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object ...
-3
votes
1answer
32 views

How to do feature selection for classification problem? Which technique will work? [closed]

I have 200 variables with 200000 records. How to find best features from this variables? I have tried correlation technique via Heatmap but all the variables have near to same correlation score < 0....
1
vote
1answer
43 views

Group_by 2 variables and pivot_wider distribution based on 2 others

Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all ...
2
votes
1answer
70 views

When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?

I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould. This completely changed my ...
0
votes
1answer
20 views

How to group a unique column and find the mean value in a separate column in R

My table of data looks like this image (but with many more lines): I want to be able to find the mean age of each region, so the mean age for region SSC20790 and so on. I believe I can use the <...
-1
votes
1answer
76 views

Renaming Column Elements of DataFrames

I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following: Data Retrieval: Take partial strings from column elements from DataFrame1 ...
2
votes
1answer
142 views

Fixed-Width vs Adaptive Binning

I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
2
votes
1answer
45 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
1
vote
0answers
16 views

Replace values before and after a certain value [closed]

I have a variable like this: V1: 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 I want to change the 0 before the 1 to 1, and the 0 after the 2 to 2. It is supposed to look like this: V1: 1 1 1 1 1 1 1 0 0 0 0 ...
2
votes
2answers
83 views

Data Quality Assessment in Image Data

I'm working on a CNN model that classifies images. After scraping image files from the Internet, I found that many of them didn't look this way as described by the searching keyword (for example ...
1
vote
1answer
17 views

Replacing missing values with mean of feature calculated from previously replaced values

I am not sure how to ask this, but I will try my best. I have replaced some missing values in a feature with the mean of the feature conditional on a second categorical feature. However, not all ...
1
vote
1answer
23 views

Need some advice on approach to select only the informative emojis from the data set?

I have a giant data set from a local elections, which contains hashtags, emojis, and comments. I wanted to make a network analysis using only emojis. So far I have a network analysis graph made in R ...
2
votes
0answers
16 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
2
votes
4answers
64 views

Skewed Data doesnt follow normal distribution

My data (continuous) is highly skewed and it doesn't follow the normal distribution. Using sns.distplot I found out that exponweib fits the data better. How to ...
1
vote
1answer
732 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
1
vote
0answers
113 views

passing a dataframe as a argument while applying a lambda function on a dataframe [closed]

I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the ...
3
votes
0answers
17 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...

1
2
3 4 5
11